07.08.2014 § 4 Comments
Opture is the term I propose for a new artistic medium which recently has been made possible by the advent of consumer virtual reality technology: immersive story experience and world exploration controlled exclusively by head and gaze tracking.
Opture is distinct from both VR cinema and VR games. It is not VR cinema because it includes not only immersion in the story world but also the power to interact with it. Due to the constraints on this interaction, however, the experience of playing an optie is much different from that of playing a game; without any button-based input, voice commands, or body tracking, an optie player should feel in a more purely cognitive state.
Opture is much more than a cross between film and game, though. Because inside an optie the player knows that their watch itself is being watched, in every split-second, opties have the potential to be unprecedentedly active, intimate, and engaging experiences. The human gaze is closely intertwined with our train of thought and our willpower, so in sophisticated works of opture players may feel no less than their imagination coming to life. This dreamlike agency is what will draw people to opture again and again.
Now, at one extreme, opture could be as game-like as letting the player control a cursor with the direction of their gaze, and blink to click. And at the other extreme, it could be as VR film-like as to correlate head motion only and always with a change in vantage, and keep hidden from the player the nature of any effect their attention may be having on their experience. The most original and exemplary opties, though, should explore the possibilities in-between these extremes, dynamically.
To my knowledge, no true works of opture have been created yet. Auteurs and theorists alike, rejoice: as an entirely new medium of storytelling and world sharing, opture calls for entirely new styles, methods, and grammars. Certainly there will be those who use this new virtual reality technology to make striking yet gimmicky statements, as the film nickelodeons did a century ago. But like the pioneers who innovated continuity editing, montage, composition in motion, the 180 degree rule, etc. — creating the film language from the ground up and paving the way for others to express themselves in it — a huge opportunity exists for us today to spread understanding and appreciation for the unique powers of this new artistic medium. Enough territory remains open for each of immersion and gaze tracking alone, but at their intersection is where the gold likely lies. This is a tremendously exciting time!
I coined the word “opture” by hiding the Proto-Indo-European root ‘op’, for choice or preference (as in “option” or “opinion”), inside the Greek root for eye or sight, ‘opt’ (as in “optics”); thus, in opture, our choices play out invisibly through our eyesight.
- Like the word “film”, “opture” refers both to an individual work (the suffix ‘-ure’ as in [moving] “picture”) and to the art itself (the suffix ‘-ure’ as in “culture”).
- “Optie” for short comes from the ‘-ie’ ending of “movie”.
- The verb “to play” an optie is suggested because of its ties both to “playing” games and “pressing play” on a video file.
- In more formal settings, “optigate” may be the more appropriate verb for playing optures, using the Latin ‘agi-’/’act-’/’-ig-’ root for drive/act/do/go/move, as in “navigate.”
- I suggest “optural” as the adjectival form, since the alternative ‘-ive’ and ‘-orial’ suffixes seem more exceptional.
- An “optographer” would be a maker of opties.
Leading consumer HMDs do not presently feature gaze tracking technology. However, there are several good reasons to expect it to be included in the near future:
- Gaze tracking can be used for dynamic inter-pupillary distance calibration. Distance between pupils varies from person to person, and even varies for each given person depending on the distance they are focusing on. Knowing the exact inter-pupillary distance at all times is critical for fine-tuning stereoscopic presentation; the tiniest inaccuracy can still contribute to motion sickness, a central challenge that VR engineers are striving to overcome.
- Gaze tracking is also necessary to achieve foveal rendering, a holy grail of VR engineering. Reducing visual processing requirements by an order of magnitude would be a huge boon to the industry.
- Finally, gaze tracking could also enable gaze-dependent depth-of-field. In cinematographic imagery, viewers are limited to the focal length chosen for them, which gives rise to a certain type of beautiful artistic experience. However, allowing the viewer to change focal length just as they could if they were there seeing with their own eyes opens up the option to elevate this art beyond the cinematographic and into the realistic. Computer animation is already capable of providing such images with dynamic focus capability, and plenoptic cinematography may be available as well.
Even if gaze tracking technology does not see wide adoption by the virtual reality community, the existing rotational head tracking functionality still offers a huge amount of possibilities for optural expression. And on the other side of things, since control interfaces based on gaze tracking which are independent from virtual reality are simultaneously becoming popular, it would be possible to create opties whose immersion aspect is approximated with parallax.
In terms of development, marketing, and distribution, it may be helpful to think of optures less like films and more like video games. The key factor here is the interactiveness.
Optie creators wield freedom with players’ experiences more on the order of game designers than film directors. A screenplay is plainly insufficient as a blueprint for an optie; a great many further considerations must be made for each scene: how it may be controlled, how it ends, and where it goes. While some opties may have a set runtime like a film, others may allow players to affect the pacing or even the events portrayed. Therefore works of opture should exhibit a variability of scope which is more on par with the gaming industry than the film industry.
Also, opties will not be able to be played as media files on ordinary VR cinema players. Each opture will have unique manners of reacting to players’ attention. While undoubtedly standard optural modes will be determined, and thus perhaps certain formats may give rise to specialized media players for convenience, I expect most opties will be too customized to come packaged as anything other than stand-alone programs.
One of the most important ways to classify optures will probably be by the extent of control afforded to the player:
- Vantage only. This is just a shade beyond VR cinema; the single difference is that the relationship between head rotation and vantage within each shot may be more complicated than the standard direct correlation, thus opening the door to possibilities beyond mere embodiment. Affecting the position of the vantage is also possible, but montage, duration, and story are all still set.
- Montage. One may further influence an opture’s editing by directing attention in certain ways or to certain targets. This could include cutting between different perspectives on a scene, or choosing unique slices of action during periods of parallel editing.
- Time. The duration of optie becomes a variable. Applications range from mass media interests — tailoring pacing using analytics to determine the level of a player’s engagement with individual scenes.
- Diegesis. Everything is up for grabs. Players affect not just the storytelling but also the story itself, its characters, and its world.
Social aspects exist on a separate axis:
- Asynchronous. Record your particular optie experience and share with someone else as a sort of personal immersive film version.
- Synchronous. Play together simultaneously. Be able to see where your playing partners are looking, perhaps as an unobtrusive crosshair, perhaps color-coded if they are ahead or behind you in time, etc. Your experiences are otherwise completely independent.
- Cooperative. Play together, but now with the power to influence or share each other’s experience. Keeping vantage bound to your head rotation is probably still important, but other factors like montage, duration, and story could be affected in new ways through combination of inputs from multiple players.
Finally, I expect opture will be categorized by the level of transparency of the players’ control. Easier opties may hold the players’ eyes, so to speak, as they lead them comfortably through the experience. Experimental or puzzle opties, on the other hand, may intentionally confuse or deceive players about the effects their attention is having on their experience (pseudoludonarrative). Horror opties may specifically disobey the intentions of their players.
Initial Theory and Ideas
To help get your mind going imagining optural experiences, here is a collection of unordered ideas I’ve had so far:
- Look to relocate. See something that looks like you’re meant to get a closer look at, or some spot from which you’re probably meant to see from next? Look at it. Hold your gaze a moment, and the optie will accept this as your decision to move there. A game like Myst would fare particularly well as an early optie: a limited set of vantage points traveled between in punctuated cuts.
- Character embodiment. First person perspective is rarely used in traditional film and television storytelling. However, because in immersive media every shot is (in a sense) first person, I predict that it will be a more palatable mode in both VR cinema and opture, i.e. that first person points of view will be mixed in with the third person and not feel gimmicky or odd. Experiencing a character’s actions and hearing a character’s words as if they are coming from ones own hands and mouth may bolster emotional/psychological involvement with them.
- Distributed character embodiment. In a typical film, a conversation between two characters is often shot in alternating over-the-shoulder shots. In a typical optie, perhaps, such a conversation would tend to be told instead in alternating first person perspective shots. Imagine being asking to choose which character to experience a conversation from the perspective of, or whether or not to make eye contact during. Normal film spectators are already capable of sustaining emotional/psychological identification with multiple characters in a scene at once — feeling e.g. both the anger of one and the shame of another. I predict that players will be able to sustain a distributed sense of physical embodiment as well, in at least two characters at once. This distributed embodiment should not compromise the aforementioned effect of each of the individual embodiments bolstering emotional/psychological identification with the respective character.
- Look down to cut. Cues may be given to the player to train them how to control montage such as this.
- Squint to zoom.
- Examine. Rather than translate rotational input as different angles outfrom a fixed point — as is standard for a sense of embodiment — a shot could translate it as different angles in toward a fixed point from a fixed distance. Thus, rotating your head would cause your vantage to swivel around, in inverted motion, as if on the inside of an invisible sphere surrounding an object of interest. This technique should be useful when an optie director wants to provide you in a particular moment with the freedom not to look around a space, but rather the freedom to examine a specific thing however you choose. Experimentation should be done to determine whether this sort of experience feels natural or comfortable.
- Coaxing. Sometimes as optie designer one may wish to artfully restrict players’ options, to guide them to experience a shot in a certain way or limited set of ways. For example some shots may be best experienced motionlessly, or others may be best with only the ability to pan side to side. However, breaking correlation between head rotation and vantage completely can induce sickness. Ultimately there is nothing one can do to restrain players’ necks, so the best one can do is coax players to watch in a desired way. The basic mechanism is to promptly but smoothly correct against certain rotations, as if rotating the entire world under or around the player’s feet; after this happens, the viewer will intuitively undo their ineffectual motion, returning to a more at-rest position. This effect may sound drastic, but once players recognize that their current optie features it, they will adjust quickly; by initiating their looks around carefully at first, friction between players’ wills and that of the designer will become quite subtle. Using coaxing, optie designers can bestow upon players the feeling of having the skill of a professional cinematographer, rather than just an average dude erratically looking all around wherever and however they feel like.
- Endurance challenge. Next-gen art-house long takes test the player’s ability to concentrate. The shot zooms out, and new images enter into your periphery — but if your curiosity gets the better of you and you look away from center, you fail. Perhaps you must restart the shot, or perhaps miss out on a visual reward. It’s like the candle shot from Andrej Tarkovsky’s Nostalghia, crossed with the self-restraint confronting the Zone at the end of his film Stalker.
The pursuit of full-body virtual reality presence is a beautiful thing, but it is not my thing. Please wake me up once someone’s built a simulator for feeling like one is inside the game Portal — experiencing a shift in the direction of gravity passing over ones body like a wave. I expect that this won’t come until we’re tapping directly into neurons, at which point “ear tracking” will become as possible as gaze tracking and I’ll probably switch over to become a “dynamic musician” or whatever they’ll call it.
Until then, there is plenty of beauty to be explored by extending the tried-and-true seated experience, as the team at Oculus Rift has been preaching. Consider Lucky’s Tale (or as I like to call it, after Mario 64’s groundbreaking cameraman character, “Lakitu”) as a genius and subtle example of tapping the freedom of vantage into a working genre. Let’s see this in an RTS soon, too!
Regarding newfangled technology allowing one to control games with ones thoughts directly, if it becomes precise enough to trump eye tracking for these purposes, then all bets are off.
A danger of increasingly comprehensive mediums is that emphasis risks falling on the sensory material itself rather than the thought or emotion which it has newfound power to express. And of course there will always be artless excess in any medium. This argument came up with film and ridiculously enough is still ongoing with video games: whether they are an art or not. Opture is approaching the ultimate medium, and it will certainly face this criticism too. Without great content, opture will be nothing.
Fortunately I have a couple ambitious plans for feature length optures ready now — optures which have deep emotional and intellectual scope. Furthermore, they both play with the opture form reflexively.