Enter VR interview: Douglas Blumeyer on immersive cinema, etc.

I was recently interviewed by Cris Miranda for his Enter VR podcast to get my perspective on immersive cinema. You can listen to the full thing (and find more examples of his incomparable breed of interviewing) here:

http://www.stitcher.com/podcast/cris-miranda/entervr/e/rethinking-vr-cinema-ai-consciousness-clones-brain-to-brain-40313488

Since I find my own voice tedious and annoying to listen to, and this is a feature length film’s worth of mostly my voice, I went ahead and typed up the more interesting parts. Hopefully this alternative form of consumption broadens its accessibility. Also I distilled down and liberally refined material and added links as appropriate.

We started off just talking about The Matrix — not immersive cinema, but cinema about VR & AI, and a big part of both of our childhoods. But we end up covering a range of stuff beyond the more on-topic stuff like immersion and interactivity — from ultimate mediums and psychedelics, to individuality vs. collectivism and generative art. Without further ado!

 

CM:

Virtual worlds will be perpetually attached to artificial intelligence for the rest of time. To me it seems like the manifestation of artificial intelligence in our time will first come out through virtual worlds. I don’t think we’ll see androids or robots walking around among us that will be sentient and self aware, you know? I think it will happen later, but first we will see them in VR.

 

DB:

I agree. Virtual Reality is built on technologies that are far ahead of what robotics can achieve in the physical world.

And here’s another thought coming at that point from the other direction. I expect some listeners may have heard of the Uncanny Valley before, which is that little gap between Realistic — you believe this is a real person I’m engaging with — and Looks-Kind-Of-Like-A-Person. In between that, where it’s like Very-Close-But-No-Cigar, we’re biologically wired to resist (I think this evolved for a reason; an animal such as a human wouldn’t want to breed with something that was not correct, that had mutated in a way that made it off enough that the brain registered it as foreign or risky). Now love is infinitely complicated, but things like love are pretty easy to elicit and replicate, to get from people, and I have no doubt that people are going to be falling in love with artificial intelligences in the near future. It’s probably already happening to some extent. But I think that we’d be sooner to fall in love with virtual reality artificial intelligences than physical robot artificial intelligences. And my reason for that is that when we’re in virtual reality our standards are lower, even when we’re socializing with virtual entities which we know are backed by a real person. If you’ve ever done something like Altspace‘s social experience, even when you throw in things like eye tracking so you get this very intimate level of eye contact, or if you get hand tracking because hand gestures are very important to bodily communication, even when you throw all of that into the mix there’s still something off about it, and so if the playing field is leveled such that even humans are kind of off then I think it’d be easier for an AI to catch up and be indistinguishable. And as the role of VR increases in our lives and we spend more and more of our time in virtual environments our relationships will on average include AI more. It won’t seem as strange.

 

CM:

Is falling in love with an AI considered cheating?

 

DB:

I don’t think it makes sense to make objective judgments. It depends from relationship to relationship. It’s going to add a new line couples may have to draw. With virtual reality porn, couples will have to decide what’s the line — what’s a real sexual experience and what’s not.

 

CM:

That line is so blurry, and also that line is moving as technology progresses. What if the AI … has a virtual representation of my girlfriend. Is it considered cheating if ‘The AI looks like you, honey!’

 

DB:

Simulating doing things with a sexual partner they wouldn’t normally do, is that what you’re suggesting? That’s a whole other line. Well, probably a really similar line when you think about it. But this isn’t just a sexual thing… maybe you’d want to simulate having other types of experiences with people you know that you otherwise wouldn’t have.

 

CM:

And what if in virtual reality everything you say and do gets tracked, logged, stored away for algorithms to learn from you and put out an AI representation or mirror reflection of you in VR. This thing would have autonomy in the metaverse. ‘I’m you in VR.’

 

DB:

There’s going to be intellectual property issues about the self. What does it mean to be me and can you abuse it.

 

CM:

So… what’s in your mind these days?

 

DB:

My particular interest is less about AI and more about storytelling. AI implies a level of interactivity and unpredictability. The intersection I’m most interested in has the least to do with interactivity and game-like structures as possible. My background is in film and while immersive cinema inherently has some interactive nature because the person experiencing an immersive film can choose to look anywhere they want at any given time, other than that necessary interactivity I’m more interested in figuring out how can we still tell linear set-in-stone stories or story-like experiences. What are those going to be like.

 

CM:

What are you finding out the more you think about these things?

 

DB: 

Immersive storytelling is a brand new thing. It’s hard to say exactly what direction it’s going to go. It’s probably going to go in many different directions. It’s informed by not just traditional film but also theater — if only because unlike in film you’re actually occupying the same three dimensional space as whatever performers and actions are being dramatized — and also from game — if only because of those at-least-subtly interactive elements.

I was looking for a good definition of cinema and there’s one on Wikipedia which says “the art of simulating experience,” which is really interesting actually; it’s not the first thing I’d think of but I think it’s astute. I’ve since noticed people referring to literature this way, too. I think you can find — if your’e looking for it — a continuum from text-based storytelling to film-based storytelling toward immersive storytelling in that they’re all striving to simulate human experience. And what that continuum describes is an increase in the comprehensiveness of the phenomenal stuff, as technology enables us to more completely capture sensory channels and convey those directly to people.

I feel like every time we make a technology step forward there’s a period of resistance to it and a fear that it’s compromising the potential for artistry, you know, that it can’t help but be less subtle and speak less purely to the soul, that it can speak only to the body and then just maybe the mind. I don’t doubt that this is true to some extent for some experiences… when you have access to convey sensory information more directly it is easier to make interesting but kind-of-dumb not-very-spiritual experiences. I do think it’s also possible to direct that in a way that it can seize the soul and convey even more awesome spiritual truths.

 

CM:

Where is this continuum leading us to, in terms of these mediums of self-expression? An ultimate or final medium?

 

DB:

I think ultimately we all want to feel more connected. I’ve heard the fractured human experience described this way: we are already one consciousness, we’re just not very organized yet. You can glimpse moments of our interconnectedness, our collective consciousness, but it’s overwhelmingly an experience of being an individual, an experience of isolation and fear and pain. I believe that things could be better if I could be inside everyone I knew more intimately, more deeply. That’s something I aspire toward and that’s something I think that a lot of artists share as a mission: to try to turn themselves inside out in one way or another and plumb the depths of what makes them themselves, indirectly in an effort to find that which is generalizable, and package these explorations and share them in a way that other people can dive into and find out things about themselves, and in turn modeled after their experience in the first artist’s work create their own experiences of this sort. I do believe we’re all artists and I think if you don’t think you’re an artist then that’s fair but you’re probably using a more specific or superficial definition of artist than I am.

So anyway, what would be next after virtual reality would be ‘mainlining’ (borrowing a term I learned from Alex Voto). To introduce mainlining: when I first started hearing all the modern-day excitement about virtual reality my first thought was ‘Oh my god, I want to play Portal in VR.’ I want to feel the boundary passing over my body from head to toe — the boundary between two differently-oriented volumes connected by a portal — and I want to feel this immaterial boundary through no sensation other than sensing the disparity in the direction of gravity between that of the volume where I’m coming from and that of the volume I’m going into, feel the two gravities pulling me separate directions while I’m still partway in both. Well, we’re not going to have that for a while. That’s going to require wiring into our brains and tricking our brains at a level past our eyes, ears, and skin. That’s what’s meant by mainlining. And that technology is completely unrelated to the hardware we’re working on nowadays, to perfect this tier – the tricking eyes ears and skin one.

But mainlining is coming eventually, and I think a lot of lessons we take from virtual reality — from immersive cinema, games, and any sort of virtual experience — can be applied to mainlined media.

Mainlining will open up even more stuff. It’s called the ultimate or final medium for a reason. This is where we get to the things I really dream about: building virtual drugs. Approaching demiurge level, where you’re essentially just a creator god. You can send electrical signals, you can write a script that when applied to a human mind will give them a type of psychological, spiritual experience beyond mere sensory information. They will react to it and populate it with their own memories and preconceptions in a certain way. And it’s just like tripping with friends insofar as everyone will come out of it having had a similar experience, one they can consider to have shared, but each person’s experience will be different from the next in way fundamentally different than the difference we think of when you and I watch a movie together and walk away with a comparable but different experience.

 

CM:

What are your predictions for the future of immersive media?

 

DB:

If you force me to predict, I’d say genres will coalesce around sweet spots of interactiveness, such as I describe in my proposed format of opture. (I cover a lot of this same material in my post about opture). And not just on a simple continuum from interactive to non-interactive; there are different kinds of interactivity:

• Can you actually affect the outcome of the story?
• Or can you simply affect the way in which events unfold, the montage?
• Whether you experience it chronologically?
• From which spatial angles you see things?
• Are you aware of the capacity in which you are interacting with any of the above, or is it maybe the author’s intention for you specifically to not understand how you’re affecting some of these?

Maybe an example would help. Suppose you’ve got a standard conversation between two characters sitting at a diner.

In a traditional film you’d get a master establishing shot and then start going back and forth between over the shoulder shots, maybe some close ups mixed in. In an immersive film, though, since every shot or vantage is in a sense already a first person shot — because you are that person looking around — it might be more natural to experience things from the points of view of the two characters. While I feel like point of view shots are not unheard of in film and TV, we see them pretty often and aren’t surprised when we see them, they’re still striking, we notice them and wonder about their importance. Maybe in VR they would be not as big a deal. They might turn out to be the more natural way to experience the scene.

Now imagine one way you could experience the scene would be always from the perspective of whichever character is talking: you are made to feel like you are the active character. Alternatively the author might have you do the opposite, always experience it from the perspective of the character being spoken to. Both of these are fundamentally passive viewing experiences but one of them could give you a different level of identification than the kind you get when watching traditional film. You might be made to be more sensitive to the words spoken. I’m way more critical of the words that come out of my own mouth, so a line with a character in some movie you might react to like, ‘Eh, some people are just like that’, in an immersive media you might react instead more like, ‘Shit, so this is how people get this way.’

So to finally get to the actual bit about interactivity: in this example, whose perspective you experience it from could depend on your choices. The experience could be programmed so that as long as you held eye contact you’d stay as one person, but if you ever broke eye contact, it would switch you. Or you make eye contact with the waitress and the other character character reacts to that.

This all said, I think game-like experiences are going to become increasingly popular and non-interactive cinematic type experiences less and less popular, and I’m okay with that.

 

CM:

What have you been working on yourself?

 

DB:

I’ve had issues with my DK2 recently, but I was able to build a dynamically spatial sound sculpture. I also worked on a mobile game for a bit and an animated film.

 

CM:

Can you tell us a bit about what you learned working on an immersive animated film? It sounds really hard. I have no clue what that would look like.

 

DB:

Storytelling is hard in immersive cinema. Editing is particularly hard. We barely made it past just technically getting control of the camera and making it do what we asked.

When it’s time for an edit, a consideration is reacting appropriately to the direction the user happens to be looking. The moment of the cut needs some flexibility where in traditional film there is none whatsoever.

Continuity editing (film theory term referring to things most people take for granted: match on glance, match on action — one shot shows a character opening a door, the next is from inside the room they’re going into — you piece the spatial reality of the film together) — that’s just not going to be as practical or important at all.

 

CM:

What’s the solution to preserving continuity? What techniques need to be implemented to produce a constant narrative?

 

DB:

We take continuity editing for granted, especially if we’re accustomed to watching modern — and especially Hollywood style — movies. That’s become the accepted way to edit film together for the mainstream, because it’s really easy to watch and understand. But this style of editing had to be researched, developed, and refined over the years, and it was by no means obvious or considered necessary. If you watch earlier films, or more experimental films throughout the years, you can find a vast world of ways to piece shots together that convey connections between segments of moving picture that are more symbolic or thematic or sometimes narrative but in a more abstract way — not as bound to visual, physical spatial continuity.

VR cinema has inherently greater replay value than traditional cinema, because you can only look one direction at a time. Immersive cinema authors can — if they choose to — create experiences that can’t be taken in in their entirety the first time through. Even if you make something that, unlike a game is always 10 minutes long, it’s always the same 10 minutes of stuff each time, you can’t even affect the outcome… just take 360 spherical degrees and divide by your field of view: that’s an objective minimum of views just to literally see all of the footage.

There’s another endlessly fascinating dimension a potential form of media could go down. The amount of feedback the creators of television shows already get these days between episodes and seasons… the influence is strong enough over episodic content that the audience can steer the direction of the show. This is already happening. But it could even be: it is the same episode over and over, but over time the people who watch it are able to affect it, make it into something new. How is this different from, say, opening a bar? I might have had some idea of what vibe I wanted the place to be, but it’s defined over time by who chooses to come to it. Maybe I don’t want to predetermine what it means, maybe I just wanted to make a template that people could take collectively and make something out of it. There could be a version history of the episode. ‘My favorite version of this episode is 3.2’. This is not too dissimilar with what’s going on with reboots. The tools are just going get easier and easier for the populace to use; people don’t bat an eye at meme generators. Don’t underestimate the power of fans: just look at Bartkira.

For me, the biggest hurdle for immersive cinema is live action capture technology. Until someone gets light field capture down for real, I’m not really interested in going for it. Look at Google Jump. Amazing, but I don’t think even they’ve quite got it. There are still some artifacts at least from what we can tell from the footage they shared at I/O. I think their technology is perfectly suited toward sharing experiences from friend to friend in casual use, but it’s not cinematic level yet. We’ve seen perfect-o still light field capture with Otoy’s crazy thingamigger spinning the camera all around, and maybe you could suture that together with whatever their holographic stages are called, piece together an actor’s performance, copy-paste them into a captured still environment to simulate live action, such that you can be there and get that parallax. It’s just not worth it with only the stereoscopic depth cues, though Felix and Paul’s Cirque du Soleil experience is nonetheless impressive. Until light field works, we’ve got animation.

 

CM:

Will light field technology replace cameras?

 

DB:

You mean will Lytro replace DSLRs? People still listen to the radio. Old forms remain. Niches will be into it forever.

 

CM:

What do you think about big studios? What do you think about their approach to immersive cinema storytelling?

 

DB:

I feel gratitude and suspicion. I’m grateful that some really creative and thoughtful people have got the money to already be exploring the real nuances and be pioneering this stuff. All power to them. It’s great. I’m sure, though, that given their context they’ll be striving toward achieving something which ultimately I’m not that into personally, as more of an experimental wacko type guy. I mean they’re probably going to be going straight for the heart strings, and the horror, and the mindfuck type stuff. I don’t know. I can’t wait until this technology is truly democratized in the way we’re starting to see traditional film be.

 

CM:

I’m more excited about what other humans out there have to share than Disney. Somewhere out there there’s an Alejandro Jodorowsky of VR waiting for the power to unleash their vision.

 

DB:

If the ultimate goal is to bring all the people, all the consciousnesses on the planet closer together, we don’t need more white dudes in capes with hammers jumping around. We need more of this handing out laptops and cameras to people in the third world and seeing what they make. Once they have 360 cameras, too, then literally anyone is anywhere anytime. And that’s the dream.

 

CM:

It’s a dream, certainly.

 

DB:

Yeah, what am I saying. No one has “the” dream.

 

CM:

Are there downsides to this level of connectivity, though? How will the individual, the self, remain comfortable with being itself while at the same time having this flood of input from the masses? How will that balance as technology ramps?

 

DB:

Well keep in mind the opposite effect, where technology helps us reinforce our beliefs about ourselves more easily. Even things like our Facebook News Feed where I feel more connected to everyone around me than I ever was before, and to things happening all over the planet. But when you think about it you’re not just getting a flat feed of anything and everything. A corporation is using the most advanced techniques they can to figure out and give you exactly what you want to hear. And that’s only going to get stronger and stronger. If that is what you want — to massage the patterns that are already forming in your consciousness — that path will be available to you. We’ll have a world of superficially connected but even more separate entities. I’m not saying that’s a nightmare either, maybe that’s cool and fine, but it’s definitely a thing to think about.

 

CM:

Would you want to live a slave to pleasure like that?

 

DB:

Hard to say. I think there’s a lot more possible than that. There’s nothing wrong with euphoria… maybe we should question whether peace is the ultimate goal, or whether preserving strife and risk would be better. You can’t create something without destroying something, if only the absence of the thing you created. Growth is violent.

 

CM:

Can you have peace for a species like ours without the self? Do we have to remove aspects of the self like ambition, greed, and whatever comes out from there, is that still human, even if we have peace. Should we remove these things?

 

DB:

I would like to think we could have both. A transcendence while preserving what we arose from. I think that greed and conflict and suffering even are interesting things. I don’t know why this word is coming to mind, but ‘zoo’. Almost as if humanity could be put in a zoo of sorts where we’re observing what we used to be for our continued pleasure and inspiration but exist apart from it. Putting it as a “zoo” sounds awful. Zoos are immoral institutions in my opinion… there’s got to be a better word to describe what I’m going for here… but if humans could be allowed to keep reinforcing their own individualities but those individualities were interesting and worthwhile to the peaceful being that they had all combined into, then that sounds like the best of both worlds to me.

I guess what I was going for here was more like a preservation as we preserve a species rather than force it to extinction. We can’t halt certain partially out of control processes of overwhelming change we kick into motion, but we can do little things to curtail the completeness and permanence of the violent outfall of the result. No one may ever write poetry again in a language saved just sort of extinction, but as long as some people still speak it… or as long as events are recorded in history books, at least they weren’t forgotten. That’s maybe the best we can hope for humanity itself in the face of transcendence.

 

CM:

How could we possibly get there, considering the state of the internet now?

 

DB:

If we can make it past all the major existential crises (such as irrevocably ruining our environment) and solve our critical social issues. We might be too stupid, warlike, genocidal to make it these things happen, sure. There’s no need for a non-scarcity economy anymore, where the richest 1% have 90% of the wealth and everyone else has nothing; we should be able to have robots harvesting all the food we need to keep our human bodies alive so we can all focus on whatever we’re most interested in becoming, exploring our own unique snowflake blinding rainbowy crystals of light, and join in some crazy shared VR where we can exchange those things…

 

CM:

It will get to a point where robots do all that. What are humans left to do? We as individual could figure out what it is that gives us meaning and purpose. Maybe the journey to figure that out is the thing. It’s individual, it’s up to you. But what we could do as a collective mass is party, party to the ends of the universe, because what else is there to do really once we’ve achieved all that? This is the future of humanity once we cross the singularity. It’ll be us, VR, and AI navigating, exploring, our selves, collectively partying, holding giant Burning Mans all over the world.

 

DB:

A lot of people agree with this.

 

CM:

There’s so much that could go wrong though! I’m holding onto this happy vision of the future.

 

DB:

I probably shouldn’t tell you to read Eliezer Yudkowsky’s fun theory. Is fun in the universe exhaustible? The way he defines fun is novel experiences to be had, problems to be solved in a more mathematical logical sense. Is it possible we could figure everything out and get bored? The party could end, the booze could run out.

 

CM:

Goddamnit. We partied from every possible angle we could have.

 

DB:

It’s theoretically possible.

 

CM:

We can’t discount that possibility.

 

DB:

Well, we’ll cross that bridge when we get there.

 

CM:

What’s the best case scenario for you?

 

DB:

That AI makes some really cool art. That it helps us find ourselves.*

Some of my favorite experiences with AI have been neural networks trying to simulate human behavior and writing. Like What Would I Say. You link your social media accounts, it analyzes everything you’ve ever said, and it generates novel things that you might say, using a Markov chain or something similar I’d think, considering each word individually and then statistically what are the most likely words you say after each other one, building up phrases from there. I’m drastically oversimplifying this but it’s the basic principle behind things like this and it’s hilarious. It’s uncanny, sometimes uncanny Engrishy humor, other times because of how accurate it is. These things are getting better and better, you can train them, like Google’s Deep Dream. People have been working on generative artwork for decades. Generate novel works by Mozart that confuse the heck out of experts. That’s childsplay though. What if we could teach them the principles that inform great works of art at a high enough level that they could create things that would inspire and provoke us. I feel like this is full circle to your fears of an AI clone of you, though, maybe.

 

CM:

Let me play the devil’s advocate here because you’re saying that art can be formulaic. Isn’t there something deeper, something that is innately human that comes out of nowhere. But with AI can’t we just tell where it’s coming from if we dig deeper into the code?

 

DB:

Ultimately humans are machines too, just really poorly understood ones. I’m not saying we’ll understand ourselves completely one day as much as the opposite, that we’ll raise machines to the level of complexity where their creative impetus will be encrypted beyond our ken as well. Are machines going to have souls? Pretty much the same concern. It’s going to be easier for machines to make some sort of inspiring and valuable art than it will be for them to pass the Turing test in conversation. Art’s just more open to interpretation and abstract while the rules governing conversation are so much more specific.

 

CM:

Are you banking on AI and VR?

 

DB:

Subconsciously at least I think I make decisions that way. I broke my wrist recently and haven’t been taking as good care of it as I would probably if I didn’t believe on some level that I wasn’t really going to need it when I’m 80 in the same way as my great grandparents do. Much to the chagrin of those closest to me, sorry about this.

Perhaps as an artist I’m realizing more and more that I’m not as interested in creating artists. What I mean is: I think there are a lot of technical people out there interested not creating films and music and games directly but instead focusing on the parameters and constraints, the rules that govern an experience, and setting a machine out to come up with different presentations of those inputs. That’s creating an artist as art. Or turning it over to a combination of humans and AIs to create those experiences. You could say this is already happening with video games: developers came up with the rules of StarCraft and they wrote NPCs that obey those rules and put them out in the world with a bunch of humans and they compose a metagame together. And that metagame is something fascinating and beautiful which they didn’t create but they knew they were going to form the substrate for. This is really similar example to what I said earlier about creating a bar. As AIs become closer and closer to the abilities humans have, these two interests are going to become more and more similar. So to differentiate myself from that approach to introducing art into the world, I’m preparing to focus more on looking inwardly just on myself. I don’t mean in a crazy egocentric way. I’m just more interested in this: when I sit down with a set of intentions and constraints, what do I come up with? Even when it’s virtual psychedelics where the actual individual audience reaction is divergent, the aura of the work is quite solid. And I want to believe that that’s going to remain interesting to other people (whether they be human or artificially intelligent themselves) — that non-collective works of art will continue to hold the attention.

 

CM:

What do you have in the works for the future? From your melting pot of ideas? How do you intend on bringing forth your self-expression?

 

DB:

I recently had a Tarot reading, and the dilemma that I posed to the reader was: I feel like I want to have the greatest possible positive impact on the world and I’m worried that I’m deluding myself when I say that the best way for me to do that is to focus on cultivating my self-understanding and my powers of articulating my personal experience and sharing those with the world. Maybe if what I really want is to help others that that’s bullshit, I’m not worth my weight in art, I should just get out there and end wars and hunger and genocide. Just directly help the people that need it the most in obvious ways. Because I don’t do nearly enough of that.

I read this interesting article coming out of the Singularity Institute about this concept called “sins of the future“. This is the idea. We look back on those who actively or passively were part of the slavery industry in United States history very poorly. We condemn those people. For a lot of those people, it wasn’t a big issue in their lives, it was just the time and place they were born into. It’s not like they were hateful people necessarily. So we live one of the wealthiest countries in the world. The economy that supports our lifestyle is built upon global slavery. Mass incarceration. Maybe people will look back on us just being okay with having comfortable lives while millions are in abject misery. They’ll think we’re awful people.

I feel awful about this, we all feel awful about this. We want to do what we can and are challenging ourselves to find the best ways to do it. But what is the best way? I don’t think the answer is black and white. We all should do more to give directly, but also think about how individually we best can make the wider and more long term and profound impact.

 

CM:

Is this where AI is going to step in and be a benevolent adviser?

 

DB:

I’ve been able to impose accountability on myself using technology, able to externalize pressure to meet personal goals. It’d be a dream if we could do a similar thing with AI on a societal level, externalize accountability as a species.

 

CM:

Or psychedelics! It’s not in the realm of impossibility that we’ll figure finally that this is medicine. It could be used for therapy. If we had the research, knowledge, and mentality, we could make amazing breakthroughs. People need to breathe more; the world we be much better off.

 

DB:

What you’re tapping into here is a really important continuum, similar to the one we were talking about earlier with the new lines people are going to have to draw in romantic relationships — what’s another person, what’s infidelity: thought policing. Once we have things beyond psychoactive drugs that merely affect us on a physical chemical level, but we have mainlining type technologies like the artistic virtual drugs I was dreaming of creating one day, I hope that our power structures will be able to accommodate the prerequisite great amounts of freedom in the types of conscious experience you’ll be able to have.

 

*I hadn’t read Dave Egger’s The Circle yet when I did this interview, but this exact phrase figures disconcertingly in it.