David Tree, Hertfordshire University

David Treat, University of Hertfordshire Law, Most of you will recognize this is Keanu Reeves' old role in the 1990s as Johnny Mnemonic, which is possibly one of the best realizations of what virtual reality is going to be like in the future, back in the 90s, when you only had 3 megabytes of memory in your graphics card and you couldn't barely visualize a three-dimensional object. I'm trying to do the same thing in two places, and that's not going to work. So, what is virtual reality? The first thing I'm going to say is it has nothing to do with computers. Virtual reality, you're here. You're in a virtual reality already, because actually your brain does not have a way of seeing the world. Your brain lives inside a liquidy soup, inside a bone encapsulation, so it can't see the real world, so actually everything you experience is virtual. I'll just let that sink in a moment. So, actually you're already... You're already in a virtual reality. What we like to do with our virtual realities, what we now call virtual realities, is we like to interfere with those senses and change them so that what you see is not what is in the real or physical world, but so that you see something that someone else has created. But then you could equally say that actually a painter creates a virtual world because they paint a painting of something that isn't there, and you interpret it, and it stimulates your mind, and it stimulates your mind in a different way. So when you look at one of Dali's paintings of the drooping world, you're looking at a virtual environment. In that case, it's something that doesn't exist. It fills all of the boxes, if you're saying, how do I tell if it's virtual or not? Does it exist in the real world? Does it conform to usual physics and the usual rules? No, no, no. Oh, it must be a virtual world. No, it's a painting. So actually, these are compatible topics already. So I'm just catching up on my notes here. So that's the first thing, just to set you there. The other thing I'd say is that virtual reality isn't just about head-mounted displays. It's not about just wearing a headset. It is about poking your senses in an interesting way. And the definition I've come up with that I quite like is that virtual reality is a synthetic simulation of senses to replicate stimuli that exist in the physical world. So that's the kind of, I like a nice big wide definition because you could have a virtual olfactory world in which you close your eyes and you smell all kinds of different smells. And someone could say to you, imagine you're in this place and you smell it and you go, oh, I can smell the coffee. I'm in a coffee shop. I can smell the sweat. I'm in the sports room. You can imagine being in these virtual places without actually being there. Just by stimulating the senses in the right way. So those are options there. However, that aside, my medium is the visual. So, and we've got another, this is where we've gone from. So this is absolutely what virtual reality isn't. That aside, I come from a visual medium. I work in the School of Creative Arts. We like to make pretty things. And we're looking to explore the world. And we're looking to explore how we work with virtual reality. The difficulties of virtual reality at the moment is actually, there is a lot of different headsets. Although there's five general ones. You know, you've got your PlayStation, your Oculus, your Vive, Google Cardboard and FOV. And Google Cardboard does cover kind of 90% of the 370-odd models of virtual reality headset at the moment. There's a lot out there. And it's a lot. And it's suffering from the same problem that the whole Betamax VHS era format was in that you have to kind of pick. And the problem we've got at the moment is that it's, as always, it's the consumer that has to suffer. So the consumer has to put their money where their mouth is and choose one and hope they get the right one. Which is not necessarily ideal when we're trying to tell people there's this wonderful new medium that's going to completely change everything but we're not going to tell you what thing to actually view it on. Because we don't really know what works best. Because they all work in different ways and they all have different advantages. My personal favorite is the HTC Vive. But that's because it just does a few of the things that I like a bit better than the other ones. Actually, a lot of them are very similar, the screen resolution. Some of them even have the same screen in them. But people will say, oh, that one's much better resolution than this one. You're like, you do realize it's the same part you're looking at. So, again, it's in your head. So you've got your format wards. The other big problem you've got is because it's such a, like most of the digital industry, it's a massive, fast-moving behemoth of an industry. So you've got this whole problem of, well, it might take me six months to develop an application for this platform. But in six months' time, two revisions of the platform have changed and now it doesn't work anymore. So I could have just thrown a load of money into production and it won't work at the other end. So we've got to think about multi -platform and targeting our applications so they work in a general sense, which makes it difficult for us to actually take advantage of the advantages of some of the headsets. So until recently, only the Vive had motion controllers where you could actually move your hands around. Oculus took a little while to catch up with that one. Vive equally was the first one to have room scale. But, again, each of the companies are catching up now. So it might just be we're too early to the game, but it's great that creatives are getting in at the beginning. So unlike most technological revolutions, it stays in science labs and gets hidden away for years and years and years. Whereas with virtual reality, it was the creatives who got in there at the beginning. And thanks to the work done by Parmalucky and Oculus, they made a big deal about the content, the actual stuff that you view on it. Because at the end of the day, it's all very well and good, and you're having a 900-pound piece of glassware in front of your face, but unless there's anything on it, it's just a nice-looking pair of blindfolds, really. And it may give you a virtual experience having it turned off and looking at a blank screen, but it's probably not what you were intending when you spent 900 pounds on it. So that's that. So there's this whole price thing. So you've got the issue of some of the headsets are 10 quid, and they might give you an experience, but they're never going to give you the same experience as the high-end headset because it doesn't have the tracking or it doesn't have the same field of view or the same quality of the lenses. So we have to do a bit from a customer service standpoint, we have to do a bit of setting expectations for people. So we need to tell people, yeah, it's great, you can have a cardboard or a Daydream or something like that, but don't expect it to be like, it's kind of like someone getting into a car. A small Fiat and then expecting it to be a Ferrari. It's the same problem. You need to make sure that people understand they're buying the Fiat, they're not buying a Ferrari, and then you should be okay. So these aren't new challenges for us to be aware of. The other thing, of course, is that virtual reality is an immersive medium. It completely envelops your world. Once you're in it, you're stuck in it unless you can remove the headset. So we have to be very careful as content creators about the physiology and make sure that we're not making people ill or unwell. So we have a responsibility as creators to look after people. So we need to make sure that we're displaying things at the right frame rates, we're not transporting someone from here to here. And, you know, VR is a completely no-holds-barred medium. If you can imagine, I always say this because it's one of my favorite things, back in the day when they started developing CG, they said the best thing is if you can imagine it, you can make it. And that's true with virtual reality, and you can make it, but there is definitely a question of should you and should you do it in that way, because you've got to be careful about your audience. You don't want to pop anyone off. So as everybody today has said, so I will say it very quickly, there is two different formats for virtual reality. There is 360 video and there is interactive real-time virtual reality. I will say explicitly these are two completely separate mediums. They may be viewed on the same headset, but it would be like me saying to you, TV and film are the same mediums. You can view them on the same screen, but actually they're very different. 360 video has its advantages. It's great for capturing real world. It's great for capturing what's going on. It's cheap. It's really quick to make. And it's very accessible, because most people can do a point and click with a camera given the opportunity. I mean, the advantage with virtual reality is even if your grandma does put a finger over one of the lenses, there's probably another five lenses to take the video. So you do have some higher chances of getting something out of it. It has a lot quicker results than virtual reality, and it's a lot cheaper to produce. Sorry, not virtual reality, but interactive virtual reality. The downfall of it is it is limited interactivity. So at the moment with the formats that are currently available, apart from the stuff that Facebook released last night, at the moment if you put a 360 video, you are a fixed point in space. And you're constrained to that. So unless you move the camera, in which case you've got to be careful of making people seasick. But you are fixed in space. You can't move it once you're in there. It's kind of like going to the planetarium. You have a fixed viewpoint on the world. So that is your 360 video. Also, stereoscopy doesn't really work 100%. Even if you have a stereoscopic image like this, where you've got two different viewpoints, because the way it's actually produced and the way you consume it, you can't actually get true stereoscopy because when you render this image, you render two orbs in space and you capture those locations. Now, it's fine if you're looking forward, but as soon as I turn around, I've swapped those eyes over in the real world. But in the virtual world, it's still sampling the wrong point. So something in front of me may appear to move when it shouldn't. So stereoscopy is not quite right. The other difficulty is, as with all of the formats, is that 360 video also has three different ways of doing it. Sometimes it's above and below. Sometimes it's side and side. Sometimes it's chroma. It's red and green and that kind of thing. So again, we haven't got to a unified format yet. So we're still working on that. The next one, my personal favorite, is games engines. The slight downfall that a few people have had is that they think, oh, great, well, virtual reality, this games engine's got its name, it's named all over it. But the downfall of this is games artists are brilliant at making video games. They don't quite necessarily understand experience design and don't necessarily understand the rules of not scaring someone literally to death. So we do have to be careful. They may understand how to make games look like something fun in a video game, but actually making it work in a virtual world, which is a separate thing altogether, is slightly different. So although we've got the right tools, we might not have the right mindset. And that's the issue for games designers to work out. The great advantage with this is we can have massive worlds. I mean, currently the Unreal game engine, which we use at the research lab, you can generate a five kilometer by five kilometer square and you can explore every inch of it. So you could theoretically grab a mountain and stick it in the virtual reality and climb the mountain. There is enough space there for you to do that. So we have this massive scale. And that's when you only limit it to one map. When you start looking at things like spatial OS, where they're looking at massively distributed virtual worlds, where they're using masses of cloud servers to allow you to hop from world to world to world, much like in some of the, the virtual reality books and films about at the moment, you could theoretically have an infinite world that would not stop, especially when you consider things like, I can't remember the name of it, Something Sky. There was a video game which didn't go down particularly well because it didn't actually happen how it was meant to. But it was meant to be a fully procedural virtual game where there was millions of planets and every planet had a different ecosystem that was procedurally generated, Dead Man's Sky. But it didn't come to fruition, unfortunately. But definitely procedural generation is possibly where we're at because there's a point where we can't make the content anymore. We don't have enough humans in content creation to make all the stuff that we need to fill our virtual worlds. So we need to consider procedural generation as well. And unlike 360 video, you get accurate depth perception, correct stereoscopy. You can put your face to the floor and it will feel like your face is at the floor. You can see things accurately. And this comes to the final point on that one, which is interaction. The big selling point of the real-time experience is that it's interactive. It's not just sitting passively watching a screen. It is a hot medium. We get to actually get our fingers in. We can change the world around us. Now, this is not the case just in the sense of picking things up and throwing them around. But we need to consider as well how we deal with narrative realism. So the ability to change the storyline based on the actions I have. If I decide that I don't particularly like this key character of your game and decide to deal with him and lock him in a cupboard for the rest of the game, how does the world react? Does it mean game over like it used to? Or is it not fair to say game over anymore? Do we just carry on and see where the world goes? It might mean that I spend the rest of my video game planting the garden and mowing the lawn. But we need to think about actual realism in the gameplay sense and narratives. But slight digression from interaction. The exciting thing about interaction at the moment is we've finally got the point where everyone said, right, okay, we kind of get these virtual reality headsets. That makes sense. We're working on that. What can we do with interaction? There's been some great techniques from Leap Motion with being able to put your hands in front of your face and they're now doing a mobile version, which means that actually you could have a VR headset that's got everything in it and you put your hands out to actually interact with the world much like you would in Minority Report. But if you want to go one step further, there's going to be these suits, which are as pictured here. This is the Tesla suit that's going to be coming out in Q3, Q3, Q4 this year. And they've actually taken the muscle stimulation pads that you use for exercise and to make your muscles stronger and they've implanted them in a suit so that they can actually tap you on the arm and give you impulses and feel things. Also, they've put a full motion capture system into it so that you can walk around that virtual world, which leads me to my next point on interaction. It's all very nice and well and good and having my viewpoint at a fixed location, but not many people spend their entire life rooted to the spot, unless you're a snowboarder, in which case you move the board, but you don't tend to stand there with your feet stuck together. So how do we make the reality more real? Then, well, we need to be able to allow you to walk around, which is where things like room scale tracking come in and these virtual suits, which means that you can actually walk around the place. This isn't a foregone conclusion yet. We don't know how we're going to do it yet. It might be through these virtual suits. It might be through room scale. It might be through teleporters, which has been tried, because at some point you're going to say, well, I'm not going to be able to have a five-kilometer by five-kilometer room that's empty of anything so that I can experience my virtual world. There has been omnidirectional treadmills, apart from the shipping concerns, of them being too big and hard to move. They generally do the job. So this is a couple of case studies of things beyond the headset. That's what I'm trying to get at here. So the first one is a 360-degree video, which is around using virtual reality to teach us how if you can press play, please. So the first video here is from Autism TMI, and it's a 360-video experience. And what they're trying to get at here is they're trying to get at this whole point of view business. So this is an empathy machine. The idea is that they are showing you what life would be like from the perception of someone with autism so that people would better understand the kind of strains and stresses they go through. In this case, it's in a shopping center. And they're trying to display to us what visual and auditory sensory overload feel like to the average person so that we can understand why someone might be having a breakdown because of nothing in particular. And it's a great example in this. I didn't realize how much lights could be a problem. And you thought, well, it's quite nicely lit. But then we've got a chap with a bag or someone with a mop here. And this is enough stimulation to go beyond comfortable stimulation. So this person also illustrates the tunnel vision effect which sort of leads from panic attacks. And this is sort of something that was the way I see the point of view machine is like Douglas Adams came up with in Hitchhiker's Guide to the Galaxy that the most important thing for the intergalactic consortium of angry housewives was that most arguments generally end up with you saying, you just don't get it, do you? You just don't understand my point of view. So what if we could put someone in that point of view? That's what the point of view is. That's what the point of view gun did. And that's kind of what this sort of virtual reality thing does. It means that I can actually tell you exactly how I feel. I've always wanted to express to someone what it's like to wear glasses because I've worn glasses my whole life. And it's very frustrating when someone goes, what's the big deal? It's so frustrating when you get in the shower and you can't tell the difference between the shampoo and the conditioner because you can't read the bottle because they don't work so well. So it's the same thing. It's being able to understand someone's point of view is where the great strength is here because you can be in that different experience. My second case study is, if you just want to hit play because it's a long video, I'll start on it. So this is something that was developed by Lockheed Martin and Frame Store, a large visual effects house in London, although I think this is the American branch of that company. So what they were trying to do was they were trying to address the problem of group experiences. So one of the things virtual reality does very well is loners who love to play video games and sit in their bedrooms and not go out, it gives them another excuse to not go out and sit in their bedroom and play video games. One thing it doesn't do is it doesn't take them outside of that environment. It doesn't bring people together so much in the physical sense. It enables you to virtually be together, but it doesn't actually physically bring you together. So they were looking at how they could apply virtual reality to something a bit more than goggles. They effectively turned a school bus into a giant virtual reality environment by putting OLED displays on the windows, transparent ones, so the kids couldn't actually tell that it was anything more than a normal bus until they flicked the switch inside, which was a brilliant touch. And then what they did is they tracked the kids They turned it into a giant motion controller and they mapped Washington DC onto the surface of Mars and then played them around this giant, I can't remember how many square miles, I think it was 250 square kilometers space that they developed. And so the idea being that you could leave from any school in the whole city, go on a drive around the place, and the kids would have a great field trip and just looking out the window. This is their prototype that they were developing, talking about the motion tracking and mapping the surface on, mapping the streets on, because that's how they worked out where to put everything, and they're using a cunning laser tracker. It's easily found online if you want to listen to the audio. It's just they talk through the process quite a lot. This is based on Unreal Engine 4 again. So it's just an example again of how it doesn't need to be a house, And the great thing with this is that they can put 40 kids in a bus and they can all go to Mars together. And like Bertie was saying earlier, the great thing about this is that they're forming memories. It's not just, oh, well, we watched a video about Mars. It was, we went to Mars in the school bus. That was kind of cool. So it's a great example of how we can use this medium beyond our headsets and take it to, and put some real motion on it. I need to just move forward on. So these are the giant displays. They're retrofit them onto the bus, but I'll have to move on because he talks for a long time. If you can hit play on this one as well. So that's group experience. That was Empathy Machine. This one here, one of the big things that bugged many people about these little devices that we now carry around is actually this was more powerful than the first machine I did any content creation on. But can you make anything on it? Not a great deal. And it's just terribly frustrating for a creative person to have a really powerful computer in your pocket but not actually be able to make anything on it. So what was great about VR was that it lasted about six months where we couldn't make anything in VR. And now games engines, both Unity and Unreal and a few others have already said, you know what? Why don't we build worlds in VR for VR? So what didn't make sense was that, you know, when we make films, we, you know, the big inspiration that they had a few years back was, well, let's stop looking through the lens of the camera. Let's have a big plasma screen so I can see how it will look on a plasma screen when I'm shooting. And they're like, well, yeah, that kind of makes sense really. You know? That's how people are going to view it. So let's view it while we're making it. With apps, you always, you can simulate it on the computer, but anyone who really wants to test it puts it on the device and actually goes, does the touch screen work? Does this work for me? And virtual reality for a time, we were doing it in games engines. So we were using a flat screen and we were trying to have three or four displays so we could show enough information for us to understand this four-dimensional world of the screen and try and build it. But we were having the same kind of problems. We're like, well, how can I interpret all of these dimensions if I'm not actually there? And you can tell if someone has never tried their experience in a headset because especially if they come to the end of the project and they go, oh, yeah, well, we built it all in the engine and then we rented the headset just for the show. And then you're like, yeah, and it didn't work because actually your eye line's at the wrong point. You can't see something that you should be able to see. You can't interact the way you wanted to because theoretically it works, but they haven't actually tried it. So one of the great things about actually using a VR headset for building it is you firstly get quick iteration because you don't have to keep putting the headset on, taking it off, putting it on, taking it off, which again for someone with glasses is very annoying because you have to take your glasses off, put the headset on, take your headset off. Put the glasses back on and it drives you up the wall after a while. So being able to have a headset where you actually build in, it brings a whole new problem for any content creation companies because you can no longer stack your 3D artists in stacks by the wall. You need to actually give them a reasonable amount of space so they can spread their arms and build their environment. It also means that people who in non-traditional, in non-games disciplines, can actually start working in the creative world, in the virtual world. So what's been very interesting over the last few months, I've been spending a lot of time talking to architects and talking to sculptors who would not traditionally be in my medium, but all of a sudden, that's great, I want that for me. And this video here is actually a demo from Ford where they've built, they're using, virtual reality to allow them to design cars before they actually build them. And one of the really interesting things here is that this wand that she's holding is meant to be simulating a torch because that's how they would analyse the curves of the surface when you're in the car to sort of see how the materials act up and whether the sculpt is right, does it feel right. So they're enabling them to use the same tools in the virtual world to create a real model. But their great advantage is they don't actually have to build a car if it doesn't work, they just change it in the virtual world. So, big change there. So, the future. Some people would say we're in the future because if you go to Star Trek as reference, he tapped his comm badge, there's a Bluetooth headset right there, you use your tricorder, which you could do all kinds of spectral analysis and all of this kind of business. You could move around very quickly and we have cars and we can do all these things. So, we are in the future. I can theoretically, there was a great art piece a while ago that enabled people in New York to see London. So they set up a tunnel and they said, oh, this is the Great Atlantic Tunnel but what they've done is they've got two projectors and two cameras and linked the two together. So, people in New York could look through the tunnel and see London and people in London could look through the tunnel and see New York. We're in that place because actually I could put my virtual reality headset on and someone somewhere else could put their virtual reality headset on and we can be in the same room together. And that's something HoloLens have been working on. Their suggestion was that, you know, daddy can be away working and the kids can be at home but they can actually be in the same room and see a hologram of the other party there. So, we've got that global village. We can both be in the same place. So, what's the future? There's two things I want to cover. There's kind of the future of the technology and there's the future of the creative industry with it. Future technologically, this image here is Lytro's light field camera. So, at the moment, the piece of silicon we're looking at in front of our eyes is actually not much developed from the same LCD displays that we used back in the 90s to run your 66 megahertz compact laptop. Other than the density, the brightness, the backlight, it's still a grid of pixels that go on and off to make a picture. It's very nice and we've made them really pretty and they're nice and dense now and we can get really pretty colors. But what it doesn't do is it doesn't replicate the way that light works in the real world. In the real world, light doesn't hit us parallel. So, if you think, probably, if you've ever used one of those Play-Doh extruders, it's probably the best example, if you put that in there and you pull the extruder down, you get all of this stuff coming out, but it's all in straight lines. But light doesn't work like that. Light comes from the light source, bounces off a surface, bounces into your eye at all kinds of different angles. And that means that when I look at someone close to me, my eyes focus on the light rays coming from that close-to-me object and automatically blur everything around it. Now, in cinematography, we've been doing this for years, we use depth of field. We focus on the thing in front of us and blur out everything behind to give you that sense of depth. But what light field displays will enable us to do is they will actually enable us to focus on the image after it's been created. So, the way that they do that at the moment is they create a grid of 16 by 16 little images of every different viewpoint. That's why this thing has a mass of cameras, because it's actually got a whole grid of cameras and it's recording all these different viewpoints so that when I view it back, I can focus on the thing in front of me or the thing in the distance. The advantage for this is it's going to solve a lot of our cyber sickness problems because it means that you get natural depth of field and natural focusing, so it also means you don't get eye strain like you do looking at a monitor. So that's a very exciting field. Stanford have made a prototype two years ago, so we're hopefully not far off at actually hitting a real live thing. This camera here is slightly bigger than the CG rendering, which was something that was about a foot in diameter. So they are still working on miniaturizing this tech. And the current challenge is that in the real time world, we would have to render 64 images instead of one, which has a slight graphical implication on your graphics card. But Moore's law complying, we should be able to get there within four years. We should have the technology to actually render this stuff at a suitable speed. The other advantage is that, another thing that's happening in the background that Disney Research has published lately was the Universal Scene Descriptor, which for the creatives means that you've been building all this stuff for film and visual effects for years. What you really needed was a big export to virtual reality button. This Universal Descriptor is going to enable you to do that. It's going to enable you to move from a VFX pipeline and converge it into a games pipeline. Or virtual reality pipeline. That would probably be the other change, is that we've stopped referring to real time engines as game engines and maybe call them reality engines. I don't know. And my final slide. Well, second to final slide. Exciting new film that anyone should watch that's coming out soon. Ready Player One, which predicts of a future world where actually nobody goes out of their, goes out of their house. When you go to school, you go downstairs, you go into your VR room, you don your headset and you go to school. Socially, this means that it doesn't matter where you are, you could go to the best school, which regardless of your physical location, which has some good things. It does have some slight concerning things about not going outside ever again. It also means that we really need to work on this social realism business within engines. We need to think about how we get those narratives to work together so that I can actually be with you in a virtual world together. And when I change something, it actually is persistent. I can't just press undo because in the real world that doesn't happen. Other developments that are already happening. The great place to look for this, to find out what's going to happen in 20 years time is to look at what's happening in Japan now. And they're already building virtual reality arcades to solve the whole, how can I afford a 900 pound headset? Well, you don't. You go and rent it. You go and use a space, visit the experience. But that doesn't work if, when we're looking at this kind of idea that it needs to be accessible for everyone because there is going to be a greater digital divide of I've got access to the entire, information of the world. Whereas you've only got access to what you can view on your website, which may be considered to be old fashioned tech by then. The other thing they're looking at is selling smaller flats, but saying, well, it has a really good broadband connection. So you don't need a big flat because you've got great broadband and we'll fit your headset for you. And hotels are doing that as well. They're saying, well, it may be a small room, but it comes with VR. So, and there's a real question there of the kind of humanity of it. We need to obviously consider about whether we want to spend all our lives in a small box looking at a screen, in which case, which pill? So do you want to go down the rabbit hole or do you want to forget about all this and start again? So any questions? Thank you very much, David. Just a couple of things.