Rolf Nordahl - Aalborg University at CopenX, September 2016

I am Rolf Nordahl and in Danish, Rolf Nordahl. I'm head of the lab that we see here, the multi-sensory experience lab. And while we belong to Aalborg University, it's actually Aalborg University here in Copenhagen. So we are placed out in the South Harbour here in Copenhagen. And I would assume that we are sort of belong to the cluster of the biggest VR centers here in Northern Europe. And when I say we, it's because we have quite a lot of people connected to our research cluster and to our lab, ranging from assistant professors, or to PhD students, postdoctoral students and so on. I have a range of things that I would like to get across as much as this today. All pertaining to this thing about VR, this brand new experience that we have in VR. And then of course when I say brand new, I mean something that is pretty much 50 years old. So, but to head off the talk, I would like to show a small movie clip. I'm going back to VSI to complete the final stage of my evolution. I'm going to project myself into the mainframe computer. How we come? Pure energy. Once I've entered in the neural net, my birth cry will be the sound of every phone on this planet ringing in unison. Listen to what you say. The first sound of psychosis is a Christ-pumped-lap-sound. Sibber-pressed. Amen! The system overloaded. Automatic shutdown in Albany. Oh my God, John. I can't believe this. Outside terminal activity control. No! Now to get out of here. Simple combination. Access denied. Access denied. What is this? What do we want? What are we going to do? What are we going to do? What are we going to do? We're going to die? We're going to die? We're going to die? Yeah, that was the lawnmower man. It was a dream of the promising land of virtual reality as it looked like in the late 80s, start 90s. And that was sort of, it belonged to my youth, this dream of what would be technological possible. It was a brave new world and it really flopped. Because back then, we first of all didn't have enough knowledge about what was going on. There simply wasn't enough computer power, processing power, to make these things come true. But we are somewhere else today. Now, they were having a very fancy lab in this clip. It looked very good. That's not how usually labs look like when I travel to other parts of the world and visit labs. Or indeed my own lab that you see a little part of here. But we have a lot of equipment in our lab. We have a lot of possibilities. I'm going to come back to that during my talk and show some different examples of what is, what that we do. Because many of the things that we do, they are not sort of what you see in the market right now. What we are focusing on is what is sort of the next steps. How will things look like in five years and ten years and so on. And we try to develop technology that you will use in those five to ten years. If you look at this slide, many of you will probably know the fellow on the right, Parmologi. Maybe not so many of you will know the gentleman on the left. And that is Ivan Sutherland. However, he's sort of the reason that we are all coming together today at this expo. He's sort of the grandfather or the godfather of virtual reality as we know it. He was one of the first people who showed an interest in and tried to develop the concept of virtual reality. And developed actually the very things like HMDs that we use all around the world today. And why not take advantage of all the theory, all the experiments that has been going on for the past 50 years. Actually, when we are talking about virtual reality, we are not only talking about 50 years back. We are using theories, knowledge that stemmed back from 1860, from the field of attention research. That might come as a surprise for some of you, but it's quite a vast literature that we have access to. Now, Ivan Sutherland, in 1965, he started to work on the ultimate display. He has done extensive work on tracking, head-mounted displays and graphic processors. Later in his career, in the 80s, he would become part owner of what was known as Sun Microsystems, developing quite powerful machinery. I have a couple of quotes that I would like to spend a little time on. Like this one. A display connected to a digital computer gives us a chance to gain familiarity with concepts not realizable in the physical world. It's a looking glass into mathematical wonderland. The screen is a window through which one sees a virtual world. The challenge is to make that world look real, act real, sound real, feel real. And is that not the case as much today as it was 50 years ago? I think so. One of the key quotes from him is about the ultimate display. The ultimate display would of course be a room within which the computer connects the existence of matter. A chair displayed in such a room would be good enough to sit in. Handcuffs displayed in such a room would be confining. And a bullet displayed in such a room would be fatal. With appropriate programming, such a display could literally be the wonderland into which Alice walked. So what is missing? How far are we? And how come we have not realized everything that he spoke about 50 years ago? Come on, it's 50 years later. Where are we? Well, first of all, if you look at sort of the technological side, we have a huge offering now of visual displays. And I think that most of you here today will know the various models that are available on the market. If we look at the auditorial displays, we have solutions ranging from headphones over to two channel, eight channel, 16 channel, and in our lab 64 channels solutions. So we're doing quite well in the auditory domain as well. However, haptic feedback kind of sucks. It's really not going that well yet. But I expect that it's a sort of niche that will really expand a lot and huge progress is being made at the moment. However, there are certain limitations. Things in the virtual world don't really, we have not come close to really simulating how things in the real world feel like. And presenting that in the virtual world. One major, major obstacle right now is that we still have not figured out how to walk, how to navigate in virtual reality. A lot of resources, a lot of work, a lot of manpower is going into this field. And it's also something that we are spending quite a lot of time on in our lab. I'm going to show you some of the things that we're doing. You can say that this field is also very complex. But let's start to think about what it takes, which changes the way we think. We need to think about what it takes, which challenges we are faced with. Well, for example, we need to think about that most people will probably not have a huge space in their living room to move about in. So when we think that we can offer users to move about in an infinite world, but in a very limited physical space, then we have to come up with new ideas, new approaches. Then we also have other things to consider. For example, how do we sound when we walk? How is the sound of the footstep? Should it perhaps reflect the different surfaces that we are visually visualizing to the user? I've been working in this field quite a long time. Not that old, but quite a long time. In 2002, 2003, I had sort of my baptism within virtual reality. I started to work as a research assistant while still being a master student. The first project I worked on, sort of for serious, was the Binogo project. It was an EU-financed project belonging to the Future Emerging Technologies program. It was heavily supported. It had what we today would call the promise of 360 photorealistic environments. Not 360 movies, but 360 environments. Some of the big problems here was that it felt very, very static to be within this environment. Sure, it looked nice. It was pretty. It was settings from the real world that was being photographed and then represented in 360 in the virtual reality. But I was thinking, what can we do to make the user experience, the experience of being in the city, more lifelike? So that people want to really explore more. I came up with a solution for making more interactive shoes. These were shoes that were embedded with sensors, actuators, and so on. That really worked. Actually, we could make people investigate the sceneries a lot more. They started to move a lot more, which was really the goal. This led into the next project that I was working on, which I was heading the Danish side of, natural interactive walking. Also, a FedOpen project, Future Emerging Technologies, from 2008 to 2012. Again, we developed new instances of the shoe, and they became more and more advanced, and we could do more and more things. Here we see just a small example of the inside of the shoe. So what can we use this for, if it's not just for, for example, these virtual settings? Well, for example, walking rehabilitation is something that we've played a lot around with, and have tried a lot of things with, and we have quite good success with it in our experiments. We will see more examples of rehabilitation, also, as was mentioned in the presentation of me, for elderly people. But to know more, if you want, because I'm just briefly going over this. If you want to know more about the whole field of walking, to be in that virtual space, and trying to have the users interact more with the environments, to walk more, and so on. We have a couple of books out, and it's not because I'm trying to sell these books just to inform you. And actually, the book on the left is free, so you can get that free of charge. Today, a lot of things are given, and you can do a lot of things very, very easily with some excellent tools like Unity. And it really becomes a lot easier to develop virtual reality with these tools. However, we also need to think about that you cannot just, as you're used to as a computer scientist, sit and program something. You actually also need to know something about physiology, how the brain works, how the senses work, and so on, to get an optimal experience. And this might not be something that everybody is thinking about when they start to develop. And this is where labs such as my own come into the picture, because we are always willing to collaborate and help. There are some different areas that I think deserve special focus, and that we have spent a lot of time on also in the research sector. For example, how to create and maintain presence. Presence is one of those words that I see being used a lot, which is good, but many times not in the right context where they should be used. So this is something where I think many people could really benefit from looking at the vast literature that is there, and trying to understand a bit better how do we actually get this feeling of presence. How do we build that? Because it relies on a couple of factors. For example, we need to simulate the place, and that's fine. Most people are willing to do that. But we also need what we call plausibility. This was something that Mel Slater formulated as a necessity to drive the feeling of presence. And last but not least, we need body ownership as an integral part to create presence. Then I heard in the previous talk motion sickness being mentioned. Again, here we have sort of a little problem because many times different things are being sort of misunderstood. We have something called motion sickness. We have something called cyber sickness. And then some fall asleep and don't care about it. But actually, if you look a little more closely, just for two seconds, you can help yourself in your development of VR a lot. Because there are very, very big differences between motion sickness and cyber sickness. Now in one, in motion sickness, imagine that you're sitting in a car, you're driving, and you get sick. Why do you do that? Well, because the ear registers motion, but the eyes don't register. Typically, of course, you're sitting and reading a book or something like that. In cyber sickness, it's completely 180 degrees around. Here the eyes are very active, register motion, but the ears don't register any motion. So of course, that has to be reflected in the solutions that you are developing. You have to rely on knowledge on that when you're building something. Also, as we spoke about before, how to naturally walk in an infinite virtual world while being in infinite physical space. We have vast literature on this subject. There are so many solutions on this. And it can simply be, for example, that you have your user walk in a straight line while you slowly rotate the virtual world that he's within. However, it cannot be done too fast. Otherwise, you will create cyber sickness. A third field that we're working a lot on in our lab is 360 movies. Strictly speaking, not virtual reality, but we go the step beyond that. And here's one of our kits for recording these movies. Quite expensive hardware, but a lot of fun. We try to look at how will this look like in the future. And we think that it needs to be collaborative. Why do we do that? Well, if you look at how, for example, music used to be used in society in the western part of the world, it was originally, and here I'm talking several hundred years ago, a social event when there was music. It was something to be enjoyed with your friends, colleagues, and so on. And it's actually only until the 1900s that it has been that social thing. Then something happened when, first of all, radio was introduced, and later on, especially with the Walkman and the iPods. Then music started to become individualized, and it was something to be enjoyed in your own ears. And you shouldn't sort of affect others with this. But perhaps we are seeing a trend towards that these things should be social again. At least when we go to concerts, when we go to see a movie and so on, typically it's together with a friend. It's actually a social experience more than it's something to enjoy the artwork or the music. So I could imagine that, for example, collaborative 360 experiences where you actually have the opportunity not only to witness the 360 experience yourself, but you can do it together with friends. And within that 360 environment, you can actually see your friend's body in the 360 environment. But you can do it together with friends, and within that 360 environment, you can actually see your friend's body in some sort of avatar or something like that. That would be great, and that you can actually communicate in this environment. And we've done a lot of different tests with this. We've also shot a lot of movies. We've had the pleasure to record some of Denmark's sort of major bands in the alternative genre, such as Spitz Neuenhack, and had a lot of fun doing that. Good. It says demos. And it says that because at this expo, we are quite heavily featured in the demo section. We have a lot of demos from our lab and also from the studies in meteorology, sound processing and so on. We have demos featured here at the expo. I'm just shortly going to introduce them to you. For example, the virtual reality supermarket. This is a project developed by some students that is featured and is concerned with how do you actually install a supermarket. How should it look like? This is an application where owners of markets and so on, corporations, they can go in and design virtually supermarkets where things are being sold. The total design of these supermarkets. And then when they're satisfied, then they can go out in the real world and do it. This gives an enormous opportunity because, of course, then they can have users in, try it out and have an almost real world experience. Jungkram Ruhm-Persen is no longer a PhD student with us. He finished this Monday and he's been working on VR exercise for older adults. So these adults that are in retirement homes and similar places. And if you think about one thing is sort of industry, the other thing is public sector. Imagine how much potential that market has. For example, just for the elderly, we could also talk about education, training of nurses or whatever. There are so many different things that you can design and develop applications for in virtual reality that could help the Danish society, European society and so on. I would expect that this becomes a very important market for US developers. And I think that's a very important thing. And I think that's a very important thing. What we've seen with this project is that it's a bike that the elderly can cycle on and, you know, peddling on a bike, sort of motion bike, is not that interesting. So what Jung did was that he created these scenarios and tried to see how that would work. And he did it in a very interesting way. So what Jung did was that he created these scenarios and tried to see how that would affect their use of this equipment and also their willingness to get more exercise. If we look at how many more seniors we will be in the coming years, it really makes a lot of sense for the public sector to invest in strategies built upon this. Because we saw a definite trend towards that the elderly really, really want to use this equipment. They become much more mobile and they really want to exercise. Jung, he finished his PhD with a presentation on Monday, his defense. But at the same time, he's been contacted by this retirement home because now that he's finished, they actually lack this equipment. They lack these experiences to give to their residents at the retirement homes. And that's not only the staff saying that, it's also the elderly and it's their relatives. So imagine if we could just make, I don't know, 300,000 elderly become more active so that we could just reduce their downtime, their sickness time with two months, three months. Which benefits could we reap from that in the society where they would not anymore need care but just those two, three months? I bet it's huge, that number. A third demo is virtual reality instruments built by Louis Thompson. And in this demo, you have a drum kit and the thought behind this is that it will be expanded to include a range of different instruments. A lot of fun. Then we have an art experience. That's something very interesting because many times when we talk about presence, we talk about experiences that mostly have to do with experiencing a place. But in this art experience, we start to focus on time. And we start to focus on something that is very unique to the human being. That we build our own narratives. So I highly recommend you to go in to see this. It is a trend and central experience. It belongs to the category of slow VR and I think it's a unique possibility that you have to experience it here. Now, when those of you that develop, you get uncertain about things, you get frustrated and so on. There is, as I've said several times, a lot of literature available to you. But there are also a research community right here in Denmark that you can make use of. And we are willing to collaborate with you. With things, with ideas, with companies where it makes sense and to help out there. And hopefully it will be some beautiful partnerships. That was all for me right now. Thank you. Ralph, can you tell me in your opinion which markets do you think have the most potential for VR? Right now the public sector would be my, if I was sitting and had a company that is doing a lot of research. And I think that would be the best market for VR. Right now the public sector would be my, if I was sitting and had a company where I would be able to develop whatever I wanted. I would choose the public sector. There are so many things you can do in there. Make a small fortune. So make a small fortune in the public sector. Alright. Thank you Ralph and thank you.