Thomas Bärtschi - Audiokinetic at CopenX, September 2016

Let's dive right into it. This talk is not going to be highly technical. I'll do my best to give you more of an inspirational talk. Because I think audio in general, and specifically 3D audio, is really a game changer. This is really an exciting time we live in at the moment. And one of the things... Let me see here if the clicker works. I'm just going to get a clicker that works. One second. Okay, it's on. So, one of the things that really inspired me 20 years ago when I went into audio production was what you are capable of imagining with sound. I think what's interesting is you can hear sounds and you can create your own stories, your own pictures. This is something that pictures can't do. As soon as you see a picture, it's much more difficult for the brain to actually think about another type of story. But with sound, I mean, just think about how many times we had our parents read for us and we were just so much into the storytelling. Okay. The interesting thing about the brain is really that nowadays, because of newer technology, we are at a point where we can actually start hacking the brain. So, it actually thinks that what it hears with headphones is the same as in reality, in real life. And for the past two, three years, there has been sort of a deep dive in the industry to try to figure out what is actually happening. How can we create this sensation, this immersiveness of the 3D audio space really doing something for the productions out there? Because audio has been a really good friend for a long time. All the productions that really stands out today, they really try to nail it on the audio side as well. And the sensation now when you actually can get the feeling that you are within the space, now when we are talking about VR, is fantastic. I have a great quote here from Oculus chief of research, Michael Abras, who says, It's not an addition. It's a force multiplier when you have 3D positional sound in VR. You buy into what you're seeing. This is really a strong quote. Then another quote that is inspiring me all the time is, The world is changing very fast. Big will not be small anymore. It will be the fast beating the slow. What I'm thinking is that if you go back several years, it was the tech that was difficult to get your hands on. It was the software that was extremely expensive. That's not what we see nowadays. We have some incredible tools and they are out there. A lot of them are free. We live in a really exciting time at the moment. It's not a matter of the size of the companies. It's the ones who can run fastest. Then this is something that most of us know, but it's still also a driving force. If we want to leverage the potential, we really have to dig into each of the categories. 3D audio is one of them for sure. I've been working in the game audio field for a long time. One of the things that really drove me was in order to create these experiences where you could see the players. They were just completely engulfed in these experiences. The tools that we use nowadays is something like this. It can look something like this. It's more about how you use it. Of course it is. It's about how you manipulate tools in order to get the given effect. At the moment I'm working for AudioKinetic. They are a game audio middleware provider. That means that they create a software that essentially makes it easier to implement and create interactive audio for games and VR. The software that ties up with a lot of the gaming engines out there. Obviously Unity is one of them. Feel free to grab me afterwards if you are working with Unity software and you would like to try out our platform. We have been developing a lot of tech for games over the past 10 years. More than 1,000 titles have been developed. More than 1,000 titles have been released with our sound engine. At the moment we have more than 100 dedicated VR titles out there using our technology. I had to put this slide in because it made me so happy to see that the Oculus Studio actually won an Emmy. This says a lot about the recognition within the industry. It also inspires a lot of us to really see that it does matter. We have been working closely with the Oculus Studio for quite some time now. To try to see how we can improve these pieces of software that enables the users to experience a fantastic gaming experience or movie experience for that matter. 3D audio is bringing to the world of mixed reality. Basically, 360 binaural audio is really something very simple as mapping out sound around the player. We are not restricted to just hearing something in stereo on our headphones. We are not restricted to having surround, for instance, 5.1, 7.1 system. Essentially, we feel that we are within this surround space when we wear our headphones. That's the thing about 3D audio that is amazing. Another thing is why is it really that important, you might ask. Because as we talked about, there's been audio in games and experiences for a long time. But I'm just going to list up a few pointers here. For sure, it boosts the experience. It boosts the sense of immersion. It draws you much more further into the virtual world. It helps identify the player surrounding 360, as I said. So it's not only here. It's up. It's down. It's in front of you. It's back. And obviously, that heightens the sense of realism. And it also helps to tell the story of the game. Because you can use 3D audio in a lot of new ways. And for sure, it gives feedback to the player in another type of way than we could before. Because you can suddenly... Let me give you a straight example. If you're playing a fighting game, Counter Strike. You can suddenly tell where the enemies are behind you very accurately. And you couldn't do that before. And a few pointers about audio in general. For experiences and games. It's very good to set the stage, genre, tone and mood of the game or experiences. It helps establish the pace of gameplay. And it's extremely good to create this emotional connection with the player. And time and space is also a big thing for audio. It can solve a lot of problems and challenges. In a given production. In terms of how to support the storytelling. And really to enhance the overall experience. So there's plenty of reasons to dive further into audio. We already had a little bit of information about HRTF from the Unity talk. So I'm not going to dive too much into that right now. The basic thing to know is that when you see this head-related trailer. Something like this. It's really something that tries to help emulate how we perceive sound in real life. With headphones. And it's also extremely good to accurately direct you. And as I said, 3D audio is something we can use extremely creatively. This is also extremely interesting. For you guys, I think. Because in HMDs, we really only see 15-20% of the whole sphere. That's because of our limited field of view. But when we look at our auditory mechanism. That is really how we perceive sound. How our brains perceive sound. We can actually get a degree of accuracy within. Let's say 2 degrees. And what I mean by that is. If we hear a sound coming from back over here. And it's turned 2 degrees to the left or right. Or up or down. Our brains is actually able to perceive that. So we are extremely sensitive. And this is also why there's been a lot of product released over the years. 3D audio headphones that promised a lot. But they really couldn't deliver. And the reason why they couldn't deliver. Was that the technology was not good enough. So one of the reasons was that the HRTFs. This algorithm that simulates how we actually perceive hearing in real life. Was simply not accurate or good enough yet. Then there's a term that is used a lot these days. Which is ambisonics. What is ambisonics? You might ask. By the way. How many people are actually working with audio here? Okay. That's quite a few. The interesting thing about ambisonics is. It's really as it says here. An intermediate spatial representation. And what it means is. That instead of mixing. If you were sitting at a mixing board. Instead of mixing to stereo. Mixing to 5.1. 7.1. Where you would actually route out the channels to the buses. And the buses would be connected to speakers. Then you really have a completely new way. New in the sense that it has been around for quite some time. But now it's really being implemented. New in the sense that you have a sphere. 360 sphere. And basically you're able to play sound within that sphere. And it doesn't know anything about 5.1. 7.1. But as soon as it gets decoded. Then you can actually choose for yourself. If you want it to decode into 5.1. 7.1. Whatever format. And the interesting thing is. That's very flexible by the way. Because you can do one mix. And then you can basically tell it. Okay. Now I want this kind of mix. I want this kind of mix. And you can also use it. Obviously for VR. Because what you do is. You place a binaural plug-in decoder. On your master bus. And what happens then is. That also has some of this. How can I put it in a simple way. This algorithm. HRTF built in this binauralizer. And what it does. It translates the ambisonic format. Into a format that fits the earbuds. Or headphones. So we actually perceive it as. Sound. You know being really. How can I say. It takes into account that. That we have a head. With two ears. And it really measures. And decodes. Exactly how we would perceive live in. Sorry. Sound in real life. I talked about this. Then there's the head tracking part. I think it's something that. Everyone should be. Clear about. The reason why head tracking. Is important for 3D sound. Is that. As it says. If you don't have the head tracking. In VR. The sound doesn't move. Because. The head tracking data. That is supplied within the VR device. Is tracking. With the audio accordingly. So if I turn my head. The sound will still stay over here. Compared to what I'm actually seeing. Mixing on headphones is a big thing. Because it allows us to. Really. Have our own. Virtual studio. Wherever we are. We don't need. A specific acoustic. Corrected room. Or. Extremely. Expensive. Equipment. Overall. Basically. We can do. A final mix on headphones. For VR. And that is. Something that a lot of. People working with audio. Are extremely. Delighted. About. To say the least. A call of best practice. Advice. For developers. Since this is not a. Huge. Tech talk. In VR. I think. The first. Thing. To do. Is. To. Try to. Download. Some of these tools. If you are interested in this. There's a lot of tools out there. And. If you are. Doing a serious project. You can actually learn a lot. Along the way. Something that I see. Still. Quite a lot. Is that. Developers. They. They tend to. Start thinking about audio. Very late. In the process. So. They. Specifically. For. VR. Where. A lot of people say. Yeah. It's. It's. At least. 50%. Of the experience. Because. If. The audio. Doesn't. Get you. In there. Then. What. Do you. Have left. A few. Players. That. Is. Something. How. Do. We. Set up. The. Sound. In. A. Of. Software. That. Behaves. Correctly. In. Terms. Of. The. You. Have. Of. Audio. That. Is. A. Source. for VR microphones, where essentially you can go out and you can record everything that's within that sphere. And that works, but the thing is that real life is really not that beautiful 80% of the time. So you need to think about if, for instance, you were out doing a shoot for 360 video, I take that as an example, and you had some actors that were interacting with each other, you can have a VR mic that captures the whole thing, but then you would also need to take into consideration working with spot mics, like you do in traditional film production. And then this is a standard we hear all the time, and we also hear it in VR, for that matter. Break the rules. We are all learning. No one is really an expert within this. There's a lot of opinions out there, but it's good that we challenge each other and ourselves. What does the future of immersive audio hold? A quick run-through here. This is mainly to inspire you guys. I think we're going to see a lot of improvement on the tools themselves. They are going to... They're going to be much easier to work with in the future. We are going to have some more high-level GUIs, so you don't necessarily have to dig very deep in order to create something fantastic sounding. We're going to obviously see the whole workflow pipeline for 360 video in terms of audio production improve vastly. Because essentially what we want is that there is one tool you work in, and you don't have to switch back and forth, between your traditional audio workstation and your HMD. We're going to see a lot of new headphones coming out that really takes it to the next level in terms of the accuracy, the HRTFs, the algorithms. And one of the products really famous out there right now for their Kickstarter is AUSIC, where they pride themselves having a technology that really can tailor the headphones, to your individual head and ears. So that's going to be very exciting to see how that works when it comes out later this year. We're going to see a lot of content, audio content from hearables. This is something new. We also saw it with the iPhone 7 coming out. No cable. People are going to wear, in the future, earbuds much longer daily than they are now. And this gives 3D audio, books, a very good way of trying to come up with certain experiences. Because look at it as you have a new set of ears suddenly. We're going to see a lot of great applications in terms of music production. Have your own studio. As I said, you have already the headphones, so you don't have to worry about a great studio. And now if the interfaces and the applications are working, We have one example right now. It's Soundstage VR. It's a very interesting product for sure. Where you can essentially walk around room scale VR, work in your own studio and start producing tracks. We're going to see a lot of interesting live applications and pieces of software come out. Where you can essentially see here that, this is a product called the Wave, which recently got a seed funding of 2.5 million US. It's a VR DJ music tool. Where you can essentially act as a VJ and a DJ at the same time. And interact with your audience. Really an interesting product. And I think it's Unity. They are building this with. We're going to see a lot of teaching apps for sound and music. Audio for VR games is definitely going to improve in the coming years. It's already good now, but it's really happening fast at the moment. VR experiences transport yourself to anywhere and feel like you're there. That's the obvious one. We're really going to see that get nailed soon. Interactive books is also a huge thing right now in terms of AR. And I still think that the sound content audio soundscapes are not really utilizing its full potential for this yet. Virtual theme parks is a big, big thing in terms of 3D audio. The void is something everybody talks about. At the moment, if you haven't seen or heard about it, check it out. It's a really interesting theme park concept from the US. Other verticals within VR, AR, where 3D audio is relevant. This is basically just a quick list. But nevertheless, there are so many areas where sound can really help the user to interact in a completely new way. Quickly, seven keys to better audio. This is really on the creative side. I think it's very important to think like an orchestra, orchestrator. Basically, when you see that you have this sound field, 360 sound field, try to imagine it as a real orchestra. You want to create something that is dynamic. You don't want the whole orchestra playing at the same time. That's not going to create any dynamics in your experience. And then context is king. Really think about when you play sound in an experience, think about what's the use here. Is it really needed? Why do we have it? Use focus and depth of field. I try to think a lot about photography when I work with sound. Because there are so many similarities. Basically, this is much easier to communicate. If you see a picture like this, something is in focus. Other parts of the picture are not. And that's how we want to work with 3D audio for VR specifically. We really want a system that also caters for, okay, I'm looking over here. I hear the sounds here. This sound in the back might be important. These sounds out here might not be that important. And that is something that comes in with the whole implementation process. And that's why you should look into dedicated implementation tools for VR sound. Remember the contrasts. Again, this is not specific for VR. But this is just specific for creating something that is of interest. This is basically how our brains work. If something is a bit too sterile, it's the same thing for too long, we lose interest. And then embrace the silence for sure. The most impactful audio productions that have been made ever had a lot of silence in them. And then keep it simple. Sometimes you might not need that much audio. And then the last bit is really the emotional sound design approach. You can also use this if you are a creative director, for instance. Try to think about sound as something that gives an emotion or should give an emotion. Even though it's a rainy sound, for instance. That can really be a happy rain, a positive rain, or it can be a dark, sinister rain. And it's much easier to judge whether a sound is actually needed within a production if you think about it in terms of emotions. Instead of, does it sound right? Maybe it needs a little bit more high-end, low-end, whatever. So, those are my advice for this quick talk. Feel free to come up afterwards. I don't know if we have time for questions. No, sorry. No? We are a little bit behind schedule. But just one question. What's the biggest challenge in working with 3D audio? The biggest challenge right now is really to get the workflow processes much faster, much better. There's still a lot of things that people think they should do when they are setting up a more advanced 3D audio workflow. And I think it has a lot to do with communication. Okay. Thomas Bartschi. Thank you.