Design, creative processes, and Cargo Cult Science

Good morning. Good morning. I'm so psyched to be in this town. This is like the most beautiful town on earth. Everything is working. How many, just of curiosity, is from Odense? Raise of hands. Raise of hands. Is from this town. How many are from here? Raise of hands. Okay. Four people? Did everyone else travel here? Really? Amazing. I'm so psyched to be here. So, a quick introduction. Hello, hi. This is me. I have spent some time at Spotify and GitHub as a designer developer. And I spent a lot of time consulting and teaching at Hyper Island School in Sweden and now also Singapore and London and New York, which is sort of like Chaos Pilot, but a bit more focused on development and some other things. So, this talk could really have been called Startups, Processes and Critical Science or like Companies and Processes and Critical Science. But I want to focus a bit more than that. And the need for this presentation really started when I was at Hyper Island. And I was teaching two very different courses. I was teaching programming, which is, you see, very simple to teach because you have this very confined space of logic that you're teaching within. So, you can say, like, write this code and out comes this output. Great. If I say something wrong, someone can, like, prove that I'm wrong and then everyone can learn. It's very simple. Now, teaching design is very difficult because we have all these subjective thoughts and we have all these processes. And it's very difficult for me to say, this is the right way to design. So, my sort of way of dealing with that when teaching that at Hyper Island was to bring in just as many conflicting methods and opinions as possible, right? So, Spotify would come in and they would say, only do A-B testing. Very good. We need to measure things. And then a creative agency would come in and say, we make our clients wear hats to understand emotionally their customers. And that's great. And they would really just conflict, right, and have different ideas of things. And they will all sort of touch upon these things, but they would have very conflicting opinions. And I think this is a great way of learning, right, and finding nuance within things, not getting the right thing. But a thought that struck me and that I kept with me after doing this was, like, surely everyone can't be right if everyone's disagreeing. So, surely there's something that we're missing. So, what I started telling people was that please share the cons of what you're speaking about as well. So, please come into the school and share, like, what's the risk with A-B testing? What are your assumptions when you're saying that it's good? And the funny thing is that no one did. No one came in and actually said, yeah, it's good. And they shared those things. And so, what ended up happening was that the students had this sort of process, which was, like, everything. So, they, like, of course, they did the IDOL and then did point of departure and market research and user service and personas and everything. Sort of by the book, which is great if you have, like, infinite amount of cash, which is very expensive. So, what I also started doing was taking people that had no process, that had no fancy word for what they were doing. They were just producing things. And by all measures, they did as well as anyone else. So, sort of the hunch here is that the way we're talking about our best practices may be misguided. But also some of the actual best practices that we're usually talking about may be misguided. What I'm out for to talk about today is not really disproving a lot of things. It's more like a mindset that I think we can use to think about these things. And so, it's going to be a quite critical talk. And I think it's hard because it's really fun to talk about possibilities. Like, many people like talking about possibilities. And I think it's fun to listen to them. So, usually, a lot of conference talks are like, do you want to be successful? Because I work at a successful company and you can be successful, too. So, like, follow these three steps. And then you'll be, like, massively productive forever. You will have no issues with your NPM dependencies anymore. And you will understand CSS finally once and for all. So, you'll get famous and rich. And you can buy a boat and retire and never work again. And this is sort of great the first time you hear a talk like that. But then you take with all of these, you take that with you and you go back to work and you hit by reality. And you start to realize that maybe there wasn't, like, all the nuance that you needed for you to figure out how to actually work those methods into your workplace. So, maybe you're a bit more skeptical next time. So, this is sort of... Difficult, but what I don't want to do is to sort of become also this hack and use person that is just, like, bashing on everything. And I'm not sure if you know this person that I'm talking about. The person that seems to be that nothing is true. The world is going under. Everyone is losing their jobs. Everything is horrible. And the kind of person that is looking more to win an argument than arrive at the truth. Right? And then you just go around as stalkers, right? So, if everyone is just trying to win an argument, this is... I'm not sure what's happening. So, hopefully we can all embrace, like, if I touch upon a subject that you care deeply about, like, user testing, it's an opportunity to learn. And maybe you will react like this first. But then we can maybe get to a point where we're like, we actually don't know. And that's sort of the key question. Because if we can arrive at that, we can start talking about some new ways of looking at things. And maybe we'll get out of here with a more nuanced discussion. So, today I want to talk about what are we sure of? How sure are we? What are the underlying incentives of us doing things in certain ways? And how then should we deal with that? How should we listen? And how should we talk? So, design, credit processing, and cargo call science. Let's start with cargo call science. So, and let's do that by jumping way back to World War II. Where a few things... How many have heard of cargo call science before? Raise of hands. The expression. Okay. So, the expression comes from... It started with the science of the future. During World War II, people were just like... Imagine that you have no contact with any like Western civilization or anything. And you're just living peacefully here. And suddenly during World War II, a lot of actual bases arrived. So, they build bases like US, for example, to get close to Japan and other targets. And then they arrive with a lot of cargo. And then what happens is that after a while, like, the war ends. Hooray. And everyone leaves. And people on these islands are left with like big metal planes and a lot of questions. Like, one, where did people come from? Two, what are these planes? Three, when are people coming back? Like, legit questions. So, this actually played out. And this happened, for example, here in the South Sea on an island called Vanu Tao. And what happened was that people left. No one came back for a while. But when people came back... They had some sort of ritual and sect and religion going on. So, this is a plane. Doesn't fly. This is another plane. It looks, I think, a bit better than the first one. But it also does not fly. Here's a wooden gun. Very interesting. Here's goggles and headphones. Here's an old uniform. So, Richard Feynman, an iconic physician. He coined this phrase in a speech at Caltech. And he said, for example, They're doing everything right. The form is perfect. It looks exactly the way it looked before. But it doesn't work. No airplanes land. So, I call these things cargo cult science. Because they follow all the apparent precepts and form some scientific investigation. But they're missing something essential. Because the planes don't land. So, this is... I'm not saying that this is our industry. That's too cheap. But it's easy to just... Mimic something. And then end up completely in the wrong space. But it looks kind of nice. And Feynman's point was sort of that. We're spreading a lot of these things. We're just looking at how the world is working. And then doing it again. And we have things like horoscopes and newspapers. And today our platforms for spreading cargo cult science is, of course, Like Twitter and Medium and Facebook. And just like being at work doesn't equal that you're doing work. Of course, right? Building an airplane doesn't mean it can fly. So, that's sort of setting the frame for the cargo cult science bit. So, how sure are we of the truth we're sharing? So, to add a bit more nuance to this. Let's talk about the weather. And I don't mean as in like cold speak. Like we have nothing else to talk about. But who is the best institute to ask about the weather in Odense, for example? So, actually the more local the predictor. The worse the weather gets. This is sort of a pattern which is weird. So, usually in the country what we have is a national station predicting the weather. And then we have more local ones. And you have information being spread nationally. And then in local newspapers. And there's a very weird incentive here that's making this happen. And it's not that the local like news or predictions are worse. It's that they have a peculiar incentive to make this happen. And to understand that, let's ask ourselves the question. What happens if you say that it's not going to rain but it does? If I'm in a newspaper in Odense, for example. You're probably going to react something like this. It's not a very nice thing. But on the other hand, if you say it's going to rain and it's sunny, who cares? That's a great bonus. So, there's this peculiar incentive that actually makes the truth bend itself slightly. When you can see this, like especially in the US. They've done a lot of research on this. And it's from the book The Signal and the Noise by Nate Silver. There's a small incentive to make you fail on the wrong side. So, if you're unsecure about the weather tomorrow, just say it's going to rain and it's going to be fine. No complaints by people in that town. So, truth can be a bit fragile. Let's talk about the secret of Spotify's success. Spotify, I think, is an exciting company. And I worked there for a long time with design. So, just to underscore success. This is how many paying subscribers there are for different services. And Spotify is actually bigger than all of the other services combined. This is a few months old. Actually, Apple announced, I think, yesterday or two days ago that they have 15 million now. But they're still bigger than everyone else combined. Which is interesting. Because they were this little company. Two people starting out. And then they went up against behemoths like Apple and Google and Microsoft. It's not even here. And then RDO, for example, started by Niklas Sandström, the Skype founder. So, I think a good question here is to actually ask, did the design of Spotify play a critical part of its success? And I think to get an answer to that question, we need to talk about... Well, yeah. And I think it's critical to ask that. Especially, like, if I would be here two years ago talking about how we design. At Spotify. Should you then listen to me? Because if it's not part of the success, then why should you be listening to me? Right? So, I sort of shared this dilemma on Twitter. So, startups are obsessed with processes. Looking back at my time at Spotify, I can't tell if our success was because of what we applied or despite it. And the old designer from RDO, which is a very talented designer. He's from Apple. He worked at RDO. And he says, for what it's worth, I still have the same question. I still have the same question about RDO's failure. So, really, we're both unable to abstract any truth because what does the failure and success tell us? But I think we can extract some truth if we talk about what's actually hard about building music streaming service. And I think what's critical to music streaming service is, one, streaming music, obviously. But it's very important to have a low latency. Like, to immediately deliver music when you click on a song. Right? So, that's a major problem. Especially if you're a small company. You don't have tons. You don't have tons of money to have servers everywhere. So, infrastructure is a problem. Two, getting music to stream is a problem. So, you're very dependent on the music labels because they know that your promise to the customers are that if you have all the music. Right? So, if you're a big label, you know that if you start being like, no, maybe we don't want to join, Spotify is going to be very keen to keep you on. So, it's a very difficult thing just getting music to stream. So, how many have seen this website before? No. Yes. Pretty much everyone. So, this website in itself has nothing to do with Spotify. However, to use this website, you need a torrent client. Right? And a torrent client is great, especially for the Pirate Bay because then they don't need to host anything. They don't need to push in data. You start uploading when you start downloading. So, this is actually the basis for Spotify. This code running in this app. So, Spotify bought this. And they sort of turned this problem on its head in the early days. Right? So, instead of spending a lot of money on infrastructure, the more users they had, the more people would be uploading songs to each other. So, if you listen to a song, you would cache that and you would upload that to someone in your vicinity if they started listening to the same song. So, I think that's a very crucial step to Spotify's success. And RDO, for example, absolutely failed on this point. They had a web client that was very slow. They spent a lot of time just building infrastructure. And another thing that Spotify did was that they launched in smaller markets. Right? They launched here in Denmark and in Sweden and in Norway first. And then they could prove that this concept works. And it was much easier working with the labels versus RDO who launched in the US. Right? Massive market. Very difficult to get all the licenses. And then difficult to expand to other big markets. Right? So, Spotify basically worked its way through iterating on small markets. And I think those two keys are much more important than design. So, then how important was the design? I don't know. And if the design is not deciding the actual success of Spotify, then how much can I learn as the designer at Spotify? That's sort of the question. And I think even though if we sort of know that already, we spend a lot of time looking at the context of things. Right? So, maybe then it's more exciting to hear about someone from Facebook who succeeded rather than some other social network. Although they don't know they're learning. So, would you listen to this kid and this parent or this pug? I would like always take a ride with this pug. It's way cooler. Even though the parent may have a lot more knowledge about driving a car. So, that's sort of opening up a few problems. And so, how should we deal with this all? So, I've got five questions that I ask myself when I take in lectures at Hyper Island. And I think that we can use that as a general framework. So, those five questions are, one, how does the company's feedback cycle look like? Where this person is coming from? Two, is the reasoning based on anecdotal evidence? Three, what are the underlying incentives? Four, are they listening or preaching? Five, are they in a similar situation? And you might have noticed that all of these highlighted words actually spell out fails. Which is feedback cycle, anecdotal evidence, incentives, listening or preaching, and similar situation. And you might be thinking that's a bit cheesy. That's too easy. Right? And like, hold on. The same letters actually spell out Leif S. Which could be a name like this guy. But it's not that memorable. So, I went with this one. And it's a shame because it could be like the framework of opportunity. Where we try different things. And like do the unexpected. And take opportunities to like have fun and enjoy our workplace. And really anything else would have been better. Like the framework of pugs. Again, great, everyone loves pugs. But this is what we got, so we got to work with this. So, let's talk about the feedback cycle. So, you learn by seeking out what you're worst at. Right? So, for example, this girl probably has a better feedback cycle than this other girl. So, to get good at something, you just try and try over again. And then you iterate on what you're doing. And then you learn. So, what does a good feedback cycle look like at a company? I think a particularly good feedback cycle is Pixar's. So, they have been around for a very long while. They have very confined time frames. Right? And they work with a movie for a while. And they release it. And they release about a movie a year. And they've had a lot of successes. They thought a lot about their process. And they can compare the different outcomes from the different movies. And they're quite similar, everything. So, it's a really good framework for them to learn. Compare that with a lot of startups and a lot of agencies where it's so different from case to case. It's even difficult to compare things. So, I think that it's obviously a spectrum. And it's not like Pixar has everything right and creative agencies has everything wrong. But I think this is a sign of a good feedback cycle. And I put a lot of weight onto what they say compared to a lot of young startups. So, that's feedback cycle. Two is a reasoning based on anecdotal evidence. This is one of my favorites. This is an article on Wired. Why WhatsApp only needs 50 engineers for its 900 million users. And so, they say things like its approach to engineering was unlike any they had seen. In part because it used Erlang. And a computer operating system called Freebie. What's more, Erlang lets coders work at high speed. Another essential part of modern software development. And they go on and on about how important Erlang is. And I think they do have a point. And I think they are having a valid discussion about Erlang. But is this the answer to this question? I think if you arrive at that simplistic answer, you're missing out on a lot on what this nuanced discussion could be, right? So, they don't know. And that's the important thing to arrive at. They don't know because they have no actual proof supporting that. So, it just proves that if a thing is trendy, we're bound to have successful companies who follow that trend. So, right now, React has been super trendy for a while. And then we're going to have successful companies who uses that. And it's not proof of React. It's just proof that we are trend sensitive. So, correlation doesn't equal causation. And is there any quantitative or qualitative data supporting that claim hypothesis? I think that's a good question. But even better is that someone else do that research. Because it's very easy for us in this day and age, especially on Twitter, to do like a search on Google Trends and say that that's research. So, this is pizza. It's the blue graph. And HTML is the red graph. I found this inverse correlation the other day. So, obviously, this tells us that the industry is changing. The market is going down. And you should be investing in pizza, I think. So, let's just leave the research to someone else, right? This has other reasons for this happening. So, one good example of that is Workroutes from Google. This is a book on where Google is doing a lot of research on observational experiments. Where they're trying to find correlations between different ways of working and good performance. We're going to come back to this, though. But this is, I think, a good example of what you should listen to. So, three, what are the underlying incentives? So, incentives are forming other behaviors, as we saw with the rain example. And I think a fair question is like what incentives do I have here today standing up on stage, right? I'm very likely to try to make everything seem as fragile as possible, whereas that's not usually the case. And there are other common incentives when people are up on stage. Like hiring is a really common one. Like people usually give talks to hire. So, then you exaggerate how well things work at your company. And that may, like, you may miss your point if you start, like, hurting to these, right? So, exaggerating challenges. No one wants to work at a company where everything is done, where you have no challenge, right? So, you're likely to play that you're an underdog if you stand on stage and want to hire. You exaggerate the importance of your role. So, if I'm a user tester, of course, I'm going to talk about this. I'm going to talk about how user testing is really important. So, I think there's a lot of incentives. And just question those. And then ask the speaker, like, what, how do you think that it's forming your behavior? So, four, are they listening or preaching? There's actually some really good research on this. So, this is a good way to tell if someone is right or wrong. So, there are two different behaviors here that correlate with a tendency to be right or wrong. So, are people looking to learn? Or are, and are they sharing their shortcomings? That's sort of the mindset that I'm saying when I'm listening. You can, of course, speak. But there's some research on this by Philip Tetlock, who is a professor at the University of Pennsylvania. And he did a study where he got 300 sophisticated political observers, a lot of people with PhD, two-thirds of them. And he got them to answer questions over a period of time. And he said, how did the Muslim Brotherhood win the elections in Egypt? And all the questions were yes or no. So, you always had a 50% chance of getting it right. And it had a set time frame. So, you know that by this date, we know the answer. Right? So, that was the setup. And then he measured how well people performed in this experiment, around 300 people over time. And on average, the scientists and the people working with politics, this is their job, right? This is what they do. This is the people we hear in the newspaper. So, they're average was worse than random guess. So, sort of like if we just left the decision to a monkey with no context. And actually, I think, so how many here has a coin on you right now? Raise your hands. You have a coin and are willing to. And any more. The more, the better. This would be really fun. Okay. And maybe you can share some if not. Okay. So, if everyone has a coin, has a coin, can you please stand up? And if you have more coins, maybe give it or lend it to one sitting next to you. And you can stand up and have a coin. Perfect. And please stand up. So, we're going to have a coin flipping competition. And I love this. I'm so good at this. So, what you do is if you're going to flip, everyone's going to flip a coin once. And if you get heads, you're going to keep standing. If you get tails, you sit down. All right. So, if you don't know which is heads or tails, just decide. Okay. It doesn't matter really. So, just pick a side. And then we all flip. And then tails sit and heads stand. Right? Okay. One, two, three, flip. All right. So, if you got tails, sit down. If you got heads, please continue standing up. All right. That's very interesting. Let's do it again. Same rules. One, two, three, flip. All right. Tails sit down. Interesting. We got some really good coin flippers here. So, one, two, three, flip. Okay. Nice. All right. So, and flip again. One, two, three, flip. Seriously. What the fuck? Okay. So, one, two, three, flip. So, please. Let's see if I can turn this on. Right. So, what's your name? Yen. Yen. Okay. So, can you... So, congratulations. You flipped, I think, seven times in a row. That's amazing. So, obviously very good at coin flipping. Can you share any... What's your technique? No technique. Nothing? Really? Did you think about something special while doing this? No, no. No. Just the heart on the coin. The what? The heart on the coin. Right. You were thinking about that? Yeah, yeah. Okay. So, that's cool. So, there's a clue, right? So, that's... Thank you very much. So, this is the exact same scenario, right? So, we've got a 50% chance of getting something right or wrong. And inevitably, because we're a lot of people, we're going to get someone who's very successful. So, this is the tricky part. We got someone who's obviously very good at coin flipping. And it's difficult for us to tell if actually it was luck or not. Because we got someone who did something. So, what would we do? What we tend to do is we then find this person who is successful and we start listening to this person. Very scary. So, this is why this research actually went on for 20 years. So, they did this for 20 years. And then you start noticing actual patterns because this is a fluke, right? And if we do it again, probably not going to be as good on flipping those coins. So, and the important part is that not only was the average 50, but they realized that there was actually people who consistently outperformed others. There was just also people who were consistently bad. So, they sort of compensated and the average was 50%. But so, they found people who consistently performed well and people who consistently performed bad. And Philip was able to like create two sort of personas that correlated with this behavior. So, if you want to be a bad predictor of the future, he called these hedgehogs. And hedgehogs typically have a very strong opinion. They're very outspoken. This is typically the people we see like on TV. And he really applied this to like dogmatism. Having a strong opinion and whatever happens to the world, you have a clear answer because you put whatever happened and like you put it into your context of how you view the world. So, and if you have that, you're a bad predictor and you may end up doing some very weird things because you think you know how something should work. The opposite, he called the foxes because they are a bit more nimble in their mindset, right? So, they may have an ideology but they don't necessarily put everything into that and adapt the world to it. They just use it to sort of make sense of the world. And the problem is really that these foxes that are nuanced, they're not very fun to listen to. And that's why we don't see them on TV because if you have a debate and you have people who just sit there and says like, well, it depends. If everyone, that's not fun, right? So, there's an incentive there from the media to find the hedgehogs who have strong opinions but that correlates with that they're actually not that good at what they're doing. So, this is scary. How many have heard of Ted Talk or read a book by Malcolm Gladwell? All right. So, Malcolm Gladwell is a very famous author. He's got some great Ted Talks, one of the most seen. But if you listen to interviews with him, he's actually typically very defensive. If someone challenges him, he just like finds reasons only for his view to be true and he doesn't look for any reasons why it's not true. So, I think this can be a good example of going towards a hedgehog. So, a good way of dealing with this, this is sort of a tangent, but if you have someone like this at your work, for example, there's some research actually showing that here's the Amazon link for a book where this is from. A great question to just ask is what would have to be possible for that to work? All right. So, if you're arguing and you have one point and they have another and they don't seem to care about your point, say, what would have to be possible for what I'm saying to be true? And this is a very interesting to say because what that does, it opens up the possibility for this very pessimistic person, maybe, and defensive person to brainstorm about why they are wrong. And they can still like feel like a winner even though you switch their attention to that they're wrong. But you're doing it a very clever way. All right. So, look for that and then ask these like kind of questions. Five, are they in a similar situation? I think this is often the most overlooked thing on the FAILS framework list. So, if you're a startup and they are a startup, are you in the same stage in your product lifecycle? Do you have a similar culture to them? And this is actually really important. So, for example, the research from Google in the work rules book, for example, talked about that. way to distribute bonuses. It's to not do it by a manager but to do it by peers, basically. So, if we all work in a team and you have maybe five like bonuses to give out, they are small but you give them to your peers, those who you see do a good job. And then you're like crowdsourcing the intelligence for the group and then that's a much more fair representation. So, that was Google research. They have this in work rules. Great idea. We tried it. It didn't work. We tried it in a company that is very different from Google. And I think the biggest difference, we tried it at Spotify, is that we were Swedes and they were American. And we were like socialistically like this is unfair. Why? And I work in a small group so the people giving me bonuses are going to be fewer compared to like the developers and they are like 100. They're going to get tons of bonuses. So, people just started like making pacts and like if you give me a bonus, I'll give you one. And it blew up, got a lot of critique. And I think the biggest just... The thing here is that it's a different culture. So, if you look at the research like yes, it's valid, it's got data, but the culture is so different. So, as soon as you change that variable, you're going to take a step back and be like, okay, the hypothesis is that since the variable changed, it may or may not work. Let's try it and be open to failure. We did it for I think three months and we stopped at Spotify. I think another thing that determines a lot of our behavior is how close we're working to the end user versus having a stakeholder. So, if we have a spectrum of different ways of working, we have working towards the stakeholder over there and end user over here. And I imagine like startups are here. You work towards your end user. And then we have creative agencies and studios and other things on this end. And this, I think, is just a spectrum of different needs and that determines our behavior to a large extent. So, we have, I think, a more idealistic approach here and pragmatic approach here. And you focus on the processes because that's what you sell. When you go to a customer and you have one person hiring your team and you got to convince this person, then what you're selling often is past success and your processes. They're like, we do this and this and this and therefore we succeed. So, you spend so much time focusing on processes and having fancy names versus maybe just a bit more pragmatic if you're a startup. And you focus on like moving numbers. For example, running A-B tests and things like this. So, measuring things is better. Also things, if we have a spectrum of small company to big company, there's a lot of talks like this culture is better than this. But I think it's often just different needs, right? So, we tend to have flat hierarchies in small companies and deeper hierarchies in bigger companies. We have no process. That's great in small companies because we know each other. It's a small team. So, we don't have to have processes. And then we have a lot of processes at big companies. And you're more flexible. And you're more flexible versus you tend to be more efficient. Then you want to measure things and be certain that we're spending the money well. Right? And also maybe the founders are at the small companies and they are originally maybe engineers and designers. So, we tend to have people speaking more about things driven by engineer and designer in a small company versus a big company which is more and more about money. So, this is not a binary thing, right? It's a spectrum. And I think especially when we're evaluating someone else's ideas, we can compare ourselves to where we are on the spectrum compared to them. All right. So, if I listen to an agency and I'm Spotify, I know that they may be using user testing to do different things. They're using user testing to get leverage at an agency to talk about that they are right. It's not that important to prove that they're exactly right because probably what the work they're doing is much better than what the customer has already. So, they're spending time on user testing to get leverage. And then if you only wish to prove things and move numbers, then maybe you're using A-B testing to a much larger extent. So, one question that I asked in the summary to this talk on the web was about transparency and flat hierarchy. I think this ties into that. So, research supports research from this book where good ideas come from by Steven Johnson. Very good book. And there's a collaboration between information sharing and innovation. So, the more information that you share within a company, the more you will innovate. And that's why Pixar, for example, has all their toilets like in the middle of the building, right? Because you bump into people going to the toilet. That's great. So, the question is then how can Apple be so innovative because they are known for like secrecy and not flat hierarchies which should prohibit like information sharing. But the answer to this question is actually quite easy. So, according to Steven Johnson at least, what they do is they just come up with a set of tools. They combine intensive sharing with sharing of information with a really deep hierarchy and secrecy. And they do that with something that they call concurrent production or parallel production. And what they do is instead of having a team work in silo for a while like the design team and then deliver a vision that the engineers can have a look at and then start to go back and forth, they just take a lot of teams and make them meet and like encourage chaos in the beginning, which is not very efficient. But it produces a lot of information sharing and overlap, right? And then maybe they use and they like that they have very strong hierarchies because that's extremely efficient as soon as you come out of this phase and focus on producing things. So, that's sort of the framework of FAILS. F, feedback cycle. A, anecdotal evidence. I, incentives, listening or preaching and similar situation. And I talked about these in the beginning. I'm not here to disprove them, but I think we throw around a lot of words that probably doesn't mean what we think they mean and we should question a lot of them. And hopefully, you can have some of these thoughts with you and we can have some better conversations during these two days. That's it. Thank you. Any questions? Or thoughts that you want to share with anyone else? You talk a lot about small companies and the flexibility of doing it right. So, when you are a stakeholder, you normally have a lot of control over the things that you do. You have a small company. But if you want to be, like say, successful like Spotify, is it important to be dynamic for the company also? What do you think? Yeah. I think it's important. Because normally, you have a strategy. And if there's a lot of different competitors on the market, you have to be different from the other ones. You say with the radio, the company, they failed because they have a vision of doing like the American market. But the Swedish market normally roll out in small areas. And if you see like Facebook, they also roll out some small different things on their websites. So, they normally test things in small groups first to see the efficiency of that one. Is that right? Yeah. I think so. There's, I think, some research on this actually in Zero to One by Peter Thiel, where he basically says that the startups who are taking funding that fail are the ones who never change their idea. So, to find an idea, you're more likely to find a good idea if you are very flexible and just move around a lot and try different things. And you're much more likely to hit on something. So, according to him, what he says that his data shows is that the successful companies that we see today are those who dare to be very flexible and move around until they hit gold. And the ones who are failing are the ones who just don't move around fast enough. Fast enough. Yeah. Because your money is going to run out if you're not flexible enough. Yeah. Cool. Yes. I get the idea more or less from your presentation or a part of it that sometimes it's dangerous to repeat what successful companies have done because they don't even know how they were successful. But at the same time, I don't know. I think that copying or try to get the good practices of people or companies who succeeded is more or less the only choice we, most of us, have if we are not kind of gurus or if we want or we cannot invent anything great anytime. So, it's just like a bit in the middle. I don't know if when you were working on Spotify, what was the reference to create Spotify? What was the reference? What was the idea that you got or it was just everything from scratch? Or you just took something in reference and you just tried to improve what they had done? Yeah. Yeah. No, I think that's a good point. And what we do is we do sort of build on the giants of others, right? So, we take what's out there. But what importance, I think my main point is that rather than just building upon it, it's so important to question that early. But to answer the question specifically about Spotify, the vision for Spotify from the CEO of the company was like, we're like iTunes but dark and we stream music. That was sort of what we started with. But then, of course, as soon as you're there, you want to question as much as possible. And right now, Spotify is working in a very different way and they have a lot of processes and they're a very big company. But then it was a small company, flexible, and just looking at other companies and looking a lot at Apple. And now they run past them. So, now the country will look at them anymore. But yeah, it's definitely more nuanced than just like not listening to people. We need to listen but also question. I'll take in here first so we can pass it. So, talking about Spotify versus RDO and those two different journeys, success and failure, how important was the design according to you? You showed us the slides but we didn't get any. I don't think it's that important, to be honest. Because if you look at what we changed during a long while, it looks basically the same as it did seven years ago. So, it may have, if we had done a major change at some point and that would have driven the numbers up at some point, it would have proven that it was important. I'm not saying that design isn't important. I just think it's, now in this case, it's difficult to prove that it's important. Just as it's difficult to prove that design is important for YouTube. Because they have been growing steadily for forever. They've been doing some design changes. And they have a big product team and they're working constantly on the design. But I think it's not as important as we generally think it is. Thank you. You mentioned Pixar as a good example of a feedback cycle. But I didn't really sort of see where the feedback was for them. In the sense of, okay, they produce a movie, they get a dollar value from that movie. Is that the feedback? Yeah. I know, yeah. It's difficult. It's difficult to measure, like, okay, does the box office returns equals our processes were successful or not? I don't know. My sort of point is that there is at least some feedback there. And then you can compare the different cases. So it's a sign that they can vaguely use. But importantly, they have a lot of projects that look very similar. And then it's easier to compare them. And they have been doing that during a very long time. So even though they maybe can do internal measurement and do, like, measure the happiness throughout the project. And then they have a lot of projects that are the same length and they're using the same processes. So they can start tweaking those and just change one variable. Versus usually if you start up an agency, it's like, whoa. Pull over the place. Yeah. Yeah. Cool. Any more questions? Cool. Thank you very much.