Nick Popoff, Engineering manager at Eventbrite

Hello everybody, my name's Nick Popoff. I'm an engineering manager at Eventbrite. This is going to be a talk about some fun A-B testing that my team did about maybe six months ago trying to prove whether this concept of assortments was going to be a good idea for Eventbrite. It's going to be a mix of introducing some concepts around pricing and assortments and then talking through our crazy A-B testing experience. All right, so in case anybody doesn't know, Eventbrite is a ticketing company. We sort of got our start doing ticketing for the long tail of tiny events, and then over the last 11 years, we've grown up into hosting some pretty huge events. Our business model is free is free, so if you're hosting a free event, it's absolutely free to use our platform and all its features. If you're hosting a paid event, each one of your paid tickets, we're going to charge a fixed amount plus a percentage of the paid ticket. So that fact is going to come up later in the talk. Yeah. And then also it's what's called a two-sided marketplace company. So I work on the event organizer growth side where we're trying to get people who want to host events to use our platform, and then there are other teams that try to get event attendees on our platform who are just looking for fun things to do. All right, so sort of the start of this project was I was in a meeting with Nels, who's our VP of pricing, and we were just analyzing what had been going on in our business with the organic acquisition channel of event organizers, and we were just looking at how the business was growing in different markets. And Nels just out of the blue was like, it just feels like this business is something that's happening to us. It's not something that is happening because we are taking actions. Like, we would see, like, hey, like, our business is really not doing well in the UK. Or, like, we're doing great business in Singapore, but it wasn't because anything we did was not acceptable. Like, we have to take charge of our business. So around that time, Nels attended a talk by this guy. If you haven't heard a talk by him, he's incredible. I think of him as the Gandalf of capitalism. So he, his website has these three minute videos that actually go over all the concepts that I'm going to talk about way better than me, so check it out. But sort of the core point of this talk that he gave is this idea of assortment. And when Nels heard this, he was like, whoa, this is the answer to my problem. But Michael Deering's, like, key statement that he always says is history matters. All of this has happened before. Like, none of us in this room is doing anything new. None of our problems are new. These are all solved problems. You just have to take an interest in history. All right, so what is assortments? So assortments is a way to rethink how your offerings work. And how your customers value what you're providing. So in this case, it's like, someone could walk into your restaurant, they're barely hungry, they have, like, a nickel in their wallet, but they'll see, like, all right, we've got toast, I'll buy some toast. Someone else is like, it's something about them. They're like, I am a healthy person, I only buy healthy things, it's worth it for me to spend extra money to have a healthy meal. So they just want to put some avocado in their toast and tell me what they want. And then finally, if someone is, like, having a brunch, it's Sunday, they're just chilling out, and they're like, I deserve the best in life, I'm going to drop $16 on whatever the heck that is. But you can see, these are psychological statements. These have nothing to do with what it costs the company to produce that toast, that avocado, whatever. So an academic thing going on here is that in the past, the way we thought about pricing is like, what are the costs of the goods like, how much did it cost me in labor, in materials, and I'm going to add, like, a 5% profit margin on top of that, and that's my sale price. That's completely backwards. The way we're supposed to price is start from the far end, which is the customer has something that they want to solve, or they have a need, or they have an identity that means that they want something from you. And unless you understand that value to them, you don't know how to price your way to solve that problem. And the gap between their aspiration and your price is what they think of as cheap. And it has nothing to do with the cost to you to produce that thing. So this concept of good, better, best, and the title of the talk is this idea that once you realize that each sale is a psychological conversation you're having with individual customers, you realize it doesn't make sense to think of that as a single pricing transaction. Instead, what you need to do is have different offerings where, in this case, someone sees almost no value in your product. They're like, you're barely solving a problem for me. Why would I even use you? But you could offer something extremely targeted just for them at a very low price that they'll actually see that value. And then going all the way up to your most loyal customers where they're like, my whole company will collapse tomorrow if you stop solving this problem for me. And they're like, just take my money. Take all my money. To them, the value is at the far opposite end. If you think you're just going to have one conversation with all your customers, that's what you're stirring up. You have to have separate conversations. And then the foundation for this is basically the most productive friendship of the 20th century. These guys, Kahneman and Trisky, if you don't know about them, they're psychologists that basically proved that the entire field of economics was bullshit. And they did this through experimentation on actual human beings. And they proved that human beings are fundamentally irrational. Almost all of our decision-making is irrational. We're not actually machine learning, computing bots in our brains. Instead, we have really sloppy, buggy, approximate mental models that we apply to our situations. And so they came in and basically upended everything. And this book is amazing. So if you want to learn more about their ideas, I highly recommend it. Check it out. So we took all this in and we tried to create an experiment plan. So what we wanted to do was figure out what are the upper bounds of pricing for a premium version of Eventbrite, at which point the customer is going to be like, it's too expensive. So we wanted to see what that limit was. We also wanted to see, so our pricing is like a fixed fee and a percentage. Which one of those things is more important than the other in terms of raising or lowering it? And then we wanted to do some classic A-B testing around visual arrangements to see, like, what will cause them to take actions. And then in all this, we wanted to do as close to zero development as possible. If we could do it with, like, one developer with, like, one day for each test, we'd try to design it that way. Luckily, Eventbrite has its own A-B testing framework, so we didn't have to build that infrastructure. We just had to set up the test. Yeah. So a challenge we had is basically we're bringing, like, you know, 4,000 new event organizers onto our platform every day. We were trying to test some crazy new ideas on them. So how do you do that? We really didn't want to give people a bad experience, but we also knew that was, like, our only opportunity to catch people who aren't already familiar with our product. So it's their raw value. And so we wanted to see how they would react to things. So the idea we came up with was sort of the classic, like, some fake onboarding screens getting you through your choices that at no point are we going to offer you something that we're going to take back later. So we only tested higher prices. And after you make it through the A-B test, we immediately show you a screen saying, like, just kidding. Like, we have a promotion where we're going to give you a discount. And then you get the low price. And we also only said that we're going to take away features. But then at the end of the test, we said, just kidding. Like, you actually get all the features. So we had to optimize our test to avoid making promises we couldn't deliver on. So this is the first screen they see after they sign up before they would go into our event creation process that prepares them that something strange is about to happen. Then we realized that our free-for-free organizers are totally different than our paid organizers. And we knew that if we didn't segment those people off and have them not go in our test funnel, there was no way we were going to get good results. So we tried to grab just the paid organizers and put them through these tests. Then finally, we just iterated on a whole bunch of screens that are sort of like this. If our designer is here, he would immediately say, like, this is not the way we want our product to work. We want our product to work at the end of it. We were just testing, like, how do people react to lists of features, prices arranged different ways. And we basically just watched to see what did people click on. And then we just changed elements of the screen. All right. So what we saw is it took maybe, like, a week each test to get to significance. And what we saw right away was the difference. Right away was the distribution of people going to the basic versus premium offering was very interesting. Like, right away we were like, ooh, like, we have something here. Like, there is actually people who are willing to pay us more money than our current fees. And also there are a lot of people who, if we have a basic stripped-down offering, that's what they want to click on. So that was, like, proving that the assortments idea was viable. So we got excited. All right. So the bad news is, like, two weeks in, we started seeing the down the funnel data, and it was super bad. Like, we saw that a lot of people were dropping out of our event creation funnel. They weren't saving their event. They weren't publishing their event. And at Eventbrite, this is really bad because this is our revenue for the next quarter if we're killing this now. So two weeks in, even though the data looked promising, we almost killed the whole thing. Because we were like, we just can't afford to test here. Our theory is basically that customers at that stage of the journey aren't ready to think about pricing. They just want to play with the product. And so us forcing them to have this conversation then meant they were like, eh, we're just going to go somewhere else. So the way we worked through this is our product manager was able to make the argument to our executives that the early experiment data had a financial upside. That was exceeding the downside of this temporary decrease in our new user acquisition. So even though this number was a very large dollar number, we were able to convince them that it was, like, worth it to let us keep testing in this area. Even though there was going to be, like, a temporary hit. So we actually kept testing for another month after this. The whole time we were stressing out about it. So here's what's hilarious. So that was four months ago. So two weeks ago, we finally did a complete cohort analysis of revenue. And that batch of users brought in more revenue than the equivalent last year. We're still struggling to explain this. We have some theories. One theory is the classic, like, when you add friction to an area, a lot of the times the people who are dropping out were never going to be your customers anyway. They're just randomly clicking around. Another theory we have is Eventbrite is a great way to host content that has nothing to do with events. We let you create listings pages with HTML. And it turns out that people bought the hell out of our product to host really weird stuff. And so our risk team will track it down and turn off these pages. But a very large amount of new signups are basically bots preparing to be used for different things. So any time we make changes in our onboarding and we see hits to metrics, sometimes all that's happened is we broke the bots. Yeah. All right. So the key lessons is that we actually did see a reasonable distribution of, like, as we raised our price, we saw a drop off of people interested in the premium product. But it gave us this really clear sense that it was, like, under our control. Like, we can adjust the pricing. And we have a rough sense of how that will impact the product. And we can see how that will impact the percentage of people who go there. Another thing that's interesting is, like, a really large percentage of people see a low value in our product. And so we think they'll be excited when we have an offering that's half as expensive as our current that just gives them the basic ticketing engine that they need. The other funny thing is we've been presenting our fee structure a certain way for 12 years. And we tested. We proved we've been doing it backwards the entire time. The fixed versus percentage the whole time we've been leading with the percentage. But it turns out that our customers are more concerned with the fixed component of the price. Then finally we proved that even if we see a hit to the onboarding metric, in this case we proved that there is a range where it's actually not going to hit revenue because these are low value organizers. Yeah. So this is the team. This is the team that I work with. So I'll end there and take questions if anyone has questions. Can we pass the mic around or someone had a question? Yeah. My question was what percentage of your overall users that were going were you feeding through these tests? Was it just a smaller percentage? Did you try to kind of do a canary test or whatever where you only rolled it out to 5 or 10% or did you just do it on the whole user base? Yeah. So we just did a 50-50 A-B test. So our volume of new event organizers coming in is not that big. Whereas our growth teams that work on attendees, they could like turn something on for like one day and reach significance. Whereas for us it takes a while. So I'm coming from the product manager perspective where you mentioned that you see a huge initial hit to revenue and that's a scary thing for a lot of people, especially the executive team. What did your product manager do that was able to convince them of this and how long did it take before that actually happened? Yeah. So one thing we've – she basically was able to chart the sort of the hit to first publish rate. So the expected financial impact of that ranging between the worst number we saw during testing plus the least worst number. So you could see the cost to Eventbrite. She was able to chart that against the sort of upside, the growth we were expecting to see from assortments. Based on what we were actually seeing. And the test, like the revenue upside. And she was able to show that basically we could accept up to a certain percentage hit to this metric and still be profitable at the end because of the concept of assortments. So the good news is that it turned out it had no financial downside that hit. So that's the only good news. So how did you come up with your initial assortments to test? I mean, do you simply have a list and just go, well, we're just going to try all combinations? Or did you have something, some knowledge that fed some initial state or initial test? Yeah. So we had some ideas because we had been talking about it. And then we actually – we worked with a consulting company that's been doing assortment consulting for a lot of different companies called Simon Kushner. And they did surveying of our customers. And they were able to identify sort of the key differentiating features where, like, this is a feature that's absolutely essential even to the lowest value customer. And then we were able to sort of say, like, okay, these are the features they will get versus all of the features that only the higher value people want aren't given to that group. They're reserved for the people who go with premium. One more question. Okay. Thank you. Thank you. Apart from the revenue upside, if you broke it down into just extra revenue from the price increase – in fact, it's a little bit more expensive than that.