Growth Engineering (Speakers: Atlassian, Eventbrite, Pinterest)

Training Center for an 딱 Learning Center year Thank you. Thank you. Thank you. Thank you. Thank you. Hey, everyone. Thank you. Thank you. Excuse me. Can I have everyone's attention? Excuse me. We're going to be getting started here in just a minute, so please grab the last bite of food and start taking a seat. We'll be getting the talk started in just a minute. Okay? Alright. Thank you. So no Camillis building. No restaurants? Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Can you pass the mic around? My question was, What percentage of your overall users were you feeding through these tests? Was it just a smaller percentage? Did you try to kind of do a canary test or whatever where you only rolled it out to 5% or 10%? Or did you just do it on the whole user base? Yeah, so we just did a 50-50 A-B test. So our volume of new event organizers coming in is not that big. Whereas our growth teams that work on attendees, they could turn something on for one day and reach significance. Whereas for us, it takes a while. So I'm coming from the product manager perspective where you mentioned that you see a huge initial hit to revenue. And that's a scary thing for a lot of people, especially the executive team. What did your product manager do that was able to convince them of this? And how long did it take before that actually happened? Yeah, so one thing we've... She basically was able to chart the sort of the hit to first publish rate. So the expected financial impact of that. Ranging between the worst number we saw during testing plus the least worst number. So you can see the cost to event break. She was able to chart that against the sort of upside. The growth we were expecting to see from assortments. Based on what we were actually seeing in the tests. Like the revenue. And she was able to show that basically we could accept up to a certain percentage hit to this metric. And still be profitable at the end because of the concept of assortments. So the good news is that it turned out it had no financial downside that hit. So that's the only good news. How did you come up with your initial assortments to test? I mean, do you simply have a list and just go, well, we're just going to try all combinations? Or did you have something, some knowledge that fed some initial state or initial test? Yeah, so we had some ideas because we had been talking about it. And then we actually, we worked with a consulting company that's been doing assortment consulting for a lot of different companies. Called Simon Kushner. And they did surveying of our customers. And they're able to identify sort of the key differentiating features. Where like this is a feature that's absolutely essential even to the lowest value customer. And then we were able to sort of say like, okay, these are the features they will get. Versus all of the features that only the higher value people want aren't given to that group. They're reserved for the people who go with premium. One more question. Apart from the revenue upside, if you broke it down into just. Extra revenue from the price increase. And factored that out. Was there equal or more retention on the number of events people were actually creating? Or were they creating fewer because it cost more and it still worked out? So that's part of the challenge of doing A-B testing in this area. Because the time delay. So people sign up. They usually publish their first event within 14 days. And then sometimes those events don't go on sale for months. Like sometimes many months. And so you have to wait to see the actual revenue happen. Yeah. So that's why it took us four months to see like, all right, did that disaster actually turn into lost money for us or not? Cool. So I'll be around after the talk. So come hit me up if you want to talk about this. Thanks. Cool. Our next speaker. Our next speaker is a tech lead at Pinterest. He works on the emails and notifications team. And today he's going to talk about notification volume optimization. So please give a warm hand for Dr. Bozal. Thank you. Can you hear me? Okay. Hi, everyone. Welcome to the talk. My name is Bo. I'm going to talk about notification volume optimization at Pinterest. So the first, what's the problem we're trying to solve? Why do we even optimize notification volume? So at Pinterest, notification drives a significant amount of user engagement, like DAU and WAU. So the question becomes, what's the right frequency we send to users? And then basically there are two sides. If we send too few notifications, the engagement metric will hurt. Like DAU, WAU, they will all drop. On the other hand, if we send too frequently, first it will hurt user experience. And also, it will actually hurt long-term metrics of the site. Because, you know, people will get angry. They will unsubscribe your emails. They will delete your app. You know, Gmail, like several other email vendors, they will penalize your reputation. Like they will have a higher probability to... put your email into spam folder so that your users could not even see those emails. So all these consequences are... can be caused by sending too frequently. And also, it will hurt your brand. Like, in the user's mind. So the key to solve this problem would be to use machine learning to do more personalization. And based on whether the user actually liked the email, or whether you really need to send the email or notification to a user to decide the right frequency for each user. So this problem has been... We have been working on this for a while. And before I joined the company, we had the first version of machine learning-based volume control system. So, like, before that was even developed, we have basically no machine learning. And this is the first version using machine learning to control frequency. And when it was shipped, I think, early 2016, it has huge impact. What it does is basically... is something called dynamic cooldown. So I will explain in details. So basically we have many different types of recommendations at Pinterest. Like some are popular pins, some are like... the boards you recently followed, etc. So basically we can train machine learning models, like a CTR prediction model, to predict what's the probability that each user clicks different types of emails. Then, basically, for each type, and we can compute the user's CTR for that email type, and we can calculate the CTR's percentile among all users. And then we can decide something called cooldown days, based on the CTR percentile. The definition of a cooldown is that, say, you send this email type today, you cannot send the same email type in the next K days. The general idea is that if this user has higher CTR, we'll make the cooldown shorter, so that we can send that more frequently. So this idea all makes sense. So basically, every day we use these CTR models to rank all the emails that are not in the cooldown period, and we pick the highest one to send. Which, you know, is quite reasonable. Then after we get V1, we've been using it for a couple months, and then we start to realize some of the issues. The first one is that, actually, there's no direct control of the total volume a user gets in a period, like in a week. This has several consequences. For example, it's very difficult to do rigorous experiments. So at Pinterest, we always try to improve the content, or improve the recommendation algorithms, and a lot of this experiment could result in, like, increased coverage or volume of certain email type. And then, if we want to do experiments, the question comes, say you control an Enable group, and your Enable group has increased volume, you really don't know. Then you see, of course, typically if you increase volume from notification, you will see some metric gain, like weekly accurate user gains. But it's very difficult for you to know that whether those gains come from just the volume increase, or it's just actually come from quality improvements. So if you want to be more rigorous to design this kind of experiments, ideally you want to say that for the control group, if the Enable group has higher volume, we want to try to match the volume for the control group. For example, we can use the actual volume to send the best available existing email, so that control Enabled have the same volume, then we can really tell whether the gain come from quality. And also, you know, it's very difficult to iterate on ranking models. Sometimes we update our ranking model, we always, because the, you know, the cooldowns based on the CTR percentile, we always have different, the volume changes. So it's also difficult to judge whether your new models are better or not. And also it's very difficult to add new email types. Once we add new email types, we increase volume. So basically, let's come to the second problem, is that all these cooldown rules are difficult to optimize. And our objective function is unclear. For example, you know, why we set 14 days cooldown instead of 12 days, right? So it's not easy to directly, automatically optimize those parameters. And that's also, like, some email types are generally better globally. So as a result, they should be sent more often globally. But if we want to, you know, even have different parameters for different email types, that's even more complicated, which we don't want to do. Then we decide, okay, we may need to just rewrite the system with a better mechanism. The objective for this V2 system, we call it the budget-based system, is that we really want to have a direct control of total email volume or notification volume for each user in a week. That's basically, we decide a weekly budget. So this basically can help us improve experiments rigor and engineering efficiency, can allow us to easily add new emails, can allow us to easily test new ideas to optimize volume. And we also want to have some clear object functions that we want to optimize. Then we can basically use machine learning to directly optimize for it. So it's going to be automatic. There's no kind of manual process for tuning parameters. So the system looks as follows. We have a notification service running every day to process all users. When a user comes in, we basically, there's a module to something called budget pacer. The budget pacer, what the budget pacer does is, every week, the start of the week, it will load the user's budget or volume from a store. And it will try to pace that volume to every day. So the simplest pacing algorithm could be, like, even pacing, meaning you have, if you have three a week, you will pace it to Monday, Thursday, or Sunday or something, or Saturday. Then the user come in, it will ask the pacer whether today we have budget to send. It's more like an ad system, like, you know, the budget is really not money, but it's the number of notification we can send. So if we do not have budget, then we don't send. That's easy. If we have budget, then we go to the ranker to rank all the available emails and pick the best to send. And then there's offline component that take an email send, take the additional signal from the users, uh, um, basically, uh, to learn the machine learning models. And it uses the model to do the scoring, uh, we call it budget scoring, you know, every couple of days. So to compute the volume for each user and publish that volume to the store. So that's basically how things work right now. Um, so what's the, basically, what's the, um, object function we want to use to, you know, our machine learning models? Um, the first question is, what's the object, uh, what's the objective we want to optimize for? Um, let's say, uh, I want to directly optimize for, uh, site engagement, like, data active user. Then the simplest version, uh, so what does this mean, right? This means that for a fixed number of notification, you really want to directly optimize for data active user. Then this actually means that, uh, if the user is too active on your site organically, you probably don't want to spend your money on them because even you don't send notification, they come to the site anyway. Uh, why bother them, right? Uh, so the simplest version of this kind of, um, modeling is that we can have some, uh, utility score, uh, to capture what's really the utility of sending a notification. And one simple version could be, like, one minus the probability they come to the site organically times the probability they click notification. So this basically assumes that your notification channels, and your organic channel are independent. And all these probabilities can be learned by your machine learning models, right? You can train a CTR model, all your email clicks. You can train another model to predict how likely user to come to the site organically. Um, so we have two plots here. So, uh, the left plot, the x-axis is the basically the activity level of the users, basically the number of days they're active in the past 28 days. Uh, the y-axis is the CTR of emails. So generally you can see that when user come to the site more often, they might more likely to click email because they love Pinterest. They want to click, you know, everything that we recommend. Um, on the right side, so basically if you use this simple formula, uh, the x-axis is still the activity level, but the y-axis is this utility score. You observe that for users that are on the far right, meaning they're very active already, their score is kind of low, right? So, but on the extreme left side, like meaning the user don't even come to the site, their score is also not the highest. The user that have the highest score are among the casual region, where they are kind of active, but not too active. They're more like 4 to 15 days active. So those other user, you really need to use notification to get them more engaged. Uh, that gives you the best ROI. Um, so, um, this V0 is a good start. Uh, I guess, uh, if you want to start with something, you could use this. But for our purpose, we want to further improve it. Uh, there are several issues with the V0 score. Uh, the fundamental one is that it's making some, uh, too simple independent assumptions on the independent of the channels. Uh, sometimes, you know, it kind of makes sense, but actually your notification channel and your organic channel, they're quite dependent. Um, so as a result, this score kind of penalizes too much on very active users. But actually, for those users, a notification can still help, uh, drive them come back to the site. And also, if you want to use this to decide a weekly budget, um, meaning how many, how many you want to send in a week, then the question becomes, actually, the every send in a week, they are not independent, uh, as well. So, like, if you send one, uh, and you send the second, the second send is not, doesn't have the same probability to drive user come back to the site. Um, so basically, we want to further improve it, and we did V1. And in V1, we basically want to, uh, learn a full machine learning models to predict, uh, your target metrics, or we can call them rewards. These rewards can be pretty flexible, like could be DEU, could be you want to optimize your revenue, you can do something there. Uh, and how about what, what the machine learning model does is that it's kind of really learning a function. The function take a parameter of the users, and take a parameter of the number of notification you want to send, and the label or prediction of the function is the reward that you define. And after you are able to train this kind of machine learning models, then you can actually calculate the, uh, calculate the incremental value of every additional email. Additional email sent, email sent. And basically, you can select the most value, valuable ones among all users to send them. So basically, with this, you are able to find out the volume for every user. And, uh, for the features of this machine learning models, uh, historical site organic, organic visitation, uh, rates are very important, and the email not, or push notifications CTRs are important. One interesting thing to, uh, note is that, uh, like, Android iOS actually have very, uh, different CTR for push notification. So it's, uh, important to separate them into two features. Um, and also you want to calculate this kind of feature in different time windows to capture, like, the more recent behaviors versus, uh, the more longer behaviors. And also it's important to use, uh, the user's sign-up age, like, whether this user just recently signed up or there has been a long, uh, veteran user. So this will allow the model to have different prediction for different users. So, uh, when we observe, what the model does is that for really new users, uh, we really decrease the volume in the first couple of days compared with the control. So that because when users just sign up, we really don't have good signals to generate good recommendation. But overall, once the user gradually gets more signals, we gradually increase the, uh, volume. So that the long term, again, is high. And also we can have other site engagement features, like number pins, number goals, et cetera, that basically capture how likely the users are engaging with your site and some user demographic feature, like country, gender, age, et cetera. So here are some results, um, uh, compared with the control group. Uh, we actually, uh, trained... We treat our user into, uh, three separate groups. Like, uh, some users are... They only, uh, enable their email. Some are only enable... Only enable their push. Some are enable both channels. Um, overall, we can significantly decrease, uh, the notification volume, uh, overall. And also dramatically increase the CTR in both email and push. And that also drive, uh, like, uh, site engagement metric at the EU. Um, so basically we are able to, like, email, we drive 15%, increase CTR by 20%. Uh, push, we decrease 5%, increase CTR by 17%. And DAU, side-by-DAU is up, like, around 2%. And that's actually 10% of DAU driven by total, like, all notifications, uh, which is very significant. And also another interesting anecdotal, um, thing that happened is that because we treat, uh, Android and push, uh, Android and iOS in different signals, in the end, we end up having much higher impact on Android users. So when we ship this experiment, uh, it actually helped Pinterest Android DAU surpass the iOS for the first time in history, which is very interesting. So this analysis, um, so on the left are the, uh, email volume in the control group. The blue, um, plots, uh, are the volume for the control group, uh, for different activity levels. And the red ones are the optimized volume. And these two plots, uh, have the overall same average volume. So we can, they're more comparable. Um, so you can see that, uh, with the optimized version, we tend to send more to the people that are not, uh, like, core users. More like, but still, uh, uh, not extreme dormant users. Like, for, on the far left, which a significant amount of users that are extremely dormant will actually drop their volume. But for the people that are kind of casual, we increase the volume. Uh, so this means that the new algorithm make your, uh, each notification more, have higher utility to, to a final metric. On the right side, we basically, this is more like from user, uh, experience perspective. We look at the access, access, x-axis is the email open rates, or CTR, of the users. And the y-axis are the volume, uh, for the optimized version and the control version. Uh, so we're also able to see that for people who generally like to click email, they receive, end up receiving a little bit more. And the people on the far left, like, close to zero, meaning that they never open email, they, uh, even receive much less than control. So this two plots generally show that, uh, the left shows that the new system is gonna, is basically able to make the, uh, notification more, each send more valuable to the site. The, on the right plot shows that even from the user perspective, it, uh, feels more personal. Um, so conclusions, we, uh, recently did this V2 volume control system by doing some global optimization, uh, uh, with machine learning. So we are able to directly control the total notification volume a user gets. Uh, we end up reducing a lot of volumes and increase CTR and user experience. And we improve the notification utility to start a wide engagement. And, uh, this also help us improve experiments rigor and, uh, engineering efficiency. And we make it easy to test further ideas. Uh, this is the collaboration pro-project. Uh, you know, CollegeMan, Gratial, Berkay, uh, which are both engineering on the team, and Jones, that's, um, that's all. Thank you. Any questions? Yes. Yeah. Yeah. Last one. No? Good. Okay. Thank you. All right. Well, I'm excited for our last talk of the night. Uh, so our last talk is from, uh, our hosts here at Atlassian. And we have Josh, who is a principal product manager at Atlassian. He's been at Atlassian for six years, and he's gonna talk about, uh, some of the work that they do on the growth team here. Please welcome Josh. All right. Can everyone hear me? Sweet. Um, so I'd like to thank John for organizing and Alex for, uh, helping set up. Um, it's really good to have a, a meetup like this where we can all share, uh, different stories. Um, today, as John said, uh, I'm a product manager here at Atlassian. I've worked, uh, in product in Jira and HipChat, and I also, uh, currently work in the core growth team, which sits separately from product. Um, today I'm gonna be talking to you about becoming a growth car salesman. It'll become clear in a couple of minutes. Um, and some of the things I'm gonna be talking about today are about product expansion. As a lot of you probably know Atlassian, we have multiple products, and expanding users from one product to the next product is very important to us. Um, I'm gonna walk you through a specific journey, uh, a Confluence journey that we have been experimenting with a lot and some of the results, um, as well as some of the lessons that we've learned while we've been doing that. And we're gonna have a special appearance by this guy throughout the talk. Um, so at Atlassian, if you're not familiar with us, um, we're, our mission is to unleash the potential in every team. And really, we're all about enabling team collaboration. We have a whole set of products that, uh, solve a lot of different problems. Um, we recognize that teams don't just have one problem that needs to be solved. They have many, many different problems. And not just in tech, teams exist in lots of different industries. Um, and, uh, having that approach and being able to, uh, have a whole bunch of different products gives us a special, uh, ability to be able to solve many different problems that teams have. So product expansion, uh, we define this as when we have people who are using one product and we're able to get them to start using a second product. We really want to do this to, uh, to solve, help teams solve the problems that they have. We do not want to be the car salesman. Uh, we do not want to, uh, basically be pushing things on you that you do not need. We really, truly want to be able to help you in, uh, all of the problems that you're trying to solve. Um, again, I mentioned before, Jira Software to Confluence is an example here. If you're using Jira, Confluence, uh, is, uh, a good product that sits beside it that solves similar and related problems and they integrate well together. Um, but it's not just product expansion. Um, a lot of you probably work at companies that only have one product. Um, but it also applies to ecosystem. If you're trying to get people to use, uh, integrations that you've built with other companies, or if you have paywalls in your app and you need people to go from one stage to the next, you need to be able to communicate the value. Uh, and it's, it's kind of an expansion. The thing that we do not want to do is be this guy and really be sleazy and salesy and just keep pushing things at you when we know you don't want them. Um, so we try to, uh, do a lot, um, with targeting and segmentation and things like that to make sure that, uh, we're doing it in the best way possible. But we know that from our customers, people use our products together. An example is the Jet Propulsion Laboratory at NASA. They use it to, uh, put software on the Mars rover. Uh, Spotify uses it as their internal service desk, and they use Jira's service desk and Confluence together. And Cochlear, which is an Australian company that helps, uh, deaf people be able to hear, uh, with cochlear implants, um, they actually develop their software for these implants, uh, using Jira software, Confluence, and Bitbucket. So we already know that there's a natural baseline here, and our goal is to increase that baseline. We don't want to, uh, again, just keep throwing things at you, but there is a lot of customers out there who probably use one of our products that don't know about other products. They don't know what the other products solve and what jobs they have to be done. And in a graph, this is, uh, this is kind of what we're trying to do here. So if we have, uh, products one and two at the top, so we can see that the, the MAU for these products is, uh, growing. Um, and then the blue line represents the users that have both products. And really, we want to be able to inflect the blue line. Um, we want to have more people using both products. Um, and instead of just it growing at a, a linear rate, uh, we basically want to be able to, uh, help more teams, uh, solve their problems, as well as it's obviously a good, uh, business effect for us as well, to have more people using, uh, multiple products because it makes our products stickier. Uh, Jira Software and Confluence is the example that I'm going to walk through now. I'm going to give you a couple of examples of, um, experiments that we've run, uh, and some learnings. Um, I'm pretty sure most people here probably know what Jira is, but it's an agile project management tool, allows you to do backlog planning, et cetera. A whole bunch of you probably use it. Confluence is a wiki, allows you to do product specs and things like that. So this is the first experiment that we ran. Um, it's basically in Jira. Uh, if you put a longer, um, description in, we actually have a system. We call it the engagement platform. Um, and it's a segmentation system and notification delivery system that we have in our applications that allows us to do very targeted delivery. Uh, we use a third-party system, a segmentation system called Lytics, um, and we basically pump, uh, using, uh, Kinesis and Lambda, we pump a whole bunch of events into Lytics, and they're able to use those segments that we create in that from behavioral events, um, to target these different, uh, notifications to people. So this one is specifically targeting people, I think, who had created five issues in the past week, so we can see that you're actually doing something. Um, and we're basically saying, hey, you put a whole bunch of stuff in, um, would you like to try Confluence? Because it might be better for what you're trying to do. Um, the second thing we did here was we know that Confluence is a good replacement from our research for people who are using Word docs. I know it doesn't necessarily work for everyone, it doesn't necessarily apply, I don't know if anyone here uses Word docs anymore, but in a lot of other industries people do, and, uh, it's a good, it's a good way of telling people that, hey, instead of uploading Word docs, do you know that there's another product? Um, the first problem we ran into wasn't an external problem, though, it was the product teams themselves basically saying, hey, I don't know if anyone else has been in this position, but they're like, hey, this is mine, don't touch my shit, my product's my product, like, don't come in and, like, put stuff in it and annoy our users, et cetera. Um, and at the same time, they're basically like, ah, we're not sure about those experiments, are they really gonna work, et cetera, so, um, the first problem we actually had to solve was an internal one, and we solved this by using a SWOT team, and a SWOT team is basically the growth team, the marketing team, and the product team who come together. We have daily stand-ups, um, and we're able to, uh, basically stay on the same page. We have a bi-weekly planning meeting that allows us to, uh, plan, use i-scoring, that type of stuff, to plan what, uh, experiments we want to run. The biggest thing this does is gives all three teams a sense of ownership, and there's nothing better than having people feel like they own these experiments and feel like that they're involved in what growth's doing. So the first lesson is to work closely with product. Bit of an obvious one, but we really found it to be helpful, and we really found that once we started using SWOT teams across the company, especially if you're at a bigger company, um, it to work really well for communication and ownership. So, we got the results back from those first two experiments, and we saw that there was actually a lot of clicks. So, um, the green is the control, but because there was no dialogue in the control group, there's no clicks. Um, so we got a whole bunch of clicks. Uh, our evaluations stayed flat, and really what we were trying to look for is like, hey, if all of these people are clicking on this, um, message, why aren't they converting? And so we were really puzzled by this, we didn't know why, and we kind of did a whole bunch of studies. Uh, we did a bit of research, and we looked into the analytics. Um, and it turns out that it comes down to different missions. So, in our products, we have people who are end users, who create issues and, uh, use, uh, edit issues, use the backlog, etc. And then admins who are used to configure Jira, configure user management, etc. And it turns out that, um, these end users are the people who actually want the software, and who want the second product, um, but they have to convince the admin, and then the admin has to turn it on, and then they have to tell the user. And when you get to bigger companies, and when you're like a 10 or 15, 20 person company, it's okay, because you can walk across the hall. When you're a 500 person company, it doesn't work as well. Um, and when you actually think about this, it's a funnel. Right? And the biggest problem in this funnel is this part, because I know when I evaluate things, if I look at a product, and I have to go and talk to IT, I'm like, ah, I'm done. Like, there's no point. Uh, it's not worth it. Um, and this is what actually was happening. So, we, that was our hypothesis, and we went after it, and we said, okay, using this engagement platform that we have, because we have the ability to send notifications in product, um, let's target the admins with some messages. So, when the user requests confluence, the user's like, hey, I actually want this thing. It sounds kind of cool. We actually started to send the admin some messages, and we said, hey, in product, hey, this is, um, your user would like this. Um, and we started to send some emails as well. Um, and so we were able to actually change. We saw a 22% increase in the number of evaluations that we had over the control, um, which was really good. The team was quite happy that we were able, that our hypothesis was correct, and we were going after the right thing. Um, but the, the trouble is that activation still really didn't change. Um, so we had to start thinking about what the next step was. The, the second takeaway here is really to understand the micro-funnels. Um, and we all talk about the double A, triple R funnel, and like high-level funnels that we all model in our products, but really, funnels lead onto funnels, lead onto funnels, and there's even micro ones in this scenario where users can't do something because they're blocked by the product, and trying to unblock it in specific ways really helped. But this isn't all we want to do. We've talked up to now about getting people from Jira into Confluence, but we don't just want people to go to Confluence and then leave. We want them to kind of hang around and actually solve their problems. We really want to do, uh, proper Confluence activation, uh, and then cross-product integration. So the next iteration, there's a lot of iterations in here as well that I'm not necessarily talking about, but the next iteration that worked for us, um, this is in Jira, and what we did was we actually started to show some of the value of Confluence in Jira, and like a powered by Confluence style thing. And you can see on the right, there's a couple of different things, so you can do a Jira report, um, or a decision, or a sprint plan, and things like that. Things that Confluence is really strong at. And when the user came in here, and clicked this, we'd tell them, hey, like, there's some information about Confluence, but actually you don't have it, would you like to have it? Um, and this iteration, we actually, down the bottom here, I don't know if you can see it, but we actually allowed people, the end users, to put a message to their administrator in. Uh, to make it more personal, and we started to use things like avatars to basically say, hey, um, I'd really like this. Hey Will, you're an admin, I'd really like if I could have this product. Um, but we didn't stop there. On the, on the administrator side, we allowed, uh, sorry, we presented a dialogue that essentially tells the admin a bit more about Confluence, and it says, hey, you're about to enter a trial, um, would you like to do this? Um, we found out that 52% of people did this, um, and this was the second screen that we presented. It's a bit small, so I'll read it. It says, would you like to give, uh, access to the users that you already have to Confluence? And if you're gonna give them access, would you like to notify them? So this was like a little growth step that we did that basically not only allowed us to acquire the Confluence product, but helped us to cross-acquire all of the users. In this case, there were 64 users in the Jira instance, and we really wanted to get them into the Confluence side. Um, so the results came back, and we were blown away. We had a hundred and forty-five percent increase in trials, and it was astonishing. I, unfortunately, I can't share the real numbers, the raw numbers, but this was the increase. Um, I can tell you that it was uh, widely talked about at Atlassian, and we've refocused a whole bunch of teams on projects similar to this because of it. So, um, it was really, really valuable, um, and even better, we saw an increase in activations. It wasn't as high, um, so it didn't follow the whole way through, which means there's more blockers down the funnel, um, but because we focused on value to the user, uh, we were really able to, uh, get this flow working, uh, much better. So that's the third takeaway, um, as a product manager. I tell my engineering teams that all the time, um, but it, it really shows through when you do something like that and show how the, the, uh, value, uh, works. We're still iterating, um, we have things to do, like, uh, activation on the Confluence side of things. Maybe we can do user, use some of the data from Jira and actually personalize the onboarding and that type of stuff. Um, there's a whole bunch of experiments that we can try. Um, the three takeaways, work with product, think about SWOT teams, think about people, uh, communication and ownership of, of, of, from all three teams, product marketing and growth. Um, think about your micro-funnels, like, really think about how everything can be modeled as a funnel, cause I, I bet you most things can. Um, and finally focus on value, uh, to the user and really communicate that in the experiments that you're doing. And hopefully, uh, we can come across uh, not like this, uh, and a little bit more like this. Um, thank you. Any questions? Should we take the mic? Right, so the question is about how do the SWOT teams work in practice. Do we get one representative or does just like everyone pile in? Generally what we do is we'll get the, the growth team. So you'll generally have like three or four or five engineers, maybe a product manager, uh, maybe a designer. Um, and they will all go. And then you'll get one person from product, and or two people from product, and one person from marketing. Um, and then marketing can give you the content, the copy, that type of stuff, and product can give you the subject matter expertise, etc. And we do like daily stand-ups is, is very valuable. Um, I don't know how many of you work in companies that have different locations, but, uh, we, as you can tell, I'm Australian. We have a Sydney office, we have an office here, we have an office in Mountain View, we have an office in Austin and New York. It, uh, there's a lot of time zones to cover. Uh, uh, a lot more FaceTime, a lot more meetings. Um, we uh, use chat a lot. We have a lot of VC calls. Um, and like really being diligent about the time zone thing. Like Sydney doesn't come on until 3 o'clock San Francisco time, so you kind of have to make sure that they know that you have a question, they have to answer it overnight. Um, doesn't always work. Haven't, we haven't solved it. Yeah? So your, I liked your slide about like territorialism being a challenge. Yeah. But do you ever think that like for your users, they're barely even aware that JIRA and Confluence aren't the same thing? And so you're, you're bringing into this an assumption that your users care that these are separate products? Like can, can you imagine just thinking of it as just a single product with a total new fixing structure? Yeah, right. So, um, we haven't, we've talked about doing an experiment, so the question was about like, instead of it being two separate products, why not just make them features of the same product? Um, I think Atlassian's mission's always been to, uh, allow you to use whatever pieces you need, and that's why we've always kept separate products. Um, I think most of you probably know that we were a behind the firewall software company for a long time, and we've transitioned into the cloud, and a lot of our services are now deployed in the cloud. Um, as we do that, I think that things like this start to change, right? You don't need to just, you don't need to actually buy something and download something, you just get features. So like, do the products matter as much anymore? And I think that that, no one has an answer for that at the moment, but like, they're big questions that we're thinking about, and we might even try some experiments where we do that. We take the product names away. I heard from your story about getting the admins to give permission for the users to use the app, and then the option to inform the user that they had, in their wisdom, given permission? I had heard from that that letting the admins take credit for turning on the features was actually part of it, so is that a correct assumption on my part, or am I getting too psychological? I definitely think there's a bit of psychology here. Um, I don't know if, I think in a lot of companies, people actually don't know the admin, right? Like, a lot of the time, last time we got 2,500 people, I don't know who the, there's a bunch of IT admins, there's a whole team, I don't know who they are. Um, so, maybe in smaller companies where the admin can be like, hey, I turned this on for you all, and I'm helping you solve a problem, we definitely see that with champions, we call them champions, and we see that all the time, where someone goes in and they're like, right, this company doesn't know how to collaborate well, because they've got products all over the place, let's standardize on a bunch of stuff, and that really works. Um, so maybe in this case. Uh, we got one more question. Yes, we have. Um, I don't know how much I can talk about it, actually, but, uh, oops, uh, we have tried to do a trial, seven days, fourteen days, thirty days, and... I don't know what I'm doing. Alright, uh, seven days, fourteen days, and thirty days. Um, currently, it depends, because when you buy one of our products to start with, it's actually a seven-day trial, but when you already have one of our products, and you add another one, it's just, you get thirty days, because you're already a customer. Um, we found that actually there's no real significant difference between thirty and seven. So we were like, okay, well, let's just do seven, because it's better for the company from a revenue standpoint, like you get it earlier, um, and like there was no difference in the activation rates or anything. Cool. Thanks, everyone. Alright, uh, so I wanted to first thank all our speakers today, uh, Josh, Nick, and Beau, uh, for coming and sharing a little bit about what each of their teams are working on. I also wanted to thank, uh, Josh for offering to host the event here at Atlassian, and Atlassian's workplace team for helping set everything up. Uh, that's all we had tonight, so thank you all for coming. Um, you can hang out and, uh, grab any pizza or drinks that are left over. Um, we just have to be out of here by nine, uh, but feel free to hang out for the next hour. Thank you, everyone. Applause Applause Thanks.