Eric Muntz, VP of Product - MailChimp

Would you be so kind, To show them what love is, Cause somewhere in their mind, Our combat can be used Ke ін the part of many these things Once upon a time Somebody she told me That when it hits you, boy It's gonna be forever Yeah, yeah, yeah You know that I I know That love Don't just me I I know That love Don't just me I know I know Yeah Yeah Yeah Yeah Yeah Yeah Yeah Yeah Lord, I know I'm trying to hurt nobody. Yeah. But this beaten track I'm climbing seems to be drafted from the tears of the many. Trying to find the one, keep on the happy, and I'm going to set you down. Because you know I let you down. I'm going to let you go. In this hope one day you will know. Yeah. I know that I, I know. Lord, come to just me. I, I know. Lord. Lord. Just me. I, I know. Love. Love. Surrounding me. I, I know. Love. Love. Love. Love. There's so many. Afraid. I know ourlonette, here there. Always. Alert my bias. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. of them is here in the crowd. When I was building this talk the first time, I asked for some help making the slides look great. And then I kind of decided I needed to be true to myself and I needed to really present my ideas and my thoughts. And if I let one of our amazing designers make this look amazing, I'd be lying to you. So I don't want to lie to you at all. The other great thing, I asked my creative director, you know, what should I do? I'm not a designer. How do I make my slides look good and resonate with designers? His advice was intentionally make it shitty, which I think I've done a pretty good job of doing. So hopefully you can bear with these slides and you've been warned. So quick about MailChimp, just to add to that. Our mission is to empower small business. Our core values are humility, creativity, and independence. And a lot of times when people ask me what I really, really love about my job and about MailChimp, it's the creativity and independence piece that I really love. I work at a company that creativity is not reserved for designers. Designers are going to be super creative. It's what you're supposed to be doing. But at MailChimp, we value it for everyone. We want our QA members, our support agents, our developers, we want everyone to be creative. And I just absolutely love that. The independence piece is one of the things I feel, I can really help control because I'm in the position to build process and figure out how teams work together. And to figure out how teams can work together independently while being creative to build these great products is something that I just love being challenged to do. I just have this little extra piece in there where I bold HCI because I think it's kind of awesome that we accidentally had our core values be humility, creativity, and independence. And the acronym there, HCI, is the same in human computing. So I just love being able to do that. So we had a hundred requests. We had a hundred servers that we managed. And kind of the piece that I really love, that's the big number here in this talk, is that we had five back-end, front-end developers. So those are developers who were writing HTML, CSS, JavaScript, and then our developers who were working on the back-end. And really, we had one product designer. And that's probably even a cheat because that product designer was also doing our front-end, HTML, CSS, working on everything. So flash forward to July 2015. We had 8 million active users. It's kind of awesome because this is two months ago and it's 10% out of date already. We're over 9 million active users now. In July of 2015, we sent 11.9 billion emails. We served 10.1 billion requests. We have a thousand servers now. And then the final two numbers, we now have about 20 back-end front-end developers and about six product designers. These numbers aren't here just to be all braggy, right? Like, look at all we do. MailChimp's awesome. Which I'm very proud of the numbers. But what I'm really, really proud of is kind of the last two numbers, those teams. The fact that we're able to do all of this and really make a difference for our customers with a pretty small team. And that team being creative and independent and really working well together. Our designers, our QA team, our developers, you know, we really try to make sure that those teams are gelling together and working right toward the same common mission. So that's, I'm proud of all the numbers. But the last two numbers are really, really what I'm proud of. So to get into what I'm going to talk about, there's this notion in software that we need to build the minimum viable product, right? This is something that's become really popular over the last, you know, 10 years. Maybe decade or so. There's a gentleman named Eric Ries who wrote a book called The Lean Startup. And he talks a ton about the minimum viable product. Lots of companies have been working on this and thinking about it. And, you know, if you're in a startup, truly in the startup world, this is something people just talk on and on and on about. What's the smallest working usable product that we can get to users? And it's great. It's helped a ton in the product world with building great simple products. The problem that I find is that we do it, we start working on this product, this minimum viable product, and we either do it with absolutely no process at all, no thought about the process, or we do it with like massive, huge, amazing, enterprise-grade process, right? It's like you're working at the government where you're just shuffling paper everywhere while you're building your product. And that is a disservice, right? Because something poorly executed, it's never going to be built. It's never going to be awesome, right? Something that's maybe a mediocre idea that you really execute really well, and the entire team is really, really focused and gets it done right, is going to resonate better with users than something that you just kind of slapped together and threw around. So my question is, what's the minimum viable process? Is there just enough process that's going to really resonate and really help you execute and help you build the right product for the right users the right way? So that's what I'm going to get into here. Now, last disclaimer, it's not one size fits all. And, you know, I get to do my job here as the dumb American dumbing down the room with Tommy Boy. I don't know who has seen this movie, but the one size fits all, this is like the best gift for me ever. So this is the fat man in a little coat, right? He's singing this song. And what happens here in a second is he kind of does this, and the coat just splits apart, right? So it really... Yeah, ha ha ha, funny, silly gift, but it really does really... What's the wording I'm looking for? He's really the perfect example of not one size fits all, right? And particularly with product. If that jacket is the product, as soon as he bends down and rips it, that's what's going to happen to your product. If you're building it with a process that just doesn't fit you or your users, or how you need to be working. So why have process at all? Why even bother? Who cares, right? Maybe just have no process. The reasoning that I think is super, super important is because your process is going to help maintain your core values as a company and as a group, and to make sure that you're really respecting trust. Right? Trust is this huge thing I'm nerding out on lately. And there's trust kind of from two aspects here. There's trust between your product and your users. I believe really, really strongly that when a user chooses your product, because let's face it, nothing's unique, right? Any product that anyone signs up for, they have made a choice. People choose MailChimp, they could have chosen Constant Contact, Campaign Monitor, tons of others. When they've said, I'm going to choose MailChimp, they have formed a trust relationship. They have formed a trust relationship with MailChimp. They have trusted that our product is going to continue to be robust, it's going to be accurate, it's going to be performant. But even bigger is that it's going to remain relevant. So if email marketing changes, or how small businesses need to use email changes, our users are trusting us to change with it. So you have to build a process around maintaining that trust. The other piece is that we've hired employees. And when you also have that same contract with your employees, yes, join our company, they are putting trust in you. They're putting trust that their skills are going to remain relevant. Never mind what happens in email, because they have to design with tables and inline styles. But designers as well. So if a designer agrees to join your company and work on your product, they are trusting you. They are trusting that they're going to be able to design for your product. They're not just going to get the gift from earlier that's like, right, no. Everything you do gets a big no. They're really trusting that you're going to value them. That you're going to value their input. You're going to allow them to move your product forward. So the way that I'm looking at process and figuring out what process is going to work is all based around this. Does it match our core values as a company? As a product team? And does it ensure that our users are going to continue trusting us? And that the employees are going to continue trusting us? So MailChimp is Agile, but with a lowercase a. I put that disclaimer in there because Agile has become this big, huge, massive, monstrous thing. I gave this talk at the front end conference in Zurich in August. And someone pulled me aside and they were, it was kind of awesome. He was a little offended, right? That I'm the one saying, no, Agile with a lowercase a. I'm almost apologizing about Agile. And he had a great point. The point is that the Agile manifesto is actually really, really small. Right? There's basically four key points in the Agile manifesto. It's instead of trusting on mega, mega process and tools, you trust individuals and collaboration. It's that you build the smallest piece of software instead of build tons of documentation and having to do all that around that. And it's that you focus on change. Right? You focus on listening and changing. And you do that by customer engagement. Right? I like to use the word user instead of customer. But it's focused on users and listening to what your users need and being able to change. So, yeah, I apologize a little bit for Agile. But that's just because Agile has become this massive, amazingly huge thing. And it doesn't need to be. And, of course, we believe in small project teams. Maybe two to five members. Mixed discipline folks. And even those team members are mixed discipline. So, you know, I say we have six product designers right now. They are doing visual design. They're working on colors. They are working on wire frames. They're working on user interactions. Some of them have started delving into HTML and CSS. Our development team. Our back end developers. They're working on HTML and CSS as well as, you know, back end database queries and all of that. But we try to stick anywhere from about two to five of them in a room. And when we can, physically in a room. And then we give them broad brush stroke and say, go. You got this. We trust you. Here's generally what we want. You fill in the details. So broad requirements. To me, that really enforces independence. Right? You basically say, I want something kind of like this. And then you trust that they're going to fill in the details properly. People ask often about requirements. Like, how do you deal with requirements? How do you make sure when there's small little teams that all the requirements are met? I just don't believe in writing and redefining continually writing, writing, writing, writing, writing requirements. The product is going to be the requirements when it's built. And you have no idea until it's built. Right? You start wire framing. Those wire frames are going to change. You build prototypes. Those prototypes are going to change. For me, the design and the prototypes are the requirements. It's all anybody needs to know how to test it, to know whether it's going to work. You don't need to write tons and tons of documentation. I mean, really, the app is the requirement. Does it work? So I say we have no product or project managers. And a lot of people look at me like, you know, that thing that dogs do when you, like, make a funky noise. And they're like, how does that? I've realized that when I say we have none or no product or project managers, what I really mean is everyone is a project and product manager. And that means that everyone on the team doing any bit of work, QA, writing documentation, writing code, whatever they're doing, they have to be focused on the product. They have to be focused on the end experience for our users because that's really what matters. So instead of saying we have none, we have none. We have 450 or so. So the only other little bit on those teams is that you need to assign at least one person. And usually it needs to just be one person who's the directly responsible individual. I stole this from Google. DRI is a term they use. I try not to use the acronym. But what this allows you to do is each of those project teams have one person who's a tiebreaker, right? Because let's face it. Arguments are going to happen. Discussions are going to happen. You're never going to know exactly what to do. Make one person directly responsible. And then you make sure all of those directly responsible folks get together and work together so that you don't end up building a bunch of countries, right? A bunch of countries with tanks pointed at each other. That's never fun. So here's an example. The thing I love about this example is this is our editor. So in about the course of three months, we went from an editor that was a little bit more experienced. That was just showed you the email. And you had to click on regions to edit things to one where you can drag and drop and do really fancy fun things. The reason I love this example is I read about a usability research approach that's instead of focusing on what doesn't work, you focus on what does work. And then you can from that just suss out. You kind of know what didn't work because it wasn't in the what does work pile. And you take everything that doesn't work. And you kind of map it to what does work. So this was the first time right after this project was completed, I started thinking about, well, we're growing. How do I build process? How do I build this for our team? And luckily, I just thought, well, I'll just mirror on what did work. And this was a project that really, really did work. We stuck four people in a room, gave them a broad brush stroke. We said, we want drag and drop. That's pretty much it. Just drag and drop. Make that work. And they realized it. We were insane. And there's so much more that had to be done. And they came out three months later with this, which was just phenomenal. It's been so, so, so great for our users. So this is our example of why small teams work and how we make them work. So how do we make this work? It sounds a little chaotic. But I believe there's some core building blocks that are really fundamental to making a team and a structure like this work. One of them is a staging environment. So this is utterly, utterly huge. Development and production, they are two separate things. I don't care how many times a developer tells you, it'll work in production. Until you see it work in a production environment, it is not going to work in a production environment. Trust me. I was a developer for 15 years. We're all liars. Don't believe us. It is not going to work until you've seen it work somewhere like production. It really needs to be production. My dream, actually, is that we have some real live users, maybe even internal MailChimp users, when you get newsletters from us, that those are run in our staging environment. Because without it really being real production, it is not going to work. Horrible things, horrible things happen like this. These are the kind of things that work in dev that don't in production. Right? I really wish that this screenshot was fake. I really wish that I just went into my browser and was like, meh, let me fake this out. This was a legitimate screenshot I took from the application because we pushed the release out and we didn't have media builds done properly. You don't do media builds in dev. You do them in production. Things like CDN. We use Akamai for CDN. You don't do that in development. Right? You don't do that in the flow to hook Akamai up to your development environment. I don't even know how you would do that, to be honest. But you do it in staging. And if you push it to staging, then you know for sure it's really going to work. Right? The big thing here is what happens to a user's trust when they log into MailChimp and they see that. It starts going down. Right? The more you have fails like this because you didn't have the tiny little bit of process in place, the more you erode that trust. And they start thinking, what's a different email service provider? Who else can I go to? Because this is ridiculous. These guys have no idea what they're doing. I take full responsibility for this one. This was me. It actually happened. I pushed it and then I went to lunch. So that's another thing. Don't push and go to lunch. Ever. That's just like awful. So just a few quick notes on what I think a staging environment has to be. It has to be a true mirror of production. It can't be like kind of like production. No, it has to be truly for real the same as production. And you have to easily be able to push code to it. Like I said, ideally really for real, for real, for real used. We're not there yet. But soon. Coming soon. So the other big building block is a release cycle. You really need to define a start and an end to what you're going to be doing. The benefits here are it really handles scope. So we have a holiday in the States called Halloween. I don't know. Is Halloween. I know hell loves Halloween. But I've been thinking of showing up to work as scope control or as scope creep for Halloween. Scope is like a mouthwash and I could dress like that and like put a mustache on and be creepy or something. Because what's funny is I'm up here telling you to have scope control with release cycles. But I'm like Mr. Scope Creep. I'm the guy who's like, can't we sneak in just this one little extra thing? But if you really force yourself, no, we have a cycle and this is how it starts and this is how it ends and these are the phases of it. Then you're going to really help your product be robust and you're going to help your users not have to deal with nonsense. Not have to deal with broken products or half-baked products because some VP dude said sneak this thing at the last minute. Again, it's not one-sided. It's one size fits all, kind of whatever works for you. But for us, we break all of our projects into four cycles. There's a research phase and then a design phase and then an implementation phase and then a maintenance phase. And all of our projects go through those four phases. Sometimes they're all kind of mushed together because it's all kind of the same thing. Sometimes maybe it's front loaded where research and design is really happening and then implementation happens weeks, sometimes even months after that. But just make sure that you've got, you know, you're really thinking about it. You know, you're really thinking about the different parts of what you need to do in your release cycle. I like to have a great win example of what you can do in a release cycle. This was MailChimp in early 2013. We thought it was pretty awesome. Woo, look at all the blue, super blue. I guess there's like a why so blue joke in there somewhere. And in basically one release cycle with all of that same stuff, research, design, implementation, and then some maintenance, heavy maintenance. We went to this. So we flattened. The other part that's not shown here that I really need to grab a screenshot of for this is this is also responsive. So we went from something that didn't work at all on mobile, didn't work at all on tablets to really being nice, responsive. It's really clean. You can even see the marketing message changed. Easy, email newsletters. Send a better email. Right? We turned around a whole new approach to marketing. Whole new design. Whole new everything. We were able to build a pattern library that took us from that tons of blue to this much prettier design. This is another one I really love because it happened in one cycle and interns did it. And I just love that we were able to take interns, two college kids, for three months. This is kind of the culmination of what they learned at MailChimp. They took our import process and they completely revamped it and got it built within one five-week cycle. So QA to me is a massive, massive, massive building block in your process. The big thing for me that I'll get to in a second is that QA doesn't have to be a bunch of 20, 30 people. It's a specific ratio of designers and testers to QA members. The only key for QA is that it can't be the people working on it. Because face it, we all love our babies. We're not going to find out if it's not happy day. And this is my favorite gift ever in the world for what it's like when you have built something and someone else grabs it. Right? Like this kid does not want to hurt his little Spider-Man. Right? And the other kid really does want to hurt it. And you can see like this is one last little frame in here where the big kid is like smiling because he knows. It's like this moment where everyone knows what he's going to do with the pinata. And if you've ever handed your code over or your system over to testers, you know the feeling that everyone's having here. Or if you've worked in QA, you know the feeling that the big kid's having. I just love this. And if you haven't been to devopsreactions.tumblr.com, it's the best site ever. You can waste hours and hours laughing at that. The biggest thing though for me is that first bullet. So test like users, not like testers. It is the easiest way to make sure that your users are going to trust you at the end and to reduce the amount of effort and requirements and everything needed in a QA process. I've worked in the government before. And what I would do was anytime there was any page that had any field on it, they would stick the Declaration of Independence in that field and hit submit to see what happened. Because it's like, ooh, are you properly handling the amount of data that we could shove in here? And every time that blew up, my response was, don't do that. Users don't do that. They're never going to do that. They're never going to stick like just pages of crap in a text field. Why is anyone wasting our time dealing with this? If a user comes into support at Mailchimp and they've done that, they would just say, well, don't do that. Your name is not the Declaration of Independence, right? Like your name is shorter. Just don't do that. So our users, our QA team, they test like users. And for us, one of the best ways to really get people to do that is to test like users. And one of the ways we help enforce that is through hiring. We hire into QA from support. So people who have been in support for a year and have dealt with the things that our users really struggle with and need to deal with, they're the best at testing because they know what those users struggle with all the time. And so they come over and they're like little mini product managers. And like I said, everyone's a product manager. It's perfect because they know exactly what our users are going to do. They know how they're going to break it. They know what they're going to struggle with. It's so great. And the other bit about not having requirements, our QA team, they're now built to where they don't need them. Their whole focus is, is this going to be understandable? Is this going to be usable? They don't care if the button was six pixels to the left of where the requirements said the button should be. If they can click it and it does the job it did after not entering the Declaration of Independence, then it passed. It's awesome. But we get these great bug reports where a QA member will say, this is unusable. This is confusing. This doesn't match our pattern. It works, but it doesn't match the pattern from how users usually accomplish their jobs. And that's like the most amazing kind of QA you can possibly get. And obviously, I've been saying trust a lot. Trust your QA team. They really know what they're talking about. I have an example coming in a minute. Yeah. So, QA, they just understand our users. Make them understand your users. We do usability testing. We do a lot of it. Every single time we run a usability test where there's an actual user using it, a QA member is present. And it just, sometimes they even run it. And we hire into the usability team from QA. It's like this funnel. It's amazing. It's really useful. So, a couple little tidbits. I have a few rules to go along with those building blocks. I think that those building blocks are really great, but there's a few rules that if you also adhere to with those building blocks, it can be really, really helpful. My first rule is never hide a problem. I actually follow this rule in life. When I say never hide a problem, I don't just mean like, oh, something broke on the servers and let me fix it. I mean anything is wrong in the company. Process is wrong. Things aren't working well. We moved to a new office and I don't like where I'm sitting, which happened this year. Just never hide a problem. If you speak up and say, hey, there's a problem, then people can solve it. If you hide it, people can't solve it. And I can't imagine anything worse as a user of a product than knowing there was a problem and the company pretending that problem didn't exist. Right? If you just come out front and say, hey, we have a problem. Sorry, our bad. We're doing our best to fix it. You maintain their trust. If there was a problem and they like sweep it under the rug and pretend it didn't happen, that's not like trust goes down a little bit. That's trust goes away. Trust is gone. That's it. Can't trust you anymore. As an example, how about my worst day ever? I actually can't remember the year this was. I think it was 2011. But I had just pushed some code that was trying to help with our forms. So we have this builder in MailChimp that lets you design subscription forms for your newsletter. And we let you style them. So we let you specify font size, font type, color, color of the page. We let you drop in images and do things. And we were caching one piece too aggressively where it wouldn't adjust when you saved your form. So I changed some code to make it bust the cache. And what it did was remove all the settings any time you saved. Not really the thing you want done. And it didn't really show you that that's what it did. So it was like, surprise, everything is gone except you don't know it. So it was really not awesome. It was pretty bad. We found it by users coming in to support. And all the typical horrible, you know, typical events that happen when support talks to developers. Like, oh, it's not broken. No problem. And then, no, no, it's really broken. All of that happened in this case. And when I went and figured it out, so, yep, oops, I really did break it. You know, it's bad things come in threes, right? So we had no backups. Which meant that it was just broken. And all our users had to go fix it. The ones who it was broken for anyway. And the only way I could figure out who it was broken for was by running a script that went through all the access logs to see what that was going to look like. So I'm sitting at work, running through this, thinking, okay, maybe it's a couple thousand, right? And the screen is flowing by like the freaking matrix. Just forever. I probably texted my wife saying don't spend any money. I think I'm jobless after today. Okay. So I finally got the final list. And it was 24,000 users. Went to the CEO, told him the situation. Said we got to own up to this. We got to go tell our users that we broke something. So this email that we broke something, not insanely bad, but it still sucks. Our CEO sent that to 24,000 MailChimp customers. And it was not, you know, didn't feel very good. Kind of hurt like down here in your stomach. But the great part of this story is that it actually turned out fine. This is an image I captured from this website. Like this is for real. You can go to this link here and look at it. What our customers did instead of yell at us and be really, really upset, there were a few upset. But the majority of them said, oh, man, thank you. This is how you apologize. This is how you own a problem. Instead of losing trust, the trust meter went up because we said we screwed up. Totally our fault. We added some humor to it. So we called it the font apocalypse. You know, if you've been affected by this, go to support and mention font apocalypse and they'll give you some money back. But owning up to it, not hiding the problem turned out fine. Maybe if I had hit it, I would have gotten fired. But I didn't hide it. I went to my boss and said, I screwed up. Here's how we need to fix it. And, yay, not fired. Woo. Spend money again. Our wife's here. So the second rule is learn from your mistakes. I added this Winston Churchill quote. And I never know, like, outside of, like, in certain countries if Winston Churchill is, like, a figure it's okay to quote or not. So don't kill the dumb American if that's a bad thing. So never let a good crisis go to waste. I think that's a really, really important thing to think through. The example here is that we used to have this thing called the plain text editor. So when you send emails in MailChimp, you build an HTML version and then there's a plain text part. We used to have this full editor for doing that where it would take your HTML part and then kind of translate it. And you could go in and edit it and just work on your plain text part. But it just felt like this really horrible, cumbersome load. It was a really horrible, cumbersome, lengthy process. So we started looking at it and figuring out, like, okay, are people actually using it? We found out that 1% of users sent something different than the version that we auto-created. We also along the way looked and it was difficult to manage. Users needed to know that once they built it and then they changed their HTML part, they had to go back to the plain text part and edit it. And that was creating lots of bugs and confusion and things just weren't great for users. We had tons of support issues from it. It just wasn't a great experience. We did some research. We asked some users. All of them said, meh, not important. Some email clients don't even support it. It's even hard to see the plain text part. It's sent with all of your emails, but you probably don't even know that it's there. So our decision, nuke it. Remove it. Just remove it. It'll take our process from five steps to four steps. It makes the process so much cleaner. It makes everything better. Designers had such a problem with the plain text even having to be there. So we launched it. It felt really good. We've reduced friction for our users. Things are awesome. What we realized was that 1% needed it massively because accessibility is a major thing. And the 1%, not even 100% of the 1%, math, but somewhere in that percentage, it was mega, mega, mega important that they be able to alter that plain text version. And for those, it was accessibility. I don't know. I assume there's similar laws outside of the US, but there's Section 508 federal law that says, meh, if you are companies of certain types talking with certain types of users, you have to be 508 compliant. To be truly 508 compliant, going from HTML to plain text, the users really needed to edit it. And we made it impossible for them to do their jobs because we removed it. Right? It was completely gone. 100% gone. So what we learned from that is it was a crisis. It was really bad. We got some super mega bad backlash. 1% of our users at that time was several hundred thousand users. And that's a big giant fire hose when you have a support team of about 150 people. What we learned there is make sure you keep asking why. Listen to your users. Ask them why. Ask them why. Ask them why. Ask them why. Until you really get to the true why they need it. Right? We were asking users why they didn't need it. We weren't asking why do you need it? Why do you need it? Why do you need it? Why do you need it? So that was a fun, horrible lesson. But we fixed it. We found a way to get it back in the process without adding a step. We ended up in this great spot where we still reduced friction for the 99%. But we still provided the ability for that portion of 1% who really needed it. My third rule is to form relationships. So the big thing here is whether you really think it's your job or not, we believe pretty strongly that it is your job to get along with your coworkers. And form relationships with them. Because of having these independent teams and these teams where there's cross-discipline folks on it, need to form relationships. You need to form the ability to communicate. You need to collaborate. And you need to have empathy for each other's work. The best way to do that is to get on a nice little small team doing cross-discipline work. I have this great image from our holiday party last year. On the left is our QA director. In the center is our, one of our UX directors. And then that's me over there on the right. And we just thought this was going to be a hilarious picture because this is what people usually expect from the three groups here. That there's just tension. There's just so much tension between the three groups. And it was awesome because our CEO saw the picture and immediately he knew that we were just being funny. Because he knew that we had formed great rapport with each other. We had a great relationship. Things work really, really, really well. That's the actual picture. I can't remember. This one's probably, I don't know. Is my shirt ruffled enough? I don't know which one came first. But this is how we work as a team. You know, kind of arm in arm, all smiles, everyone getting along. It's not, I don't want to lie and say that, yeah, you know, this is how it always is. There are moments where it's like that. But we form relationships together. And, you know, we do things like, have coffee together. The guy in the middle makes a great cappuccino. And I don't at all. And I love cappuccino. So it's kind of a nice way to, hey, make me a cappuccino. Let's be friends. But finding those types of things with your coworkers. And we sit down over our coffee and we talk about the struggles he's dealing with, with, you know, UX and design. And I talk about the struggles I was dealing with at the time with back end development. And next thing we know, just sharing cappuccino, we've got tons of empathy for each other. And we're doing that instead of that. I just realized how just amazingly comical his face is. He looks like he's kind of not struggling too bad with my choke. But I look like I'm really, really getting after him. So, yeah, rule number three, form relationships. You have to get along with your coworkers. It's super, super, super, super, super important. So what are the big challenges then? I'm not going to sit up here and say there's no challenges to this approach. That having small process and, you know, not tons of crazy overblown agile process is easy. It's difficult. Cross team communication is really, really, really hard. In a group where we give each individual group independence. How to not have them, like I said earlier, form into countries. It can be tough. You have to figure out how to get them together. You actually have to have meetings. It's hilarious. There's a video on me of our, actually I think it's now been taken down. But there was a video very recently of me on our jobs page saying, we never have meetings. Ha, ha, we just hire great people and we like knock down walls and get out of their way. And now we have meetings all the time. I loved it when we didn't have meetings. And I love it when we're having meetings. And the reason I love it is because we're getting teams together. And we're talking about what we're doing. That communication from management has to be about core values. It has to be about culture. And it has to be about your users. And what you're doing to make things better for them and championing them. These teams are working really, really, really, really hard. Because they're all cross discipline. And they're all wearing a bunch of hats. They need down time. So we try to work into our process. Cycles for individuals. Or specific teams. That are just research. Go out and work with the UX research team. Go out and work with a bunch of users. Go out and work with support. And just research. Your deliverable for this process is research. We actually have an email developer in the crowd here who is working on a research project right now. And it's his deliverable. Go and figure out how this may work. It gives you down time. It gives you down time from the constant of produce, produce, produce, produce, produce. Go out and research. We also can do like bug blitzes. Where you figure out, find something broken and figure out how to redesign it. Or find something broken on the back end and figure out how to redevelop it. Use that to get really good solid down time. And focus on users. Hiring can be super, super tough in this environment. It can be really hard finding folks who want to help. It's a lot. Saying you're a designer and a product manager and a project manager. And you're going to be on this cross discipline team doing all this. It's asking a good bit. But you got to be careful hiring. Make sure you're not lying to those folks. Yeah, yeah, you're just going to do just design. Just wire frame things. And they get in and oh, now you have to do all this other work. So there's definitely a two-way street with that. Like I said earlier, it works really well for us to hire up from within the company. Because they've kind of gotten in. They know how we do things. They've really bought into core values and approach to problem solving. We have this amazing program that we started two years ago now. It's called an apprenticeship program. Where members of support or other teams. That might be interested in design. Front end development or back end development can apply for an apprenticeship program. What we do with this program is we bring them into the product team. And we save their job wherever else they were. And then for three months, we train them. And we have them actually work on something on the product team. At the end of that three month cycle, everyone decides. Are you going to be good at this? Is this something you really want to do? Yeah. And each, both the product team and the team member decide whether they're going to stick this out. And move on to the product team. Or they're going to go back. We've had four apprentices. And we've had three stick on the product team. And one go back to working on the integrations team. And that part's awesome to me. That it's not 100% success. The fact that it's 75% success. And we've had one person go back to the job that was saved for them. And still be productive and happy in that position. I think has been a super, super, super great win for us. And the other three have been just ridiculously awesome. So it's really great. So I have some shameless plugs here at the end. And wow, I have spoken way quickly. So this is going to be a short one. Hopefully that will be Q&A. So shameless plug for work at MailChimp. We're hiring. We're 100% in Atlanta. So if you're in the U.S. or want to come to the U.S., there's jobs. There's some kind of, I don't even know what's going on. But there's some kind of contest. And one of the things you can win is that Freddie that's embedded in that plexiglass there. I also want to put in a plug for the UX newsletter. We have this amazing newsletter that some folks on our UX team write. It's designers and developers. They just published Article 1 of Volume 2 yesterday. Yesterday. And it's super awesome. It's about a new release we've got out called MailChimp Pro. So check out that newsletter. And then big thanks to everyone. This is my first time in Copenhagen. And this city is so, so super cool. It's neat seeing the bikes and almost getting run over by a bunch of bikes in the street. And I hear that the beer is really awesome. David from the BBC said it's great. But I haven't yet had beer. So I think beer is coming soon. And then I'd love to talk. So if there's questions, this is my favorite part of everything is a nice, decent Q&A. So hopefully there's questions. Fantastic, Eric. And you're not early because we have 10 minutes for Q&A. Okay, great. So you're perfect. All right, good. Yay. Q&A. You go ahead, Mia. Hey. Howdy. Mike from Clue. I presented yesterday. And my whole team has been watching on the stream. So thank you for the stream. This has been really, really helpful. Hi, team. Question that we had is about process. And you mentioned, I forget the four stages, research going all the way through to maintain. Yep. And does the same team go all the way through that process? Do you complete all of that within the five-week delivery cycle? How does that actually play out? Yeah. So the whole team does. The whole team does go start to finish research, design, implementation, and maintenance. Sometimes that plays out in five weeks. Kind of depends on the feature. We try to not be one size fits all with that also. Like I said, that example I showed of the drag and drop editor, that took three months. And so kind of depending on how big a project it is we're working on, those first two phases may take a full five weeks. And then maybe the implementation phase takes one nice solid five-week chunk. And then the maintenance phase is kind of ongoing for projects. So maintenance, we do a one-week code freeze, hard code freeze where we're testing. And that's the start of maintenance. But then maintenance can last forever. But we also take maintenance and make it one of those projects. So if there's something that's fully released for users, research and design will start with our support team and with our UX research team where they'll take an existing feature set in our application and figure out how it's working, what's working well, maybe what's not working well, and then turn that into a five-week cycle. So the research phase starts there, and then design comes in to figure out what needs to be changed and how to improve upon it. And then that's it. And then we go to implementation and maintenance. So, yeah. But just for clarity, then, the designers and the engineers are also involved in the research phase, which may involve going out and talking to users, that sort of thing. Yeah. We actually do this really weird, dangerous thing where we send developers to talk to customers. I agree with that. There's been some interesting results of that. Developers are very logical. Yes, no. What? It's working. But it is also phenomenal for the developer to watch users use the product and get to talk to them about what they're doing. So, yeah, we send developers all across the place, meeting with users. We put them on video chat with users. We found this, if you're doing mobile work, one of our UX researchers from four years ago developed this great process called Hug Your Laptop, which is how do you... Someone's using your mobile app. How do you get a screencast of it? How's that going to work? So you just grab your laptop, hug it, use the phone in front of your camera, and that's how you get the feedback. So that's been exceptionally useful for us. But, yeah, we send everyone, developers, designers, sometimes even infrastructure backend, like way on the backend, engineers. Cool. Thanks. Yeah. Another question? Who wants to... Over here. Chris? To run Mia. Yeah. Where is it? No, not run. Walk. Where is it? It's okay. Thanks. Yeah, I'm just wondering, do you guys ever do user research with... Not necessarily with the tool, but, like, with the emails that people actually get, like, to see how people respond to emails that you send, like MailChimp emails? So I work at Airbnb, and we send a lot of emails. Yeah. And we're always wondering how... Like, what a good way to test the response to those emails is, like, in a qualitative way. Of course, we measure everything, but, like, I'm just wondering if you have any good ways to do that. Yeah. So part of the feature that we just launched, we launched something called MailChimp Pro two weeks ago, maybe three weeks ago. And a feature set there is called multivariate testing. So we use that pretty heavily. We've actually been using it secretly for a while. We've been testing it. And what that allows you to do is set up basically A-B tests, but with combinations. So you can test your content, you can test your template, and then you can test subject lines, from names, and send times. And then the system will do the combinatorial testing from there. So if you say you've got two subjects and two pieces of content, it'll build four lines. It'll build four actual email campaigns, and then split your list across all four of those and send that out. And then you get this gorgeous report that lets you really see which one of those was the best. One of the best things and best and worst things in email is knowing that people don't go below the fold in email. They do on websites every now and then, but in email, it gets abysmal underneath the fold. So there's that. And then you can connect analytics. So figuring out from the individual email, when it takes you out to your site, where they go from there. We use a tracker called Goal, which is a MailChimp-based service. If you install this JavaScript snippet, it'll show you where users end up on your site. So if you set up a goal, in this email campaign, I want people to end up four pages deep, or And then you can go back and slice and dice your campaign and say, show me of the 20,000 I sent to you, which ones actually completed goal X. So we do a lot of testing like that. Thanks. Cool. Yeah. Thank you. We can squeeze in a couple more questions, depending on the length. Yeah. Oh, great. Thanks for being right next to him. Keep going. You're next. Yeah. So I'm just thinking, when you have this process that is pretty simple, and you have the requirements in a broad term, do you fix the time scope of the projects before actually starting, you said, having a start on an endpoint? And how does it work out? I don't think I understand. Okay. It's just when you have the requirements in a broad perspective, and you have a project, you said they could vary in length. So it could be three weeks. Is that time scope set upfront, or is it based on whatever is coming up during research, and what feature we actually need in order to complete this? Or how does it work? Yeah. So you're basically asking about timing the launch of things, because some things may take longer than five weeks, and some things may take shorter. So... We have just a five-week cadence, where we cut a release at five weeks. So if a team gets a broad-scoped project, they go in and do a deep dive, and say, this is going to take 10 weeks, 12 weeks, 14 weeks. And we say, okay, well, we're going to release at five weeks. You keep working. And when you get done, and you're ready, we'll cut the release. Sometimes we move the release. I mean, it's one size fits all. We can't say... We can't say with absolute certainty that every single thing will be done. Every single thing we do has to fit within a five-week cycle. But because we've got about six different teams working on individual parts of the application, we can say these four are going to release at our five-week cutoff date, and this other team is going to keep going. And they're going to just keep working on their thing. Maybe next time they'll be done and ready. We do try to de-scope, though. If it looks like something's going to take six months, we're going to try to de-scope. That's when we go back to the process and say, okay, what in that plan can we break down? And then we start talking about minimum viable product, right? What in that giant thing that's going to take six months? Is there something we can chop off that saves us three months? And then let's get that done, listen to our users, and see if we even needed that extra piece of functionality that was going to take three months. So we just... We're very fluid about timing of those things. Usually at the end of a five-week cycle, we start planning the next five-week cycle. And we announce, sometimes it's not even five weeks. Sometimes we say, yeah, this one's going to be seven weeks. I think our current one we're doing that with right now. We're saying, well, we're going to do some bigger things. It's going to take seven weeks instead of five. But we try to sit at five just to make sure that scope creep doesn't crop up on everyone. And it keeps the team united and gelling toward things. But I think if you set too many strict rules, you end up in a bad place. You end up building things that your users don't trust. Or you end up taking way too long and you never build anything. And your users stop trusting that you're going to remain relevant and continue doing things. Does that answer it enough? Yeah. Thank you, Eric. I have the feeling that people could just ask you questions for hours and hours. Yeah, do it. So what about after the next talk? Eric is still here. You can't leave. You have to stay at least five, ten minutes. Do you have a team with you? Where's beer? I do. They can answer questions too. Yeah, because is there a designer on the team? And then I think there's probably someone who… Yeah, so Fabio. Can you stand up? Bald with a giant beard. I'm just curious. Male team. Giant beard. Yeah. So… And Alex too. Tall girl. Stand up, tall girl. Yeah, I get to embarrass my team. Okay. So after the last session, which is the next one… Okay. …stay around. And everybody, if you have more questions, just let me know. If you have more questions, find Eric and his team. Cool. I love questions. Ask me questions. Cool. Thank you so much, Eric. You're welcome. Picking you up on stage now. This for you. This thing? Danish stuff. Really? Yeah. The Danish term, you'll have to have Eric… I mean, Chris explain what it means. Hygge.