How to measure your product

We are very grateful to have Sveld Doce from Mixpanel, who co-founded Mixpanel almost 10 years ago now, and is going to talk about how to measure your product, which, as you heard from Gustav, is really the other side of the coin of growth and everything that helps you build something that people truly want and will use. So with that, Suhail. All right. Hello, everyone. Happy to be here. Awesome.

I thought I would start off by just kind of like, I sometimes try to change my presentation a little bit on the fly, depending on the audience. And so I thought I would just ask a really quick question, just a quick show of hands. I'm going to define a user as just someone that uses your product, whether it's a B2B company, paying free, doesn't really matter. They did something in your product.

How many people have zero users right now? OK, cool. Awesome. And then how many people have 100 or less? Well, OK. You have greater than zero, between one and 100. OK, awesome. Great.

And then how many people have more than 10,000? OK, awesome. All right, so we've got, I'd say, majority of people have zero users right now. Cool. Awesome. Great. So I'm going to start off with just a quick about me, for those that haven't heard of Mixpanel. I started this company called Mixpanel.

I think the first line of code was written in like October 2008, technically. I was like 20 years old. I was in my parents' house in my bedroom. And 10 years later, the company has about 300 employees. We've raised a bunch of money. And we have about 7,000 paying customers.

And we're starting to close in on around $100 million in annual recurring revenue, just to give you a sense of, yeah, 2008 to now. Cool. Awesome. If you don't know what Mixpanel is, we make pretty graphs like this. We help you measure what people are doing inside the product. So this is like someone inviting a colleague to their app.

We might help you measure a funnel and see the conversion rate of people opening your app to viewing an article. We'll help you measure retention. I know Gustav went over that a little bit today. And I'm just showing you this just because everything that I talk about today is basically something that you could probably do in Mixpanel or even any other tool.

There are a bunch of tools that are great to measure these things. But I want you to know that if you walked out the door and you wanted to apply any of it, you totally could. There's a freemium plan. And we cater very, very much to startups. Those are all of our early customers. So feel free to do that. Awesome. Great.

So today, what I wanted to dive into was I thought I would take a top-down approach to thinking about analytics and metrics. Instead of coming at you with charts and visualizations and numbers and things like that, I thought it would be better to think about the problem that you're trying to actually solve and ignore for a moment that you have to do complex analysis and calculations.

You will figure that out. It's not hard. Or you'll just use tools that will make that a lot easier for you. So that's one thing that I thought I would do. And so I picked these three things. I tried to think pretty hard about what the heck did I care about when I started a company? What mattered to me? What was on my mind?

And I thought, well, I think the first thing is just like, do people even understand what I made? And Mixpanel had a really hard time with that in its first 18 months. We had to change a lot of things. And most people do. The second thing is, OK, great. They kind of understand what it is. But do they find it relatively easy to use the product on day one?

And that's kind of this never-ending mission that you'll end up taking when you start your company and when you ship your product. And then the last thing is, I kind of tried to pare it down a little bit. But I thought the most important thing would be then next, are people coming back and using my product? And I think that you'll find that, like, I know Gustav talked about this.

And I'm going to be extremely repetitive, because it's one of the most important things that early founders make a mistake with. Yeah. So from there, I thought that it would also be helpful for you to see it kind of broken down in a formula. And so this is the formula. And I like to think about things from first principles, right? Like, what are the things that would cause you to grow?

And they're pretty simple, right? There's just people that visit some page or the app, right? Basic. You experience this every day. And then there's people that sign up. And then of those people, how many of those people did something valuable, like watching a video or making a recipe or taking a scooter ride?

And then of the people, how many of those people end up coming back some period of time later? And then how many people will spread that product, like tell a friend, share it? Maybe you even have a sales team. Who knows, right? And these five things, like basically equal growth. And these are your levers.

And the thing that I want to kind of point out and I want to stress is that today I'm going to cover the basics. But I want to kind of implore on you that the basics are really important. Even 1,000-person companies get the basics wrong.

In fact, you'll find that if you don't get the, even these 1,000-person companies that get it wrong, they tend to overcomplicate the number of things that they should actually measure and track. And it's really easy to make 25 metrics, conceive of them, align them, divide them all by, give each team five, and then really overcomplicate something that is actually quite simple to do.

When you overcomplicate it, what happens is that companies, even large companies, have immediate paralysis. Even mid-sized large companies have this decision-making problem. I used to have this funny phrase at Mixpanel where I used to say that all data is contestable. You could see this number go off and to the right. And you'd go, oh, well, I don't know. It was sunny outside.

And that's why people did it. And people will have these arguments no matter what, because causation is really hard to discover. And so what I find is that just picking three to five things on North Star and really simplifying it will do more good for you. And I know, because not only have I watched thousands of companies make this mistake at all sizes, I have made this mistake.

I had a piece of paper that had 25 submetrics. And it was nearly impossible to actually keep track of it. And what people end up doing is, in reality, what people care about is there are metrics that guide the team, and there are metrics that help you monitor and assess if anything is going wrong. And today, we're going to talk about things that guide the team. Cool. Great.

So with this formula, you should be able to assess anything going wrong in the company. If you have a really confusing landing page, you should be able to figure that out. If your sign-up process is too difficult, you should be able to figure that out. If you have a product that people don't really value, you will figure that out with just that formula. It's just that simple.

And you can get more complicated after that. You could say, OK, landing page views are down all of a sudden. Why? Ooh, let me look at that segmented by country. Maybe something happened. Maybe I changed something in a different country. So you should be able to assess any reason why you're not growing just starting out with that. And that's why I say starting simple will last a very long time.

So this is what it looks like, linking those two ideas together. You have this funnel of people going from visits to eventually retaining and spreading the product. And on each one of these things, you're trying to basically solve one of these three problems. Is it easy to understand? Is it easy to get started? Are people coming back because they find it valuable? And then it loops.

And the funny thing about this is, even if you're Airbnb and you're thousands and thousands of people, this process of optimizing each one of these steps is never ending, no matter what scale you're at. It will always be changing something. And you'll probably have competition. So you'll have no choice. So awesome.

And then once you are able to measure those five things and assess what's wrong, you have your levers. And you should be able to take action and fix any of those things. Put a team on it. Maybe you're fixing it. It is really that simple. Awesome. So we're going to start with the first thing, which is, is my product easy to understand?

In order to illustrate this idea, I decided that it would be somewhat sympathetic. Because I could show you really awesome landing pages of great companies. Or I could just show you what it's like to start out.

And I also wanted to mention that having a really bad landing page, there's this really awesome quote that PG used to have in our batch that I thought I would impart on you, which is that people basically just have their mouse hovering on the Back button. And they're just waiting to say, I don't know what this is. It's confusing. Back.

And that's immediately how most companies are just losing most of their users from moving on. And so just remember that your enemy is the Back button. So this is what Mixpanel's first front page looked like in 2009. That's not what our logo looks like anymore. This is a design by me, original design by me.

And in all of its glory, if you actually go read the words, there's tons of grammatical mistakes because I was horrible at writing. And for some reason in my mind, I thought, I know what the tagline for the company should be. Metrics that'll make you drool.

And I think the reason why was because I thought these graphs were so pretty looking compared to what was out there that I thought that would be the value proposition of the company. And of course, this is totally wrong. You're all laughing. I wish I could have made this landing page, presented it to you, and then you'd all laugh at me. OK, I should change that thing. But I didn't have that.

I had zero users at the time. And I tried to target small startup, big company. I didn't know who I was targeting, application developer. I'm trying to target everybody. Every mistake that one could make on a landing page is right here, the Sign Up button. At least I kind of got that right. Thank you, Adora, for teaching me how to optimize buttons that slide, made it yellow.

But that's about maybe all that I got right, just maybe roughly the Sign Up button. And then after a lot of work, a lot of hard work, a lot of iteration, a lot of talking to customers, a lot of measurement, eventually, a couple of years later, it kind of turned into this. I managed to get the logo right. The Sign Up button's ginormous and yellow.

And what we figured out, though, after a lot of time and a lot of energy, was that we transformed from metrics that'll make you drool to actions speak louder than pages. And you'll notice it's extremely prominent. The reason why was because the number one question that we would get as a company was, that's cool, but how are you different than Google Analytics? I don't get it.

And so it turned out this tagline for us at the front page lasted the next five years of the company. And because it very much differentiated us as something where we would measure engagement, we did something different. Google was about page views. We were about what people did in your app. And that resonated.

And then from there, we tried to explain what various, we tried to urge people to go down the website and look at different features and see if those things were valuable. We tried to be benefits-oriented in the copy, tried to show customers and case studies who maybe found it valuable. And we've probably made four changes after this.

But I thought I'd present this one just because it was the one that maybe lasted, like the copy and the general essence of what we did and said we did, what we said we did, kind of stayed the same, largely, but lots of design changes. So the real question is, great, so how do I measure, how did you measure whether people really understood, whether they understood what you were doing?

And so I think there's one tried and true number. And it's very simple. It's just, did people even bother to sign up? Bothering to sign up doesn't mean that they're going to use your product. That's a whole different matter. Bothering to sign up is like, I don't know. I'm kind of interested. I'll kick the tires.

This seems kind of interesting and cool to me. And that'll tell you a lot about whether what you're doing is even something valuable for people. The second thing that you can try, these are just ideas. Trying to find the ratio of people that are going from just hitting your page, and then doing any kind of thing after that. So instead of S-A-K-A, instead of hitting the Back button.

Like, did they click on segmentation, or funnels, or retention in that case? Did they bother to do that? I know there are a lot of B2B companies these days. Actually, one quick question. How many people are making a B2B company? Yeah, great. So that's about half the room. So I thought I'd give a B2B slant.

One thing that I discovered was that people, a lot of your users, will click on a pricing page link at the top. It's like second or third most clicked on link if you're a B2B company, I think, generally speaking. I think this is totally contestable. But I find that people clicking on your pricing page is actually a pretty decent indicator of like, OK, cool. You have something kind of valuable.

OK, how much? And so even something like that, I think, can tell you a lot. And then if you're in consumer, the benefit of being in consumer is that you usually will have hundreds or thousands of users such that you could actually go do an A-B test. Whereas in B2B, it's a lot harder to achieve that.

And so I think Gustav is absolutely right that you should try to conduct as many experiments as you can and really A-B test your copywriting and find out what resonates with people. But these are all things that you can do to basically figure out, hmm, is this easy? Cool. So the second one is, well, is it easy to get started with my product?

And the thing about this is like a lot of people, the thing about these first two things is that if you don't make the first step easy, if it's easy to understand, a lot of people put so much energy. When we're trying to build a company, we try to think about all the cool features and things that we could make that would be really valuable for customers.

The problem with that is if you don't even get step one right, you don't even have a chance at any of the subsequent steps. So when we think about is it easy to get started with my product, forgot-about-password is kind of a useless feature if you have a very small percentage of people that are basically not even bothering to get started.

In fact, I remember there was a YC company called Zenter, which turned into Google Slides. And I remember Robbie Walker, during one of our Tuesday dinners, said that they didn't even implement the Save button for their online PowerPoint because they knew that none of their users would ever ask, like, hey, can I save this?

They just didn't even bother to do it because they wanted to see would anyone even use the PowerPoint presentation product. And when they imparted that on me, I was kind of like, wow, that's crazy. And for, I don't know, maybe like 18 months of Mixpanel, we didn't even have forgot-about-password because we were just so ruthlessly trying to prioritize things that would really matter to customers.

And it turned out we didn't get very many requests anyway. So this is really important. And here's an example of easy to use, something you use every single day. That is literally one step. You just go to Google, you type it in, and you immediately get to use the product. There's no sign-up. There's no CAPTCHA. because there's none of that, right?

A more complex example is like Airbnb who might have like 10 steps or something like that. We worked with Airbnb for a while helping them optimize initial user experience.

And even though Airbnb probably has like 11 pages of, like you have to go to Airbnb and then you have to fill out this form and then you get taken to the results and then you click on those results and then you click on many different results because you're trying to price compare and you're going through this very complex checkout process.

So there are actually many steps to booking something online. But Airbnb had to kind of figure out like, how do we A, describe what we're doing, what we offer, what's useful for the world and B, how do we help people get started as quickly and as easily as possible? And like, this is airbnb. com right now. And it's, you know, they have this very simple, like book unique homes and experiences.

You fill out these very basic things and you hit search and that's it, that's their getting started experience. It's not their entire flow, but it's the first way to get started. And they even try to like kind of impute something like what you might be able to find. Something imaginative on Airbnb. But the thing is, is that this seems simple, right?

This seems like something that you could just like easily make, it doesn't take, how hard is that, right? You can do that with Twitter Bootstrap in no time. The problem, the thing that's really hard is like, figuring out that that's the right thing. That's really hard.

That takes like, they probably changed that, you know, 50 times before they figured out like, this is the right current optimal thing at this stage for the company. And so it takes a lot of work and you have to grind very hard to get to this point. So question is like, okay, great. So how would you measure something like that? Well, I think the really simple answer is a funnel.

How many people go from a landing page to signing up to doing that valuable thing, like watching a video, doing, in Airbnb's case, it would have been just doing a search, just one search at least. Once you get past that, measuring your funnel for the entire initial user experience. There's like weird things where like, there's actually situations in games, games can have like 20 steps.

And I've noted, and I've seen hundreds of gaming companies' funnels. And even though they might have like 20 or 30 steps, sometimes you can, they actually convert very highly. Like they might get like 80, 90% conversion rate through all those 20 steps. Just the first two are like actually the ones that have the greatest drop-off. And then the last thing is just like actually speed.

This is one that I think a lot of people don't actually do, which is like how fast can someone just get started? And that will tell you a lot about how complicated your experience is. Sometimes slow is okay, depends on the business. But speed is really important. Does it take them like five minutes to get something figured out, or does it take a minute?

In Mixpanel's case, we cared a lot about like, we had a complicated flow. Because not only did you have to sign up, but you had to be able to write code. Just really complicated. And that was the one downside we weren't really sure, where people, with Google Analytics, you just copy and paste the JavaScript. With Mixpanel, you had to like actually go and track a line of code.

And so we had this question in our mind, was that easy enough? And so speed became really important to us. How fast can someone do that? And I wanted to give you a couple tips and tricks. Things that I have seen that companies do that are generally like not great. Number one, you're a startup. You don't have lots of fraud problems. You probably don't have a lot of people spamming your service.

You probably don't have a lot of like fake actors in the system trying to sign up. So if you don't have those problems, don't optimize for them until you do. And I just want you to know that generally, email and text confirmations have an enormous drop off.

Like, I'm talking like, you get, of the 100% of the people that come, like only 30% will like basically go on and like click on those email confirmations, or 40%. You'll spend a lot of time trying to optimize that. And then to make matters even worse, like you have to hope to God that it doesn't go to spam. Like, it's this really horrendous experience. So just be really careful.

You'll lose a lot of users that way. It's really important to iterate on your initial user experience. That will be a never-ending process. You will do this forever. Even as the, especially as the product changes. And it becomes really critical to like have someone on it, at least one responsible person just thinking about that all the time.

And then the last thing is, is not every company, not every product is capable of doing this. But to the extent that you can just let users into the product, it's always better to do that. In Mixpanel's case, it wasn't possible because like we needed to get your data, so you had to sign up.

But we experimented with like, what if someone just copied and pasted the JavaScript, but they didn't sign up? And we just collected their data. And then after that, we like asked them to sign up. We did experiments like that. So, to the extent that you can achieve this, that's awesome. And good examples of this are actually like Airbnb and Google where you just try it.

But there are always exceptions to these rules. For example, like Pinterest makes you sign up. They didn't used to, but they do now. And there are good reasons for that. And then the last one. Definitely not least, are people coming back? To me, this is probably one of the metrics that is greatly ignored by most startups.

And it's often the reason why I've seen companies, even with millions of users, actually die. I've seen probably 15 different companies that grew virally and then just died. The company died. And I wanna demonstrate this in a graph. I call this the shark fin effect.

And the reason why I call it the shark fin effect is because, well, one was because in Mixpanel, the default line color for your first data point is like blue. And so, to me, it always looked like a fin in the water. And it looked a lot like this. And what happens is you're slaving away and you find a way to go viral. Maybe you're going super viral on Facebook or Instagram or something.

And you're like, wow, I have struck gold. I'm acquiring tons of users. My life is awesome. I'm gonna be a billionaire. It's great, right? And the problem with this is that if it happens too early, but you didn't quite think about retention, it becomes really problematic. And this hurts companies.

So basically, what happens is the app goes viral and then eventually, the rate at which you're losing users becomes high enough that you can't acquire the new ones.

And it turns out that even though we like to try to be extremely rational people, it's really hard because when the app's going viral, you're just like, oh my God, I need to just take advantage of this moment because what if I don't get it? And what if my competitor finds it? So I'm just gonna optimize, optimize, optimize.

But it turns out that it ends up not being very valuable because if you're just losing all the users, it's really, really hard to reactivate. If you talk to people that have ever done a reactivation campaign, reactivation campaigns generally don't go very well. You usually just lose those users until they decide that maybe they wanna give the product a second chance.

So it's not like, your response is like, when this happens, a lot of people's response is like, oh, we lost the users because our app was invaluable. Let's just email them again and see if they'll come back. And that's usually not a great idea. So it can be really, it's really important to really think through retention. And you really don't wanna be like this guy.

The shark fin effect is really, really bad. In fact, LivingSocial is a great example. A lot of people don't know that LivingSocial was basically this app that was on Facebook that had nothing to do with the Groupon deal cutting thing. And they went super viral on, I can't remember exactly what the product was.

And LivingSocial literally had to pivot the entire company to basically competing with Groupon because the app wasn't very retentive in those days. So there's some pretty basic ways to measure that people are coming back. The most obvious one that I hope most people know is that it's just new users.

So making sure that you're tracking whether a brand new user, someone who just signed up, what percentage of those users will come back like a week later or 30 days later. It's really important to track a longer period of time because a week later is just not a harsh enough metric.

And it's important to see how fast you'll end up losing users because you'll have to figure out a way to re-engage, to find new ones. And that becomes really, really hard at some point if the number of users you have it starts to become quite high. And then the second one is just daily active users. There might be people that disagree with this. I don't think there are too many.

But I now think that monthly active users has become like the new BS metric that's like very similar to like just like number of registered users. Like number of registered users is like a very silly metric. I remember seven or eight years ago, I think like, I wanna say it was like LinkedIn came out and said, we have 200 million registered users. It was like, who cares?

How many people are using LinkedIn? And I now starting to think that like MAU is like becoming quite close to that now. Daily active users is really hard to maintain. I mean, how many products do you use every single day? You have like, if you just look at your phone and you go through all of the possible apps, like how many of those do you actually use every single day?

So I think it's a really harsh metric. And we actually found at Mixpanel, so even if you're a B2B company and you might think, well, I don't know, I'm selling to businesses. So like, can I really maintain like daily active users, especially on Saturday and Sunday?

And we actually found that there was like a really, really strong correlation with daily use and rate of churn, like rate of like people no longer paying for the product. And it didn't take very many users. I think it was like 1. 4 daily actives would like reduce churn to like a dollar churn, but like to like, I don't know, like sub 10%. It was incredible. And so I really stress measuring a DAU.

And then the last thing is a B2B slant again is something that I think a lot of B2B companies make an early mistake with. It's actually not just a B2B company. It's actually companies that do subscription. So I don't know if there are any companies that would ever think that they may offer a subscription of any sort, you know, like if you're a meditation app or something like that.

And it's just dollar, it's monthly dollar churn. So if you're charging your customers monthly, many businesses charge annually, but usually most startups start with monthly because an annual commitment sometimes can be hard for early adopters. It's just measuring revenue churn. I wanna give you a slight story. If you have revenue churn and your revenue churn is say 7% per month, right?

You might think, awesome, I'm keeping 93% of these dollars every single month. This is awesome. I'm doing really, really, really well. The problem is, is it's 7% monthly churn. And if you extrapolate that out over 12 months, you're actually losing 58% of your revenue in a year. It's really harsh. And a lot of people, for some reason, this thing is not mathematically intuitive to most people.

I don't know why. It wasn't even mathematically intuitive to me early on. And if you think about it, you're spending all of this energy with marketing and improving the landing page and improving the getting starting flow, and you're doing all of that.

And if you don't have retention, if you're losing that 7% every single month, you have to somehow make up for that 58% that you lost going into the new year. And if you start thinking about it, you're like, wow, we're making a million dollars and we're losing $580,000 and we're starting the year like that. Now you have to do that and grow on top of that.

And once your base of revenue gets really high, it becomes really hard to keep up. And you kind of have this really long arc of like a shark fin effect basically. So it's really important to look at your monthly churn and this will be the single cause of like death, but the problem is it's like the worst kind of death. It's like death that occurs.

It's really like, it's really flat growth, but it occurs like eventually in like four or five years if you're able to acquire users. And I mentioned this, not only because I talked to some founders about this, but this is also something that like really hurt us in the early stages of Mixpanel and we really had to figure out a way to fix this. And there's all kinds of ways.

You can use product, you can use pricing. There's all kinds of ways to experiment with this, but this thing probably gave me the greatest number of nightmares and lack of sleep I think ever. So I really wanna stress that on all B2B companies here. Really think through this number and think about like almost graph like, okay, what would our rate of growth be if churn stayed constant?

It's really important. Cool. I wanted to go over two things. One, I recognize that there are a lot of people here that have zero users. So they're like, what the hell do I measure? So what if I have less than 50 users? Well, the truth is, is I couldn't really think of a better idea than just basically you just have to talk to your users.

And I thought I'd give you a mini story about at least at Mixpanel what we did. When we had like no users and I had that page that just said metrics you'll drool over, one trick that I used is that I had like maybe 10 or 11 or 12 customers. And honestly, I just put all of them on IM, like on G-chat or something. I don't know what the kids use these days. Maybe like WhatsApp.

But I honestly just like would just badger them on like WhatsApp or G-chat or whatever and then just ask them. And I remember one time we were trying to redesign our funnel UI. And we used to have a vertical funnel, but we had a competitor that had a horizontal funnel. And it seemed like that was a better idea. It seemed like that was more intuitive for customers.

And so I didn't really know how to like get the data because like how was I really going to find out whether they liked one of those two UIs? And I didn't have the, it was just me and a co-founder. So like how the heck were we going to like do an A-B test on that feature? There was no realistic way to do that.

And so all I did was I just made like a really crappy version of the UI that I thought would be better, the horizontal version of a funnel. I was like kind of grayscale. And I made like a really beautiful, colorful version of like the vertical one, which was kind of our control. And then I just asked like 11 people which one's better.

And then I want to impress this thing because Gustav mentioned that it's the delta, it's the difference that matters in A-B test, not that this is like a true perfect A-B test. But like I asked 11 people on IM, got feedback in a day, and 10 out of the 11 said horizontal funnels. And then we just made it and we built it and we never looked back. And it was totally the right decision in retrospect.

I don't think there's any way that you can kind of skirt by talking to customers. But here's the benefit of talking to customers, even though it feels maybe it's more tedious than looking at a graph. You are going to get way more information and depth from talking to customers than you ever will looking at a data point in a graph.

No matter how much you slice and dice and segment the data, no matter how mathematical you are about it, you will never get as much information in building your own intuition than just talking to customers. And it's really critical that you do it. And the one thing that I wish I had done back then was I wish I just wrote it down.

I wish I wrote down tons of the feedback because I think that it would have helped, my co-founder, it would have helped employees in the company down the road really go on the journey, make them feel like they went on the journey with me rather than having our roles kind of feeling like they were so split up.

And then the last thing that I thought I would bring up is to impart what I kind of said at the beginning, which is that I think that one of the biggest mistakes that people make with analytics is being really overcomplicated, thinking that they need to be super sophisticated, thinking that they need to track all kinds of crazy cohorts and they need to have like a dashboard with like 30 panes on it that all like loads, you have like mission control.

I don't think that is necessary. I had that, it didn't work. It was really hard to run the company that way. And what also happens is that when you build your team, it will be really hard, it will be confusing for your team. Of those 30 panes, like which one do I need to care about? And I think that we need things that are, humans just need to simplify things. So I'd pick one North Star metric.

And I think in a North Star metric, I would choose in this case is like, what is a number that you're willing to bet the company on? Right, like if that number goes. south, you deserve to die. And if that number goes up, you will like, you will have like made a huge dent in the universe. Like what is that, what is that metric?

And I'm not saying you need to choose that metric forever, but choose it for six months. Choose it, commit some time to it. If you find out that it's actually like the wrong metric, which it will be, probably the first time you choose this, usually your metrics are wrong. And then you can change it, but commit again to for another six months, but choose that one number.

And then like, if what you have to do at the beginning is basically just like print it out and like put it all around the office, do it. Because people will start to be maniacally focused on it. They'll show up to meetings and go, yeah, but that number is like down. What are we gonna do? And where you can get really complex is discovering why that number is down or why it's going up.

That's when you can get really sophisticated. That's when you can slice and dice it and figure out what the retention is and things like that, and measure it over a funnel, but keep things really simple. And then the second thing is like, don't boil the ocean. Like pick three to five other things. I think less is more and just stay there.

Telling even large companies, most large companies don't do this. And it usually is very discombobulating for the workforce. It's totally fine to have numbers that you wish to monitor, but don't focus on them. Awesome, I think that's it.

And then for the people that were like slightly bored because I went over the basics, I thought I would help you out by just kind of having some advanced topics for later that you could read about. Really, these are things that I found really useful. My favorite one is this one, The Next Feature Fallacy by Andrew Chen.

And it's just this idea that we always think that the next thing that we make will be the thing that will change the trajectory of the company. A. k. a. the next feature's right around the corner and we're gonna be huge. And I would really recommend that you read that article. It's really awesome, one of my favorite ones.

So for advanced stuff, these are just people worth following, reading about, and can give you more ideas than what I've given you today. Awesome, thank you. (*audience claps*) Geoff, Q&A? Okay, cool, all right. Yeah, yeah. I want to be fair. I don't want to be like if every speaker chose someone in the front.

Who would you suggest the conversion, accessible conversion, targeted conversion from visitors to registered users, registered users to subscribers, paying subscribers? Yeah. Yeah, so that's, the problem with that is that it's so, it's so, yeah.

Oh yes, I think someone asked, what are like good benchmarks for like those, given those five steps, visiting, signing up, using the product, what would be good conversion rates for any one of those steps? And the hard part with benchmarks like that is that it's so dependent on the business.

In B2B, like going from like a visit to a sign up, like I think like a four to 5% conversion rate is pretty good. I think I like swapped notes with Stuart at Slack because we kind of had similar landing pages. I think I asked him, what was your conversion rate at Slack? And I think we both ended up being somewhere around like four to 5% from visit to sign up.

But that said, like, I don't know what that would be for, you know, something like Airbnb, where it's like getting to a landing page and then, you know, searching for your first place that you would wanna book. So I think it's really dependent. I know that Mixpanel publishes, on its blog, publishes lots of benchmarks.

So I know the marketing team there that does that and they like publish all kinds of benchmarks for like, what should your retention be if you're a gaming company, if you're an e-commerce company, if you're a B2B company, things like that. Social company, a video company, things like that. So I think that you could probably go there and find various benchmarks reports.

But it's really dependent on the business, yeah. So when you're going through the sign up process, how do you see numbers vary if you're like, create an account name and password and doing all that, kind of sign up versus a single like button login with like Google or LinkedIn API? How does that like transfer, do more people sign up if they have that like smooth click option?

Do you see more people drop off because they have to use one of those? So I think the question is, what converts better? Like those like social buttons where you can just like quickly authenticate like Facebook Connect versus just like standard sign up? Is that right? Yeah, like all of these questions, they require like a lot of specificity because it's actually like really unclear.

And in that case, I actually don't know. I truly don't know. And so I think the way to answer this question is to find friends in tangential industries that like don't compete with you, but basically do that. And then just be like, hey, what is your conversion rate? Have you tried something? I don't actually think a lot of those numbers are like secrets in reality, right?

Unless you, I mean, you can't go to your competitor and ask that, but like most of these things are not like super duper secret type stuff. And like, you know, they could either give you the information and save you, you know, the six months or they'll just like say, or you try it out, but it's just, it's sort of strange to keep it a secret. So I'd rather give you ways that you can discover that.

I would just ask friends that have like have a startup or even if they work at a big company, sometimes they're willing to share it. Yep. Cool. Yeah, so you talked about making it easy for a user to start using your product. And then there was a slide of the Airbnb landing page. Yeah.

So how do you balance simplicity with informing your visitor, like what it is that you do and why they should use your product, right? So if you don't have that brand recognition already where the visitor already kind of has a sense of what they're gonna do. Right. So how do you balance those two? Right, yeah. So the question is Airbnb's landing page was super simple.

And how do you balance like, I mean, what if a lot of people just know about Airbnb and then just know what to use Airbnb for? And then that's why they can keep it kind of simple versus like having something more complex. A good example of this is like Craigslist is crazy. If you go to like the front page of Craigslist, it's like absolutely nuts. And like even Amazon, if you go to amazon.

com, right? Like even though Amazon has like huge brand recognition, you go to amazon. com and it's like, whoa, there's like million things going on at once, right? So how have these companies found the balance? How have they found how to deal with this complexity? And the truth is, is that I find that it's important to have like a guiding principle.

Like what is it that you're trying to actually optimize for, right? If in the case of YouTube, let's take YouTube as a good example, right? YouTube could just like be like Google. They just be like, what do you wanna look? What do you wanna watch, right? They could do that, but they don't. And so I think that the question really is, is like, what is YouTube?

What is the team at YouTube trying to optimize for? Are they just trying to get you to like watch a video? Is Airbnb just trying to get you to search for any kind of hotel? Or not hotel, but Airbnb. And so I think that like the team has to develop first a hypothesis of like, what matter, what do we think matters? And then from there, it's important to experiment and try to figure that out.

It turns out like, it's not too, again, it's about the Delta of like what people will actually gravitate more towards. Like if you make a really simple experience, but like people are like really confused because they don't even bother to do it, or you ask them, then you kind of have like new information.

And I think the key is to find the minimum number of things you need to add to get someone started. And I think, I don't think there's anything, I'm sure Amazon has tested that front page like crazy. And they found that it's like quite optimal, yeah. Yeah.

In your personal experience, do you think that short tutorial videos, like one to two minutes, I think even will be a big part of the experience? They can, it depends on the product. I'll give you an example in Mixpanel's case. Oh, sorry, the case, yeah, the question was, are video, are like one to two minute video tutorials effective? Fair? Yeah, so I think that it's complex.

We did a video tutorial video in our flow, for example. And, you know, like I would say that, I would say that like a lot of people didn't watch it, truthfully. And this is what I mean by experimentation. Like a lot of people didn't watch that video tutorial. And we actually, I think, eliminated it in the first three steps.

But what we did find was this interesting thing that occurred as a result of looking and measuring things, which was that, well, we have a feature in Mixpanel where you can like see every action that a user takes. And so I would literally just like go to every user and then just like watch, like a creepy person, just like sort of watching like everything that like someone would do in the product.

And eventually when you do that, like when you do like hundreds of those, and you've just watched hundreds of those, you start to develop some sense of like, okay, what you develop is an idea, a hypothesis. And all of these things like, all of these things require art and science. And I think that companies that take it too far, they like take it too far to science, end up getting it wrong.

They end up optimizing for like weird things. Like you can always make a button more yellow and bigger. And like, but that doesn't make users happy eventually. And so anyway, to answer your question, what I found was that lots of people were just clicking randomly on like all the side menu options. And my hypothesis was, well, they hadn't integrated Mixpanel yet, so there's nothing to look at.

So what the heck were you doing? Just like clicking on like funnels and retention, like why would you do that? And my hypothesis was, they were just curious like what the product did. What is it that they did? And so we did this A-B test where we like had a control with like nothing, you know, just the regular, you know, clicking around, seeing like some image.

And then we had another one that had like videos. And then we tried to see like what caused people to like do their integration, their first integration, what was the conversion rate? And it turned out like the video like super won. Actually, we didn't even have more simple tests than that.

I think we just emailed people a video and said, watch this video, super basic, didn't even bother to implement it on the website and had an amazing conversion rate. And then we decided to add videos to all those tabs where there wouldn't have been anything anyway. And so that's why I wanna say like, I wanna stress this, it really depends.

And what matters is that you run the experiment to develop a hypothesis, i. e. the art, not the science, and then measure it, which is the science to find out whether the thing that you thought of was in fact right. Yeah. All the way in the back. I have a question related to the return, the revenue return. Yeah. So how does that relate to the retention?

Because you said 7%, let's say people are 58%. 58% loss in revenue over a year, yep. But if you curve how you look at, when you look at the retention, the thing that's on the Mixed Metal website that if you have 35%, if you curve the retention, does that have the effect as a reboot for the art? Yeah, so these are slightly two different things.

One is like user retention, and then one is like dollar churn. And so in the case of dollar churn, the reason why I bring up dollar churn is that even, like you could totally have a company where nobody uses the product and the company continues to keep paying for it, right?

But I find that like most companies are like pretty incentivized to cut their spending and employees are incentivized because they lose it in their budget. And so with respect to dollar churn, you could have a lot in a B2B company, for example, you can have a lot of user, you can have even one user using the product, but the company could continue to spend money on it.

And like you might, so like a 40% retention rate like might be okay, right? As long as the company is like continuing to spend, but losing 40% of your dollars in a year is really hard and not good retention. Does that make sense? So those things are not exactly linked. They're not directly linked. Yeah, yeah.

Yeah, like to give you kind of a benchmark, like a chat app that's like often used a lot will have like 60, 70% retention. Like those are some of the most retentive things. Like something like Slack would probably have like 80% retention. It's kind of like unbelievable.

Whereas, you know, like a product like Mixpanel when we first started out, like the first year, I think we had like 30, 40% retention and that's not that great. But it was our first few years, yeah. Are these all metrics are applicable for hardware startups? I mean, it's the same like, do you know that we need to go for it?

Yeah, so the question is, are these metrics, can they be used for a hardware startup? Yeah, I don't see any reason why. We worked with Jawbone for a long time and we worked with Fitbit. And most hardware, does your hardware company have a software component to it? Actually, we do like a subscription model. Like we go with our product and it's a service.

Okay, so it's a hardware company with subscription model. Yeah, yeah, absolutely. It may be more challenging as a hardware company to track, but it's like sometimes, but it's totally doable. Most hardware is like connected to the internet somehow. So I don't see any reason why.

Like a lot of the scooter companies, for example, they're like all connected to hardware and they have a software component. So there's really no difference, yeah. Yeah. Oh, I wanted to, at a security point, I saw the first version of the website. Yes. For startups, for graphics, for developers. And then with the later version, it wasn't so imperative to have a tie-in for savings.

So that was one point. It was just a benefit. I was curious about when you were going to say that you wanted to send a person that you weren't told to want to do. Yeah, so the question is, the first page was like trying to target like various kinds of sizes of companies. And then like the second landing page of Mixpanel, like in 2012, basically didn't kind of disposed of that.

And it had something very different. It was marketing like features and stuff. And how do you figure out like what to do basically from that point to the next point? Yeah, so I think that the basic gist is that, so you can clearly see like I had no idea who our customers were.

And then after like a few years, you just, you talk to users, you ask them like what size of company they are, if they sign up, if you want to. We didn't do that. But we just spent a lot of time. It was so obvious that like everyone that was signing up for Mixpanel at that time was a startup. So for a small, in our case would be, we'd call that a small business.

How long did it take you to figure out? Because it was a cat, I think, in general, in a way to figure out. I think like it took like maybe no longer than six months to figure out that we were not gonna sign up a large company. Yeah, and we just didn't have the product. It just wasn't there. We hadn't worked on it long enough.

So the only people that were willing to basically try it were companies that could take the risk to try it. And most of those companies didn't have anything anyway. So it was even easier. It was easier to target companies that had nothing and were willing to bear risk. And then most of the time, I would always employ this trick where I would just, I would charge them money for it.

And then if they said no, I would just give it to them for free because the feedback was more important. But I always wanted to see if they would pay for it. And so the only people that are willing to do that are also small businesses, turns out. So for us, it was just like, it kind of like hit us. It slapped us in the face. It wasn't like this hard thing to figure out. Yeah. Yeah.

Yeah. So you mentioned that you've been doing that for a very long time, right? Yeah. So how do you know what the quality of life is here? Like, what are your passwords? Like, make current users happy versus like only building three on you? Yeah. So we, to be clear, we weren't necessarily building things to bring new users on.

Oh, sorry. The question is, you guys built, forgot about, you didn't, you decided to not build, forgot about password. How do you know when to build quality of life features versus features that just acquire new users? Yeah. So I think that we didn't, we didn't, it wasn't so much that we only built features to acquire new users. We built features to acquire new users and to keep existing users.

So we were very focused on like parts of the product that would actually make a difference, right? That would be clearly differentiating compared to all of the other possible options out there. And like one of the things that's like not different is whether you have forgot about password, right? That was like in the bucket of things that like maybe don't matter.

And so I think the question is like really like, when does it become a problem enough for you that it's worth building, right? Like if you, if you're finding that like, it turns out that, that the number of requests that you're getting for forgot about password is like increasing and then manually doing this process is like annoying, then just start, then you should just build that feature.

Because you know that if you don't build a feature, they can't log in, then you can't retain them.

them and then you lose them so it becomes this like flywheel effect where like the only way to retain them is to do that and in fact even though mix panel didn't build forgot about password for like maybe you know 12 to 18 months we went eventually we found that people like people would try like multiple times I when I was looking at that stream of every action that a user would take I found that like people were like going to the login screen like five times and I was just so confused it's like why are they going to the login screen five times I was like oh I think it's because they can't log in like some people don't even hit forgot about password there's like try try try and I'm sure you've all done this like with a bank or something you're like I'll just do it tomorrow you know you don't even bother right and so it turns out that even that happens and so we went so far even though we didn't do that I like to think that like we really made up for this later on we went to so far as to like we found this like clever trick that Facebook used which was that when you fail to log into mix panel like three times on your third time we just send you an email that says you know we just sent you an email just go to your email and click the button you'll log in right now don't worry about forgetting it don't worry about changing your password just like login and it turned out that like that had this like great retentive effect because we were really focused on retention at that time yeah okay a couple more questions before you yeah how do you gauge product market fit how do you measure product market fit so I think there's a lot of people that probably have already will talk about or have already talked about like you know people will say very and I think they're they're all right so I'll give you a kind of I'll give you a quantitative slant versus like a qualitative science some qualitative science or things like when people are willing to spread you to their friends that kind of thing the quantitative version of this is really just measuring your retention like how many people it's really measuring like DAU it's really measuring the percentage of people that come back a week later 30 days later it's a number of people that like become almost like they'd like need your product or they like feel like life would be a lot worse and I think that you can you can totally measure that by how they use it and so I think the quantitative measurement is basically finding like some benchmark some sense that for your industry maybe that's like 30% and just remember like that benchmark is probably like average and your desires to basically be above average and so if you can like significantly beat that benchmark then I think that you've like really you've you've done better than probably product market fit I think that's one thing I think another thing is like you can try to measure like overall like repetitive use like frequent use in a day so like you can make pretty harsh metrics you can be like for example with DAU you could say I don't want it to just be daily active user like they did one thing I want it to be that user has to have watched five videos and then and then then and only then or do they count as a daily active user so you can be particularly harsh about that metric right someone coming and like to see that's why I say ma use kind of a BS metric now because you come and watch one video and then never come back again and you like count and that's not like a very healthy business so I think that find ways to be harsh about your North Star and I think that you'll find that that you'll find that you reach the product market fit yeah yeah I think with mixed panel it was like 30 40 percent retention was like really really horrendous and we totally didn't have product market fit yet yeah okay it's like a Peter Thiel question what's one piece of conventional wisdom that you think is generally wrong about startups in general gosh that's a hard last question it's like everything else is like fine gosh I think I don't know I'm not I need to think about it but I think that one thing that has like become pretty clear to me is that I think I think after having done mix panel for about a decade I kind of figured out that I tried to impart it on you in in the in the talk today which is just that it's it's much better to just like I think that we tend to complicate things as humans and we don't we tend and that comes that comes through in like everything for example like we we tend to have we tend to like make a version one of a product and then we get to like version two and we're like okay we're done we like we did it we solve the problem and then like often a founder will like kind of get distracted and then go and like work on the next product the next thing and they haven't yet figured out like their retention is probably not great or they're operating at negative margins or like all these things that same thing even happens with numbers like we pick our three metrics then we continue to get more complex as a business and then we add the next 20 things and we all we do all of these things to like in hopes of like feeling like we control the situation better and I just think that those are all like distractions and mistakes and that I think it's really hard for the average founder to actually just be focused on something boring we tend to have like 80 or something and like we just just I decide to get very easily distracted I think it's hard to do something do the same thing for four or five years because we don't like feel challenged anymore and I think that that's a huge mistake that founders often make and I don't know that anyone say oh that's that's that's like totally wrong Suhail but for whatever reason like every founder repeats it so there must be something I don't know something there must be some kind of wisdom that we all think this is like the right thing and I made those mistakes I made new products that I shouldn't have I measure too many things over optimized things that didn't need to be optimized so yeah it's really cool thanks you.