Analytics for startups

Hi, everyone. My name is Ilya. I'm one of the co founders at Segment, and I'm here to talk to you about how to set up analytics and the analytics foundation to build your MVP and to measure these primary and secondary metrics.

So this is gonna be a little bit more of a tactical guide around what tools are there in the analytics space and the marketing space, which one should we actually be using, how do I set them up. Cool. Just before I get started, a lot of you may not know what Segment is. It's an analytics API where you can send analytics data and then toggle on different tools.

So we sit in the cool market position where we can actually see what tools startups are using and we can bring that information to you. Secondly, we've been around for about six years and so we have information about a lot of different startups as they grew and what tools they used. So we're gonna be sharing a lot of those today. Cool. So why talk about why why even focus on analytics?

So obviously, primary and secondary metrics drive the MVP and product market fit process And you're using that to actually test product market fit. You're also using it once you get out of product market fit strict search to actually focus the team.

So maybe there's going be an acquisition issue in the company that's preventing your growth or maybe the users that you're getting aren't engaged, or maybe you're having some monetization issues. And so the funnel actually is a forcing function to understand your business and where founders should be actually spending their time.

And then finally, all the way from you know, two, three person team to a Google with 1,000,000 employees, you're actually using metrics to operate and drive teams. So eventually you have an engineering team, you have a marketing team, and so what goal do you set in front of the marketing team? Use analytics for that. Okay. So today we're gonna cover a few different things.

So first you always start with the funnel when you're thinking about analytics. So that's the sequential series of steps your users go through to actually get value and then pay you as well. Then we're gonna talk about collecting data for your analytics tools. Then we're gonna talk about the top three metrics. So this will include primary and secondary metrics that work for most products.

And then a product market fit methodology that you can apply on top of that. And then finally, we'll make recommendations about what tools are the best in the market right now that help the product market fit journey. Okay. So to start, always start at the funnel. So we'll make an example funnel for Netflix, which is a company that we're all super familiar with.

Any b to b product or b to c product actually has this type of funnel where you acquire a user, you engage a user over a period of time, that loop is called retention, and then finally you monetize the user. And then metrics, both primary and secondary, are performance indicators on top of each stage in the funnel.

So on top of acquisition, you can ask yourself how many net new users did I get this week versus last week and what's my growth rate there? For engagement, you take a cohort of users.

So from, you know, Sunday to the following Monday, you have 16 people sign up and then you can track that cohort of users week over week and see what percentage of them are still using the product four weeks later, which is a good example of how to track retention. And then we talk about monetization, which is how much net new revenue did I make this week versus last week. Okay.

And then you apply your own custom business funnel to this. So if you're Netflix, we're all familiar that users sign up for Netflix, then they play videos in a loop. Netflix is obviously very sticky, watching it a lot. And then finally, the trial runs out, you do subscription upgrade and you get access to more content. Okay. So how do you collect data once you have this funnel?

So there's analytics APIs out there. I'm using Segment as an example. You basically wanna say user, user one two three in this case, has done user sign up event and they happen to be an organic user, which means they're not invited by someone else. Then if you're Netflix, you might say the user is video played and then eventually subscription upgraded.

And so this is how you instrument your tracking in your mobile app or your web app. Then you think about event properties. So imagine you're Netflix and you're holding one of these video played events in your hands and you're wondering questions about it. So, you know, what video is the user actually playing? How long is the video? How far did the person get inside of the video? Right?

Equivalently, if you're holding a subscription upgraded event, you're going to want to derive monetization as a North Star metric. So if you're a subscription business, you wanna send your monthly recurring revenue. If you're a transactional business like e commerce or retail, you wanna send the actual value of the transaction. Okay.

So then you push this out into your web app, your mobile app, then you start seeing the data come in. You look at the debugger, you see, okay, user sign up is here. Everything looks good. You add your first analytics tool. We'll use Amplitude as a good example here. Amplitude and Mixpanel are pretty awesome analytics tools out in the market right now.

And then you start seeing data flow inside of one of these analytics tools. So this is Amplitude. You can start seeing user sign ups growing as soon as you launched the real like mobile or web app. Okay. So now that you have analytics set up, it's time to focus on three different metrics. The first one is the acquisition metrics, sign ups per week.

It's really nice if you're a b to b business to cut this by the invite type. So you have organic users which are just signing up from coming to your website, direct sign up. And then some users are inviting other users, so those are invite type, right? So when you're thinking about growth, it's really important to think about the organic user in that case. Right?

Another example why event properties are important. The way you create this is you go to Amplitude, you say event segmentation report, user sign up, next, here's your graph. Right? So it's as easy as that. Just website, send data, gets to Amplitude, and then you can see the amount of organic users every week.

And then if you're working on the acquisition step as a secondary metric, you can basically say, today, 218 users organically signed up in the last week, but by the end of the month, we want that to be at 300.

And we're going to execute projects a, b, and c this month, and then we're going to watch this graph every single day on a TV dashboard in our office or our apartment, wherever we work, And then we're going to see if our efforts are actually driving this, right? So that's an idea of data driven operation of a team. You set a metric and a goal, and then you drive towards that every day. Okay.

The second one is retention cohorts. So someone recently asked about retention. We'll talk about that right now. So with retention, you wanna think about cohorts of users. So you wanna say Monday to Sunday, let's say December 10 through December 17, '16 users signed up.

And then you look at those 16 users as they use your product on week zero, which is their sign up week, week one, week two, week three, and week four. And the general idea here is like, you can convince your mom or your grandma to use your product once, but even your mom or your grandma won't continue to come back and use your product over time every single week. Right?

And so if you see users that are addicted, are coming back week over week, that's a really good sign of product market fit. So this business can see that the December 10 cohort, only 6. 25% of those users are still around on week four. And that's a pretty low amount, right? So you probably want that to be somewhere between 20 or 30 at least.

And so you can set a goal saying, I'm gonna talk to a bunch of these target users and try to figure out why they're not getting value out of the product and then make some changes as well. Okay. So what metric do you actually pick? This is taken from one of Gustav's slides, pretty awesome. You think about what value your company is giving to your users.

So Airbnb gives you value by letting you stay at different rental properties around the world, right? And they want you to do that at least one time a year, otherwise they consider you a churned user. Equivalently, Facebook gives you value by letting you look at the news feed and you know, connect with your friends and they want you to do that at least daily or monthly once.

And so when you think about product market fit, you basically have these two different curves that happen. So we have that cohort of 16 users that signed up in one week and we track them over time. And so what ends up happening is for products that don't have product market fit, they end up tending to go to zero because people just don't care about the product. Right?

And that's the definition of product market fit. For these tools and the products that do have product market fit, you'll see some kind of natural plateau. Don't mind this axis. It should be somewhere, you know, between 2030% at least. Okay. So how do you create this graph? It seems kinda complicated. Right?

Luckily, both Mixpanel and Amplitude have really awesome reports for this. So in Amplitude, it's a retention analysis report. You say users enter the cohort with user sign up and then they return with the video played or the subscription upgraded event. That's the value event. And then you press next and out comes this graph.

And then you could look at four week retention for cohorts, improve the product and watch as new cohorts that strike the four week mark do better or worse, right? And that shows you whether your changes week over week are actually improving. Okay, finally, revenue. So this is the primary metric that you want to be thinking about.

You'll use for a subscription business, the subscription upgraded event. You'll do a property sum over new plan monthly recurring revenue. You press next and out comes your weekly net new revenue graph. And then you could set monthly goals on top of this to make sure you're growing at the rates that you want to be. Okay.

Finally, if you have a founding team, it's really good to basically put all this stuff on a dashboard, and then put this dashboard on a TV in your office. It's incredibly, incredibly important. Basically, this is kind of the difference between being a data driven team and not a data driven team.

A lot of founders actually set up their analytics, but then don't look at them ever again because it can be painful. Right? While a data driven team will put it on a TV and talk about projects, talk about whether those projects are actually changing the metrics that they're trying to drive, and then just completely understand the business every single day. Right?

And this kind of company and this kind of founder will actually scale to build better high performance companies because the next team of employees they hire will also be looking at that same TV dashboard and be driven off those same metrics. So really important, get a TV. Next, what you wanna do is have some kind of social accountability around your metrics.

So if you have your friends, your parents, your advisors, your investors, package up how your business is doing into an email. This helps you synthesize what is actually happening, and then send it out to those advisors, and tell them where the business is struggling and what your plan is to fix it.

This allows the advisors to quickly understand the business and then respond back with much more appropriate advice. Cool. And now we'll go into the startup stack. So these are tools that we recommend that help this kind of tactical process of setting up these metrics. So I'm gonna talk a little bit about that MVP business workflow that Michael talked about earlier.

So initially, you're building an MVP. Segment built about seven different MVPs before we actually found Segment and all of those failed and eventually we found we found one that worked. And the process of actually building MVP is incredibly important.

So once you have that little experiment built, you wanna enter private beta, which basically just means getting like ten, twenty, 30 customers to actually try this product, and then having very direct lines of communication open with them.

What Segment does nowadays, every new product we ship, we open Slack channels with each one of our customers, and we have the product managers sit in those Slack channels and talk with the customers. For the products that don't get product market fit, the customers just stop responding, and we're asking, asking, asking, they're not responding, right?

And for the products that do have product market fit, the customers are immediately being like, oh, we don't why don't you have this feature? This is broken. I tried inviting my team in. This is not work not working. So instead of you kind of pulling at the customer, the customer starts pulling at you. That's a good feeling of product market fit. Okay.

So at some point, the private beta is going well, you feel like people really care about this, you understand your target customer, then you want to get a larger market segment to use it. That's the launch that we talked about earlier. Try to get there as quickly as possible. And then a launch is just more users that you get to test product market fit on.

And so if you feel product market fit there, then you can start scaling the company, right? And you hire salespeople and you start doing paid marketing and things like So different tools will guide you throughout this process. So as you're building an MVP and you're about to give it to the first group of customers, install Google Analytics, install Amplitude.

Google Analytics will tell you who's coming from the Internet to your website, and then Amplitude will tell you which features are they using, how engaged are they with that feature set. Unless you're able to stand over the shoulders of all of your users a % of the time, Analytics is the next best alternative for that.

We also install live chat on the page, so either Slack with your customers or if you can't do that, then maybe have a live chat available. In the beginning of segment, customers would ping us day and night, and that's where we got the most valuable feedback from them. So just as many open channels of communication as possible. Next, data warehouse. This is something that we recommend.

It used to be expensive. It's no longer expensive today. Basically, if you have a non technical co founder on your team, they'll wanna ask questions around the data, and they'll always ask the technical co founder who will have to provide the answers. So data warehouse kinda democratizes the data, not only for the non technical co founders, but for everyone else in the company that you hire after.

Company dashboards, obviously, I should probably move that to the left. Email and push tools. So as soon as you as soon as users sign up, you want send them an email. I'll talk about that in a second. And then and then a help desk. So at some point, you'll have so much support tickets if you start feeling product market fit.

And if they're all going to your Gmail, one founder will just get overwhelmed and not be able to answer them. So you want to have a shared inbox where multiple founders can respond. Okay. Now I'm going go through a few different recipes of these different tools that we found really helpful in product market fit. So the first one is improving product usability.

Almost every product that's launched is unusable or highly unusable for the first three months while you have the kinks. And we see this with every single product, no matter how much effort we put into it ahead of time. As soon as customers hit it, they start using it in ways that you just don't expect.

And so there's this tool called FullStory, which helps you look at sessions of customers as as they use your website. So I'll tell a quick story on this. This is a new feature in Personas, which is one of Segment's products. And we launched and the metrics looked horrible. So like customers were coming in, but they weren't actually completing it. They weren't they weren't using the product.

And we thought, oh god, know, this likely doesn't have product market fit. We have to go back to the drawing board. Then one of the designers on our team had this amazing idea, let's look at the full story. And so we see this user going in about to start this creation workflow.

They find this button, they clearly don't understand what the button does, they get so frustrated, they just exit the page. And so we saw this with multiple different customers coming in, and so we're like, okay, we just have to fix that button. We fixed that button, immediately all the metrics got better. Right?

And so that's why it's important to have this type of viewing, either stand over your customer's shoulders, which is great, what the Stripe co founders did, or get full story, which is a more scalable way to do that. Okay. Call this the forty three minute founder email. So when we launch a segment, we would wait about forty three minutes and we would email the customer and say, hey, I'm Ilya.

Thanks so much for signing up for Segment. Your next step here is to add a source to Segment. And if you have any questions at all, please email me or call me anytime I'm available for you. Since we launched that email in 02/2013, we've had hundreds of thousands of responses to it. So it's the connection between you and the customer over email that if they get confused, they'll respond to it.

What you can use is a tool called Customer. io. It's a behavioral email tool, which will say, every time a user signs up, wait, you know, thirty minutes, forty minutes, fifty minutes, whatever, and then automatically send them this content. And then you could template the first name, the company name, and so forth based off of your analytics data. So huge recipe we recommend.

And then finally, for democratizing data access, I would say this is more advanced. So this is after you're in your MVP stage, you're feeling good about product market fit. You might wanna install data warehouse like Google BigQuery and then Mode Analytics is a BI tool that works on top of it.

This lets you ask questions on top of the raw data that you might not be able to do in Amplitude and Mixpanel. Just any kind of question you can ask with SQL, and then even the non technical co founders will eventually pick up SQL and then start asking these questions themselves. Okay. One common failure mode that we see with customers is trying to pick the perfect tool.

Mixpanel or Amplitude, you know, BigQuery or Redshift, and spending way, way, way too long thinking about that. The truth of the matter is, you shouldn't optimize for picking the right tool right now. Both Amplitude and Mixpanel will give you exactly the same result at your stage. Instead, get through that decision as quickly as possible, but set yourself up for change in the future.

So this is an example diagram that shows a customer of segments that used different tools over a period of about three years. And so you can see that they used about eight different tools between 2,015 and 02/2017. Then they either hired someone or they decided that their tools were no longer doing the job, and they switched from one set of tools to another.

So best in class tools change every two years. Just be prepared for change and don't spend too much time trying to perfect your your choice right now. Okay. This is my recommendation of what tools are the best for the MVP process. I'll walk you through them right now. So Google Analytics is for understanding what users are coming to your website. Amplitude is for feature analytics.

Google BigQuery is to democratize data access with a data warehouse, which is just a database of data. Mode is the the tool you use on top of that to ask questions on top of BigQuery. Intercom is like a list of all of your customers. They're really good CRM for early stage folks. FullStory is for improving product usability. And Customer. Is for emailing your customers.

Once you get big enough, can start using Google Ads, Facebook Ads to actually do the paid acquisition. Okay. So I'll just jump straight to the slide. When we were really young, we or when we were in 2011, '20 '12, we didn't have a lot of money and so we didn't want to use all of these different tools. And so we used Google Spreadsheet for our CRM.

We didn't use Asana or Trello because they were too expensive, so we used emails. And then the only things we paid for were basically GitHub and AWS to host our product. Right? So today, happy to say that Segment is now free for early stage startups. So for all you folks, there's a Bitly link down here that we'll send out afterwards and it's in the deals.

And then also Segment went out and we did a bunch of deals with these customers, so that if you're early stage, you can now get all of these tools for free. So enjoy that, and start using them to basically accelerate your product market fit process. Thank you. Thank you.