How to Optimize Your Product Using Analytics by Dan Olsen

yeah so today I'm going to be talking about how to use analytics optimize your product if you recall my talk yesterday didn't really talk about analytics much it's much more about qualitative techniques to define your product get clearer on your customer get clear on your value prop right so that's all before you launch your product and today is all about let's assume we've launched our product and now we actually have a product in market we can use analytics to improve it and optimize it I'm real quick about Oh real quick about my background I mostly got the intro yesterday technical background started out in submarine design moved out here a while ago to go to business school that's where I discovered p. m. as a career I asked people where's the best place to learn PM since I'd never done it they said into it so I worked it into it for five years which was a great experience after that I worked at startups I did my own startup and I became a product management consultant which I've been doing for nine years now I'm also very passionate about sharing best practices so I've for four years now I've been running a monthly speaker series called lean product and lean UX it's in Mountain View I'll tell you guys more about that my twitter handles at Dan Olsen I'm gonna be giving away a few books that's how you can win a book and I post all my slides and videos on dan - Olson comm if you want to check out the sides later so since this is now our second day together for most of us I actually want to share a secret that's not in the book it's one of the closely most closely-guarded p.

m. secrets out there and it's the p. m.

motto and I'm guessing most of you have not heard before as anyone knows spider-man's motto we're in Silicon Valley someone's got to be Marvel yeah okay anybody else got it with great power comes great you got the keywords that was good that was good it would have come up on a Google search yeah with great power comes great responsibility so the product manager motto is similar but it's a little different basically and it's with great responsibility comes no power so we're all laughing we're crying a little bit on the inside because we wish we had some power but we don't but anyway PMS are responsible for a lot of things I had a lot of great discussions with PM's yesterday about all the things they need to do you know if you don't have a designer then you've got to do some wireframes you always fill the gap so we're responsible for a lot of things and I've been speaking to audiences and product management since 2007 answering questions and building content and that's what led to the book basically the lean product playbook which came out almost three years ago now and I wrote it as a comprehensive guide to all the things that you need to know as a product manager and it's meant to be a playbook to give you a hands-on guide like a six step process which I covered three of the steps yesterday right I'm actually like I said I'm going to happy to give a couple of copies away today the way you win is just by tweeting you have to be here to win so sorry for the folks online and if you want to throw on a hashtag like product counter product management that'd be great so again the context for what I'm talking about today is you've got a product in market and you want to use analytics to improve it right before you've launched your product it's hard to have any analytics once you've launched it you can and just like I had the lean product process for pre-launch I have a process for how to think about an approach using analytics to improve your product post launch and that's the lean analytics process so this is chapters 13 and 14 in my book basically it all starts out with figuring out what are the metrics that we want to track that aren't basically important figuring out the baseline value like where are we at today right if we're not tracking that today figuring out for each metric which one offers you know how much upside potential is there what's the ROI if we try to improve that metric right thinking about that and this is all at the global level for your whole product in your whole business basically right and so what you want to do is look across those and figure out which one has the biggest upside potential once you identify that metric then you're going to select that metric and focus on working on that metric and we're gonna go into a metric specific optimization loop now right for that metric and what we're gonna do there is basically as the team said great we want to improve this conversion rate we're gonna brainstorm different ideas for how to improve it we're going to figure out which idea we think has the highest ROI we're going to design and roll it out and we're going to see how the metric changes and we hope that it goes up into the right right but if it doesn't we've still learned right so there's value in going to the loop even if the first or second time you don't really move the needle a whole lot on that metric and we're learning and iterating and overtime you will find something that moves that metric over time you'll find something else eventually over time it'll start to get harder and harder as you find the easy ways to improve that metric it's gonna get harder and harder you're gonna get diminishing returns so at some point it makes sense to get out of this loop go back to the global set of metrics and say okay we we fix that out conversion rate as much as it makes sense to you right now let's go back and figure out what's the next metric that we should be improving right and when you're thinking about metrics there can be a million different metrics that you can track it can be a little overwhelming so it can be helpful to have a holistic framework for how to think about the metrics and a while ago dave McClure created a great holistic framework called start up metric for pirates it was mentioned by tal yesterday as anyone who's heard of this before are or start at metric for pirate right so it's a great framework and what's so great about it is it applies to any business it's a way to just start with metrics at a high level and cover the landscape and it's a ar-ar-ar because that's the the first letter of each of the five parts so the first one is acquisition acquisition is all about you know how do we get people to our website or to our app to learn about it if they don't even know about how do we get them to the point where we can tell them about our app and hopefully convert them to customers they're not customers yet they're just prospects right now the next a the second day's activation and that means how do we convert those prospects if we got them to our landing page or our site what percentage of those are we converting to actual customers however you define customers it may be paying it may be registered maybe an email whatever it is that's what that means and I usually would call this conversion but then it wouldn't spell our like a pirate so it would mess it up so it's called activation basically then once their customers by definition they're using your product right that's what a customer means they start using your product not all of them keep using your product hopefully they keep using it that's the third that's the first are the third were letter is retention right what percentage of people are continuing to stay active right and then the next are is referral which is hey hopefully as a function of people using your product they like getting they tell their friends right Josh Ullman talked about three different types of reality one is word of mouth one is demonstration the other one is you may actually have some social mechanisms where people can invite their friends or if you're a communication product like sock you're gonna have some implicit viral loops there right so for b2b this is often less important it's not like the CMO of IBM is Facebook friending the ce CMO of Adobe saying you gotta check out this b2b app right so that one is the one that may not apply as much in your set case certainly for games and social products it's critical but these other three are definitely important and the final R is revenue right this is hey by virtue of the customers using our ducked we should be making some money they're paying us or we're making money by monetizing their behavior or something like that right so that's that's the last one right and the goal here is to you know you can hopefully you can see how this applies to every business is to figure out hey at this point in our business which one should we be focusing on there's five choices here which one at a macro level offers the highest ROI and so you want to focus on the right metric at the right time and I call that the metric that matters most for right now right and that will change over time MT mm for short it's like right now for our product for our business what's the metric that matters the most to do a quick little mini sample let's say we're we have a new product we just were about to launch it next week or we just launched it say today and we're like just trying to decide should we focus on acquisition should we focus on getting more prospects to our webpage should we focus on conversion you know getting people that come to our webpage to convert to customers or should we focus on retention one of these is gonna be better I offer the highest ROI right now and there's no one answer for every situation but in general the answer that I give is if we just launched our product more than likely our retention is probably never we want it to be not where we want it to be so we can go spend a lot of money on acquisition and buy Google ads and Facebook ads but then people gonna come we have an optimized conversion yet so not that many people are going to convert our product isn't too sticky yet so a lot of people are gonna stick by drop out right so in general makes sense retention first till it gets to a decent point and and I talked a lot in other talks about the leaky bucket metaphor retention is basically how much water is staying in your bucket versus leaking out customers being the water then conversion and then acquisition that's the way to get the most ROI on your marketing dollars in general and then I like to you know there's again there are a lot of different metrics you could focus on in the context of product market fit which is what my book is all about I think there's actually in my opinion there's one metric if you just have one metric you say hey we just launched a product and we want to track our product market fit I think there's there's one metric so what I'd love to hear what you guys think some of the metrics that would be good what metric do you guys think would be good at track if you just launched a new product to see where you're at with pockmark effete referral okay what else time value retention yeah people actually purchase yeah there's a lot of different things you can do right and again I think just in the context of part market fit I personally like retention rate you know just got done talking about the leaky bucket stuff right so I want to explain what retention rate is because it's a certain way of looking at it but it's basically what percentage of people are still remaining active over time and it's best understood and usually displayed as a retention curve so what you do let's take the easy example say we just launched a product we take all the customers that sign up in the first 30 days and we throw them in a bucket and we say ok let's watch them not calendar days but for each user let's index the time frame for when they join and first started using it and then let's see how they drop off over time and they will drop off over time right so the first thing you notice on this curve is this it doesn't start at a hundred percent it starts at twenty seven percent what does that mean that means like over around eighty percent of people after they use it the first time never came back this is a real retention curve for mobile apps it's pretty brutal there will always be a gap everybody loses people they go and they they either forget about it or they just realize it doesn't meet their needs there's a lot of reasons why people don't come back right so that's just a reality then it decays and then it eventually goes down and it can do one of two things at the end here I've shown it where it flattens out which is the good case but for most new products frankly it goes down to zero it's just a question of when is it 30 days 60 days 90 days what that means is all those customers you work so hard to get in the bucket eventually all leak out right and so that you know obviously that that's that's basically what you need to see to see how you're doing if conversely you're holding on to some people that's great that means these people have stuck with you for 90 days your product is doing something valuable to them now over time as you improve your product you're gonna want to not all throw all the customers into one big bucket and analyze them as one big group you want to slice them and analyze them separately and say hey you know great that's those are the guys in the first month let me handle those bugs and those issues but we fix those let's look at the next group of people that's where cohort analysis comes in all that means is dividing your user base into different segments and looking at the retention curves for each of those segments typically done by time let's take the people that sign up in January February March but it doesn't have to be by that it could be by acquisition channel it could be by customer segment but here's an example of cohort curves so we have three cohorts here again we have what percentage of users are returning over time this time we have weeks and sign up instead of days but we've got three cohorts court a is in the blue diamonds court B is in the red square cohort C's and the green triangles I need some audience participation here so speak up guys I know it's the beginning of the morning think about imagine you were the product manager for this product which cohort would you rather have as your cohort of users who would rather have cohort a blue diamonds as their cohort raise your hand who would rather have court B red squares as a cohort region who would rather have green triangles court C did you guys have a secret meeting before my talk and say when Dan puts it up let's do cohort C no but do you have PhDs and retention rate that I don't know about no your brain just knows why is it why did you why did everybody pick the green triangle it's got the I know no blues the highest blues the highest green set a highest at the end right your brain knows what matters the most is what's highest at the end right but yeah so your brain knows that right and and when I give the other talk that's all about talking to people in groups of five to eight at a time I sometimes get the Spock quant types that go yeah Dan but that's not too significant alright and that's just people talking to you so the secret real secret of if you want to have a quantitative measure of product market fit that's behavioral not attitudinal it's where this line ends up so what I'm telling you is we're if this flattens out if it goes to zero you don't have product market fit if it flattens out whatever percentage of flattens out at that's a measure of your product market fit it's a real-live believable measure and the higher it is the better right over time what you want to do over time is say we launch our you know MVP beta two years ago twenty four months ago and this is our retention curve you know we managed to you know hold on to say 5% of people after 12 months what we want to do say the next six months we work really hard listening to customer feedback fixing bugs adding features fixing UI fixing messaging well we hope to see is if we look at the next group of people that sign up six months later that that whole curve moved up and most importantly that the tail value the asymptotic value at the end went up and hopefully in the next six months we continue to better understand our customers and improve things so that's what you want to see is see where you're at on retention right and make sure you're not at zero if you are get it flat and then steadily over time improve it that's how you improve product market fit and use analytics I like to share real world data to support my points whenever I can this is real word data from the Android app store so these are retention curves percentage of users still active days since app install and what they've done is they've done four cohorts they've done it in a very particular way the red curve up here that's the top ten apps in the Android app store the next curve is the next 50 apps and X is an X hundred the next is the next 5,000 so what does it show it shows that the top apps have the best terminal retention rate and then the next best apps have the next best right so it just shows you that basically it's a direct measure of that now so you want to you know make sure your buckets not leaky enough right and then you can focus on conversion and acquisition the other parts of the are startup metric for pirates framework at some point you need to go beyond that framework it's great to get your head around the diff five different areas and to focus and get retention down and get acquisition and conversion at some point when you get to the revenue part you need to take into account the specifics of your business because different businesses are in different revenue in different ways that's where the concept that I talked about in my book the equation of your business comes in you know and I have a technical background engineering background so if I want to optimize something I want to be able to express it quantitatively as an equation and so that's how I think about your business and if you're thinking about gosh I don't know how to approach my business as an equation you can always start with profit equals revenue minus costs and for most tech businesses what matters the most is the revenue the marginal cost isn't as important as a marginal revenue and so the key here is to break the revenue down at this is an actionable so yeah we want to go grow revenue that's not actionable what we want to do is break it down mathematically so we get to the point of having actual metrics I'm going to do that for a subscription business model let's say we have a success a subscription product and we have like a 30 day free trial that people can do and then they have to upgrade right we can break revenue down into by the way this is for a given time period a given month a given year whatever it is we can break revenue down into how many paying users do we have times what was the average revenue per paying user may have heard of our poo as an act for that right so it's paying users time ARPU right that's a way to break it down those two together just equal that and then we can break down the paying users into okay for this month that we're looking at how many new paying users did we get and then how many would repeat paying users do we have from the previous time period when you're doing these exercises almost always beneficial to break distinguish between new customers or users and returning basically right the repeat paying users actually just ends up being well how many did we have in the last time period times 1 minus some cancellation rate or churn rate basically right so this is all just we can Moltres this whole thing out revenue equals this long thing if I wrote it out right back on the new users again I mentioned we had some free trial so you can say well how many trial users do we have and what was the conversion rate that will give us how many new paying users that we have and for the trial users we may have various marketing channels like SEO SEM viral and we have a conversion right so we started out with the high level revenue that's not actionable and now we've broken it down into several items that could be actionable all right we could say hey do we think it's better to focus on the trial conversion rate if we exactly the best way to improve revenue or is decreasing this cancellation rate the best way or is increasing the revenue per paying user the best way or is you know fixing our marketing the best way that's the kind of discussions that you can have right and for SAS products because of this one - factor churn rate can be very very powerful it may sounds like a small improvement to take your churn rate from 6% to 3% but it has like a nonlinear been improvement in like revenue and lifetime value so anyway what we want to do is break this down for your business and then the what I think about each one of these detailed metrics I view it as a gauge or a dial right and a gauge or a dial is a numerical measure it has a minimum possible value it has a maximum possible value this would be for a percentage metric right it goes from 0 to 100 and then it has you know where is it at today I couldn't find one that had an arrow pointing up here so that's a current value right but that's the way I think about each metric is what's the min value what's the max value where we at today and that's the framework to figure out which ones are the critical few as we look at each of those gauges how much upside potential do we think there is for each of those right how much do we think we can move the needle hey if we think if we like redo the design of this page we can move the conversion rate from 5 to 10 right what would the revenue impact be using the equation of the business we can calculate what that would be and how many resources do we think that it would take to move the needle that much it's usually developer time maybe money and then what we want to do is look across again the metrics and say which one do we think offers the best ROI now when you do that you typically see one of three ROI metrics ROI profiles the first one is what I call a good ROI and the way I show this is we've got return here and investment like if we invest and the and and the the yellow dot is where we are today and the white line shows if we do the best ideas if we pursue the best ideas and we invest as much how much return will we get so hopefully that makes sense this is a good ROI right we're down here with little bit of investment we can go up here and get a lot of return right there's bad ROI situations right here we're up here on the curve now and so we can put in a lot of investment but it's not going to lead to a lot of return and those are the main ones that you see but there's a third type that you can see if you if you really analyze things carefully and that is one like this where if you analyze things just right you realize gosh there's something that really with a little bit of investment can make a really big improvement in the return so the return curve is fundamentally different shaped I call these like Silver Bullet opportunities right and they do exist especially if you haven't started doing analytics optimization or I want to close out with a case study of applying the lean an election process from end to end and again I need some audience participation you got to pretend you're the product manager here I used to be the head of product management Friendster back in the day did anyone use Friendster back in the day alright cool so it was the first social network and so it was a great opportunity for me to back in 2004 well work on viral loop optimization when I showed up Friendster everyone in the building was like viral growth is critical we need to get new users from our existing users without paying obviously it was implicit everybody believed that I'm like great what are we done on that nothing I mean I think okay all right I'm gonna apply my process and I need to figure out what framework and and how to approach this so the first thing I did is it okay let me define the viral loop and then let me figure out what metrics I can put on there that make sense right so the viral loop is basically we have active users that are using our site they have the opportunity to invite their friends right not all of them do but some of them do the friends get an email saying hey you know Daniels I want you to join Friendster they either click on the email or they don't if they click on the email they get to the registration process either get through it or they don't and then if they get through it they become users some of which remain active right so the first thing I did is I define this kind of UX flow and where people could drop off the next thing I did is okay let me figure out what metrics I can instrument here to track and optimize this the first thing was what percentage of users are active right not all of them are active the next thing was what percentage of users are sending invites not all of them are sending invites in a given month or a week or whatever right there's actually a second factor when you invited your friends it's not like you had to invite one at a time you can invite multiple so there was invites per sender as well to kind of capture that step the next one was just hey what's the click-through rate on those invitation emails and the next one was okay what's the registration conversion rate what percentage of people are getting through right so those were the five metrics that I use to instrument that loop and now you know taken together they mentioned your viral ratio and now I had to do you know this is a tough part I'd be like great which one should I focus on I'm the PM I'm gonna headed p. m. for Fenster I've defined five metrics which one should I focus on I want you to picture that you're in those shoes and I have I have a little bit more information you guys did but not a whole lot I just started let's make it simple let's boil it down to these three percentage of users sending invites invites percenter and conversion rate I want you to think about which one would you focus on again I know you don't have a lot of information raise your hand please who would focus on percentage of user sending invites okay who would focus on invites per sender who would focus on registration conversion rate okay that's the biggest pop that's the most popular one right the next step in the process does anyone remember what it was it was like what are your baseline values I don't like tricking people but I didn't show you the base I had you vote before we showed the baseline values I do that to show you the power of baseline values you ready here's the baseline values percentage of user sending invites was at 15% of users were doing that average invites percenter on average when people invited friends it was 2.

3 and the registration conversion rate was 85% now that we have baseline values why don't we revote who would like to focus on improving percentage of user sending invites got some more hansi that's the power of baseline you people that change your vote all you need to know you have no idea how invites work or anything just know where you're at and I'll talk about that in a sec who would focus on invites percenter now and who would focus on conversion rate a lot less right great awesome so what's going on there's a quick hack again your brain knows this stuff right just by knowing the baseline value you know it's called the upside potential of a metric that's what I call it and not knowing anything about the metric or the product you can usually make a pretty good guess on which one is going to offer the best upside potential what we do remember I said each of these is a gauge right has a minimum value what's the minimum value for registration process yield zero maximum value 100 because it's a percentage what's the baseline value where are we at today we're at 85% so what we can do is say hey let's quantify what the max possible improvement is not saying we would get it but if we just you know has the most amazing outcomes and ideal you know results what's the most we could do we're at 85% so the max is 15 more points right if we divide that from where we are it's an 18 percent from the maximum improvement if we could do everything perfect in the world would be 18 percent right the second one what's this one percentage of user sending in but it also goes from zero to a hundred percent right where we at we're at fifteen percent how much Headroom do we have here we have 85 percentage points that we can go up we divide that by 15 that's 570 all right that's why your brain just knows hey 15 versus 85 that's no-brainer that it's got a bigger upside potential right this third one invites average number of M at ten percenter what's the minimum value for that one zero what's the maximum value infinity next how many people in the world seven billion how many on the Internet four billion all right cool yeah so it's funny I'm okay affinity it's like guys you get all excited like yeah all right but it's big right I don't know we're at 2. 3 right we question mark divided by 2. 3 equals what right I don't know but there's a really good chance it could be even bigger than this one right it's fundamentally a different metric that's why it's important thing about the min and Max these are percentage metrics right so is anyone feeling a sense of deja vu right now from something I said earlier remember I said that there's three profiles right there's a bad profile of ROI metric that's what that one is right we're up here we're an 85% it's gonna take a lot of effort a lot of effort to not get a lot of return second one good ROI curve right we're down here we feel like hey there's a lot of upside that we can do last one I'm not sure we're not sure yet but it seems like this average number of invites m per senator could be a silver bullet metric because it's got so much he's gotten near infinite potential right up there right okay so for those reasons we decided to focus on this right the next step you do in the process is great we're gonna focus on that as a team let's brainstorm all the different ways we can prove that metric right and for each idea we try to estimate what's how much do we think it'll move that needle on if people are inviting 2.

3 friends today for the idea that you just brainstorm what do we think that will move it two three four five six what do we think it'll do and then we think about what's the expected cost how many engineer weeks or days is it gonna take and you want to run some high-level ROI filter and figure out what you want to do what we decided to do it's they're popular now but back then one thing you didn't know is to invite friends you literally had to manually type in an email addresses there wasn't an address book importer and so we were the first social network to add that we decided let's add it and then again we had to do the ROI cuz you had to build one for each Gmail for Yahoo for hotmail you had to build one so we basically realized hey most of our users on Yahoo let's do an MVP of that and see what it does so here we go we've got our baseline value cruising along 2. 3 again this is how many when somebody invites their friends how many do they invite per time right 2.