Behind the product: Duolingo streaks | Jackson Shuttleworth (Group PM, Retention Team)

dingo is a$1 14 billion company that's hitting alltime highs too just keeps going up I think it doubled in value in the past 6 months streaks is the most impactful feature we have right now over 9 million users with a year plus streak if you look at the numbers I think has been our biggest growth lover what dualingo really focuses on is you know how do we help users build habits around language learning getting user come back the next day is the biggest problem to solve let's get into the motherload of learnings from the Journey

of streaks talk about the key lessons insights and also turns along the way I'd say like test everything we've run in the last four years over 600 experiments on the streak so every other day we've actually set up really good infrastructure for copy testing we used to say continue our like standard CTA is continue and we changed that to commit to my goal and it was like a massive win there's so much human psychology that you all learn through all these experiments of just like how to motivate people would motivates what demotivates say that you

played a mobile game that you've done it for 3,000 days in a row I don't know maybe that hits a little bit different than you've learned Spanish for 3,000 days in a row today my guest is Jackson shettleworth Jackson is a group product manager at dualingo leading the retention team this is a different kind of episode that I'm experimenting with where we spend the entire conversation focused on the journey and lessons of a single feature in this case dualingo streaks dualingo is a $14 billion business just over the past 6 months they doubled in value

they're hitting all-time highs in usage and market cap they're also one of the very few successful and also the single biggest consumer app business in the world and as you'll hear from Jackson the streaks feature is the single most impactful feature that most contributed to this growth and success in other words you could argue this one feature created billions of dollars of value which to me means it is worth studying in depth in our conversation Jackson shares the history of the streaks feature all the biggest wins and wrong turns they've taken along the way what

he and his team have learned about what works and doesn't work with a streaks mechanic and also how they set up their teams to operate in a way that allows them to run over 600 experiments on this product and continue to find big wins I hope to do more episodes like this on features and products that you'd love to hear more about so leave a comment either on YouTube or on substack and tell me which product or feature you'd love to see me cover and if you enjoy this podcast don't forget to subscribe and follow

it in your favorite podcasting app or YouTube it's the best way to avoid missing future episodes and it helps the podcast tremendously with that I bring you Jackson shettleworth Jackson thank you so much for being here and welcome to the podcast yeah thank you long time listener first time caller so I appreciate it so this is going to be a really interesting conversation I've never done an episode like this before where we basically Al spend like an hour an hour and a half going into one feature of one product but this is a very special

feature it's a very special product um have you ever spent like an hour an hour and a half just talking about this one feature with anyone well internally so when we when we onboard new folks to the team we'll do I actually just did this with somebody that joined the team recently where we spent and I was like hey let's spend an hour just talking about the streak we got through an hour and I got through about 30% of of what I I I wanted to share um there's just so much we'll talk about this

but there's so much that we've learned over the years uh but never anything externally like this I think we've shared bits and pieces of learnings but this will be like the the motherload of learnings hopefully how duelingo built the streak we should title this the motherload of learnings of duelingo streaks this episode is brought to you by pendo the only all-in-one product experience platform for any type of application tired of bouncing around multiple tools to uncover what's really happening inside your product with all the tools to you need in one simpl to use platform pendo

makes it easy to answer critical questions about how users are engaging with your product and then turn those insights into action also so you can get your user to do what you actually want them to do first pendo is built around product analytics seeing what your users are actually doing in your apps so that you can optimize their experience next pendo lets you deploy inapp guides that lead users through the actions that matter most then pendo integrates user feedback so that you can capture and analyze what people actually want and the new thing in pendo

session replays a very cool way to visualize user sessions I not surprised at all that over 10,000 companies use it today visit pendo.io Lenny to create your free pendo account today and start building better experiences across every corner of your product PS you want to take your product LED KN how a step further check out peno's lineup of free certification courses led by top product experts and designed to help you grow in advance in your career learn more and experience the power of the pendo platform today at pendo.io Lenny pend this episode is brought to

you by vanta when it comes to ensuring your company has top-notch security practices things get complicated fast now you can assess risk secure the trust of your customers and automate compliance for sock 2 ISO 2701 Hippa and more with a single platform vanta V's Market leading trust management platform helps you continuously monitor compliance alongside reporting and tracking risk plus you can save hours by completing security questionnaires with Vana AI join thousands of global companies that use vanta to automate evidence collection unify risk management and streamline Security reviews get $1,000 off vanta when you go to

vana.com plenny that's v.com Lenny okay so for people that don't know anything about what dualingo streaks are can you just first give a brief explanation of what is dualingo streaks what is this feature all about well presumably people know uh dualingo is a language learning app um the the Dual lingo streak tracks currently anyway how many days in a row you've done a lesson so you come to do a lingo you do a lesson your first lesson you know you'll start a streak and then every consecutive day that you come use the app you'll extend

your streak and and I should actually put an AS around how many days in a row because we also have build flexibility into the featur so we have these things called streak freezes it's like insurance for your streak so you know it's a pretty simple feature you know in in theory but over time we've layered on challenges and goal setting and rewards and you know social features you know a lot of our notifications are tied to the streak so pretty simple feature to understand but it's been a really rich feature for us to to build

on top of some people might be this be like du lingo streaks like what's the big deal there's other people that are like holy I want to learn everything I can possibly learn about du lingo streaks so many companies are trying to copy what you've learned from this give people a sense of the impact that this one feature has had on dualingo success and growth this is and and this is not just the you know subjective retention PM talking I think this is our biggest feature at with the exception of the lessons and I think

it's actually worth start that like STS are a great engagement hack I'm kind of of the opinion that any team you know any app out there can introduce a streak and if you figure it out it probably works um you know to retain users but at the core like you have to have an app that people want to use and people really like using dualingo it's fun it's delightful you learn something and so it allows us to layer an engagement mechanic on on top of that like the streak is really powerful so it ships a

disgusting amount of daus again it is just it is it is our um one of our golden geese and and again what's cool is that like you know you look at notifications notifications for dualingo is massive I mean you know us sending better you know whether it's copy or timing so many of our notifications work because they reference the streak because users care about the streak and so not only is it itself Us iterating in the streak a huge dri of daus but it's also something that enables other really high you know valuable features I

was looking up some stats before I came on and it's pretty crazy so we have right now over 9 million users with a year plus streak so n million of our users have had uh used duelingo every or almost every day for uh you know well over a year um which which is pretty which is it's I don't know it's I like to imagine things in terms of like you know okay well like if you put all these people in a city or you know in a place where would it be and like ah it's

like a very large city nine million people so for a year have been have a yearlong streak that's incredible I was just looking at the the stock of dualingo so duelingo is A4 billion company at the time of recording this it's hitting all-time highs too it's like just keeps going up I think it doubled in value in the past six months something like that and what I'm hearing from you is streaks is this other than just like the core learning feature which is just like the product of dingo this is the most impactful feature in

terms of growth and in that retention specifically I mean if you look at the numbers I think pretty objectively has been our biggest growth lover for for driving use and also say a lot of it's just related to how we think about growth at dualingo and a lot of what we try to do is organic growth you know we think about growth you know just as much as bringing new users onto the platform is not losing them if you're just bringing people onto the platform then they churn that's not going to be sustainable and so

as much as we can do to keep our users coming back and actually retaining as users it's going to make it it's going to give us a much easier base to continue growing daus off of perfect okay talk about how this feature originally came to be what was the original version of it what was the original Insight that led to be one yeah so the oldest streaks are as old as dualingo itself we when we launched dualingo we we launched with the street and I say we I was like just graduating undergrad when when dualingo

launch um so this is well before my time um but we launched with a streak feature the you know the initially how the how the feature worked was you'd come to do a lingo and you'd set a goal for yourself um and it was an XP based goal so dualingo a little bit of like you know nomenclature we have a experience points based system that drives a lot of our features in the app and the way you'd said it is you'd say you know based on what your language learning goal was you know maybe you

have a 10 experience Point goal versus a 50 experience Point goal and so extending your streak would be hey can you hit 50 experience points if that's what you set your goal to be and this worked well I think this also speaks to like how dualingo initially grew you know we were Luis launched it with a TED talk you know we probably had a Tech forward user base you know initially and so this whole idea of like an experience Point space streak system made sense but what that also meant is that you could have a

user come and use the app do multiple lessons a day and maybe they just set too hard of a goal for themselves and then lose their streak which I I don't you know you don't need to be an expert in streaks to understand that's that's probably not good you know the nice thing with how we initially set it up though is it really did connect with what your goal was so if you are serious let's track how good you are at being a serious language learner but I'd say one of the most impactful experiments we

ran was about this was actually as I was joining we or just after I joined we we had run this experiment um was to Move It from a XP based streak to just do one lesson a day and you'd extend your streak and and you know the risk that you can sort of Imagine is well then users kind of care less about it because it's not connected with their goal and we saw none of that I mean this was huge driver of Da just making it making it easier to extend your streak but I think

really importantly is still meaningful right like the unit of use and you know as you're thinking about building a streak like I think it's really important to think about like what the unit of use of your app is um the unit of use for dualingo is like doing a lesson and so if what we care about is users coming back every day and doing a lesson because it shows that they're actually engaging with the app then it doesn't hurt us to make our streak focus on just do one unit versus multiple and so that was

like probably The Big C change experiment that we ran um you know at the time was moving from an XP based Street to a one lesson streak it's also simple and I think that's like one of the things to think about with streaks it's always easy as a PM to like over you know have a million goals or objectives for what you want your feature to be and and and potentially build a more complex feature you a one lesson streak it's just easy for more users to understand yeah that's exactly what I was going to

say and just for folks that aren't super familiar with XP it's basically experience points and you get them from like doing things it's like yeah you do you do stuff on dualingo and then based on what you did and how well you did it we experience points this actually drives a number of our features in our app so leaderboards is the big one you know we have a leaderboard system where you try over the course of the week you battle with 29 other people and you want to win that's all driven by XP so we

do have other features in the app that really benefit from this XP system the streak is just no longer one of them awesome okay so I want to talk about the journey from that point to what it is today but a quick tangent I saw Lise tweet just this week someone asked them how do you decide whether to optimize for learning or engagement and he's like no question uh everything we do is focused on engagement because you won't learn anything if you're not coming back to the app as an engagement PM that was like the

coolest thing he could have ever said it's well again very much as a you know object subjectively I guess as a engagement PM I mean that's how I've you know I I'm sure my learning the the learning folks at dueling go cringe when they hear it like I see myself as a learning PM as much as an engagement PM because the easiest way not to learn on dualingo is not to come back the next day and so if we don't make the app retentive you will have no opportunity to engage with our learning features now

I do think that there is a long tale of learning where if you start to dumb down and honestly this is this is something we we wrestle with the streak as well you start to dumb down the experience and users aren't actually learning they're not going to care I mean streaks work best when they're sitting on top of an app you know uh that that users care about um but yeah I mean if if you can't come you know if you don't come learn on dualingo you know if you don't come back to dualingo then

you're not learning so um now we I mean we track a lot of this with you know the work that we do we make sure that as we're making changes to the streak we're not hurting the learning experience and we don't have a ton of interaction with it so we're constantly thinking about this but thank you Lis for saying that it makes sense to me okay so let's get into the motherload of learnings from the Journey of streaks so the first version went from XP to one lesson talk about the key lessons insights and also

wrong turns along the way to what we see today dualingo has very much uh you know has a has a strong like test it philosophy um you know we're willing to test a lot of different honestly we'd much rather test it than like debate it for you know days and days so we actually followed up this experiment with and this is this is a little bit later with hey what if we make it even easier to extend your streak and so we actually tested hey if you if you do one exercise just one exercise in

a lesson will will extend your streak and and and a lot of like the Insight was good like you look at the funnel hey there's a lot of users who are starting but not finishing lessons they're not extending your streak the loss of version doesn't kick in they don't you know come back so this kind of followed that train of thought what we realized when we ran this experiment is um you know daus mo moved not one bit and and what we would what we were doing by and this if we go back to like

unit of measure you know we had we had dumb down the unit of and nobody thinks about I just want to come you do one question on dualingo nobody thinks about that um so we had more like a a less clear unit of measure that we were basing our streak around and and the users that we are capturing with our streak you know you come you do one or two questions on duelingo then you leave we're like the least engaged users imaginable and so you know I think that's something also to think about as you're

building your streak is like what is the user that you're solving for so not only what is the kind of habit that you're building for but like what is the level of commitment and that was an example where we over indexed on a type of user who we honestly just weren't going to keep uh that was a very easy shutdown uh shutdown decision that's an awesome story I just just to comment in that real quick just like you went like let's just go to the extreme and make it just like streaks yeah get everyone going

streaks forever and then I love that it turned out and this it's not bringing users that you want and it's dumbing down experience it makes me think of Farmville where you have to go and like Harvest your crops every whatever hour and just like like that lasts for a bit and then eventually people are like what what the hell am I doing with my life exactly no we I mean and we test every so we I was looking at the numbers as well we we've run in the last four years over 600 experiments on the

street so every other day effectively we're running an experiment and then they range from Big like this like changing how the mechanic Works to like let's swap a string with another string and see if that copy is better you know for for users so so we're constantly testing on on every I do think that like I'd be more careful running that experiment now like at some point your streak gets big enough that again I got nine million users on the streak I gotta be really care those are our best retaining users you know you got

got to be careful but in the early days of the streak I'd say like test everything I mean see what don't get don't get super caught up and it has to be like this just test a bunch of stuff and see what speaks most to users because I I think you know again you will constantly be surprised by the the insights that you get from whether you we we shut down about half of our experiments so half of our of our experiments lose we still learn a ton by virtue of of running them so super

super valuable that's actually a good success rate uh a lot of companies have only 20% experiments be positive what's your policy on is if if it's neutral do you ship it or do you kill it it really depends if if we're adding something and it's and it's neutral we tend to to to shut it down because it's just more cognitive load it's something that we're going to have to start building around and a UI element that we have to figure out how it fits into our system I'd say when we do ship a neutral experiment

is something that that we have real uh conviction around okay yeah maybe this was neutral but it's going to give us a new platform to then build on top of so that experience might be neutral but now we can build these da positive experiments and my general take there is though in that case build that in as part of your V1 so that it you know you make sure that like at least your hypothesis around this road map you know has play you know should probably be the case but in general shutting down a neutral

experiment so you don't introduce more complexity to the app is is is the way we tend to go makes sense all right what else what else have you learned along the journey well I mean maybe I can talk about a few different ways that we structure you know a few different themes that I think we we we lean on um so the first is focusing on the zero to 7 Day user experience and I would say this is if you look at the kind of you know what our break break down of experiments are we

run definitely more than average number of experiments on getting users to go from a zero to 7day streak um and a lot of this is because we've looked at the data for you know our retention curves and what we found is that once you get to seven days L of version kicks in and and and you retain so going from a one to a two day streak huge jump in retention two to three day streak slightly less but still huge and it's up until day seven once you hit day seven it flans out it's not

to say that like you know if you're a you have a 30-day streak you're way more attentive than day seven but not in the order of magnitude that it is from from day 1 through 7even so we do a ton of work to get users to that point where L of version kicks in and then they don't want to um they don't want to lead the app one of the fun ones that we did and it's it's honestly as much about process as it is about the feature itself was um we have a feature called

streak goal and it is again so much of the stuff seems so obvious in retrospect um but you know it was really novel at the time you know we we had this idea of like hey maybe we'll just goal users to hitting a certain streak length as you can imagine this is pretty powerful uh you know user psychology um and we started with the simplest version of this and this is how dualingo does a lot of our testing you know rather than design the big complex feature for V1 just do like the simplest encapsulation of

what that feature can be see if it has legs and then just add to it iteratively over time this is partially how we get to you know 600 plus exper experiments on streaks they're not all big ones but we started and it was funny we actually took a learning from our monetization team so one of the strings that they had the pieces of copy that they had worked really really well was I think it was your like 5.6 times more likely to finish the course if you subscribed to plus what you know now it's super

our subscription and it was it was a really good hook that if you if you really cared you'd sign up um and so we ran a similar we we we had a similar thought where it's like oh let's just tell you how much more more likely you are to finish the course if you get a 30-day streak and so we started with that and I think it was like you're seven times more likely to finish the course if you have a 30-day streak and just that message when you started your streak I was telling you

that was awesome huge win um you know indicating an outcome and like dualingo doesn't have we don't we have a gem economy but we don't actually have like you know it's just it's it's all you learning but being able to actually talk about in terms of the outcomes that a user would think about in this case trying to finish the course was a huge win so this is where we started and we're like ah goal setting all right we we should go much harder on that found some found we found and now let's just beat

the heck out of it um so we followed that up with another experiment where we we tested different lengths so we test 14 days and and 50 days and we found that they were all good but they they appealed to different users and so we started to realize all right well we probably need to be more thoughtful about who we give you know know these different options to and and and so then we followed that experiment up with all right let's start with 30 days and then we're going to let you opt out you know

we we'll say no I don't think I can hit it and our thinking was and then if you say no we'll hit you with an easier goal because we just wanted to get you to commit to a goal and this was a fascinating one because it was a good win to give users that easier goal to try to capture capture them before they said no but it was almost just as big a win to add that opt out button so we tested separately and I'm a huge fan of testing like way too many arms for

an experiment just to like be able to isolate your hypotheses but but we captured you know just what happens if we add an opt out button and adding an opt out button when you would think like as a PM oh I'm like you know now users might not engage with my feature that's a bad thing but it was a huge win to let them do that and and the learning here was that this intentionality of saying no I want previously it was just a continue button but now it's like no I want to hit 30

days and and and having that be an intentional decision for them yes or no even though again this had no impact none of or no impact past this screen everything that I'm talking about now was uh you know just that screen that day and then was all thrown out so that optionality was was was a you know a huge insight and so because of that we built a a goal setting feature where you could choose between different goals giving users that optionality was was uh likewise a huge win I'll say one one final learning on

this um again you talk about like friction and and good and bad friction we thought once we built a you know a goal picker screen where you could pick between different streak lengths we we were like oh well let's like recommend that users do a hard you know a harder goal thinking that okay well harder goal is going to be better retention and and we'll pre-select the harder goal for them and this based on all the learnings that I just shared with you you probably imagine lost pretty significantly we realized that like yes we could

speed users through the screen more by virtue of picking a goal for them but like that act of selecting I think it's 30 days I think it's 14 days was where we were getting so much of the engagement from this feature there's so much human psychology that you all learn through all these experiments of just like how to motivate people what motivates what demotivates feel like you guys should write a book on human psychology and motivation I I feel very much like a amateur armchair psychologist with everything that at least as far as people who

want to learn languages on their phone go like I really understand those folks so one one kind of theme I'm hearing here so far is you guys are basically just like mining for gold just like looking for like a vein in some mine and once you hit it you're like oh this worked you just go crazy on just testing all kinds of things to see if you how far you can take that one little thread yeah and I think you know I I I shared that that the how this idea came partially from you know

monetization win that they had I think there's a lot of you know because duelingo runs you know I don't know what percentile we're in but it's got to be a very high percentile of you know per capita experiments run for a company you know based on company size we're just constantly learning so many things and there's a really great cross sharing of monetization say hey this thing worked for us is that something you can use so I'd say it's rare that we go into something where it's just like let's just let's just try something typically

we have some insight because of all these experiments that we've run that hey if we do this I don't know it's one here or it's worked here it's driven this user engagement if we you know sort of massage that make it you know try it in this um you know scenario or like a different screen you know we come in at least with a strong hypothesis that this will work a lot of times we do look and we're like you know a lot of the apps that we look at actually are games themselves so it's

like all right you're playing Royal match or you know mon there's a new Pokemon trading card game that I'm spend way too much time on you know you you you look at these games and see what they're doing and it's really good um fodder for what we can do but a lot of times you're at least going with as strong hypothesis based on what you've seen work elsewhere got it so just to double down on that point it's not like just random experiments it's here's a hypothesis where fairly confident or has a chance to be

true let's try it it's not just just try everything and I think that like how strongly you feel about the hypothesis directly ties to how hard that experiment is like with copy for instance we we've actually set up really good infrastructure for copy testing and I'm of the opinion that companies should run as long as you have the user base to do it like copy test constantly like the amount of copy test that we've had that is just like that have won and you I don't know you just you just try things and you figure

out what wins is is definitely Legion and if ship like massive WIS from Little copy changes is there an example of that of just like the impact going back to that goal screen um we used to say continue our like standard CTA is continue um and we changed that to commit to my goal and it was like a massive win even just you know and again it was like okay users tapping on that what are we asking them to do commit to the goal what is that going to lead them to do commit not you

know churn just that little copy change that one time right there led to to huge wins and copy changes are so cheap like it's just you translate you know for us you know we we have a lot of uh you know users all over the world and a lot of UI languages so but like just come up with a bunch of ideas translate some strings this is one where you know that the feedback that you'll typically get from Lis so all of our changes at dualingo go through product review that you know are reviewed by

Lis so Luis reviews every single change that you know we we propose or every experiment that we run to believe with copy he's just like I don't know test it like there's there's nothing better than to be told by Lis I don't think this is going to win but sure if you want to and a lot of times you know to his credit he's right and a lot of times you know our our intuition was right but it's just so cheap to to do it I think when the lift is smaller it's great to have

a hypothesis but you know you don't need to beat it up too much so on this thread I didn't realize Lis reviews everything you're planning to change and this may be the answer to the question I was going to ask which is one of the criticisms of running a product and Company this way of just experimenting constantly with all these micro improvements and changes is it can lead to something like a monstrosity of a product and experience that isn't consistent and cohesive and just like that often happens is the solution to that having the founder

basically review all the changes is there anything else yall do to avoid it moving in a becoming farm bill or whatever is a good example that yeah I our our product review structure where we've got our head of product design one of the lead um you know the the product management leaders and then Lis and PR and because they see everything it you know they have a really high product bar and so that helps I think over time we just as PMS have to look at okay where is our feature headed and so you know

we do this with the streak at least on a quarterly basis to look at okay well what what have we learned you know how is our streak developed and how do we imagine this going in your future I don't think you know it's it's easy to do you know and end in some sort of awful local Maxima if you're not constantly looking at your road map and think and you know you don't have a clear strategy you know for me it's like if you have a clear strategy for where your feature is going hopefully all

of those AB tests are not just done you know to get some cheap gain they're done with a long-term goal in mind I do think though and we do this every now and then with the streak eventually you just hit a local Maxima and you say like you talked about lunch and neutral experiments this is a great example where it's like all right cool now we need to throw a bunch of this stuff out based on all the learnings can we reset this real estate can we reset this UI reset this feature in such a

way that is just as good as what we have now but is way more plain or or simple that we can again start to layer on those are those are really hard experiments to to get win you know to win obviously because they are so optimized but they're really important to do otherwise yeah you just end up with a a kitchen sink of a feature one other uh tidbit I just want to mention something that the ADV in Advantage you all have that other companies don't have is people want to learn like they want to

learn the language and so getting pushed to come back to an app for something that they want to do is not an advantage a lot of products have so anything you want to add there of just like this is why duelingo might be a little different from what you're working on I mean that that is definitely a benefit if I had an app that this is this is actually why I think a lot of mobile games do streaks differently because you know to say that you played a mobile game and this as somebody who plays

a lot of mobile games uh you know that you've done it for 3,000 days in a row like I don't know maybe that hits a little bit different than you've learned Spanish for 3,000 days in a row I think the comparison set is much larger though than a lot of companies give them credit for you know themselves credit for and I think that there's mean there's a lot of ways that companies think about their there are very few companies I imagine out there in the world saying oh we don't do some degree of good for

our users uh even if it's like a game it's like I don't know you're like making your given somebody uh uh a moment away from the craziness of their lives and so I do think though that is contingent on companies who are going to figure out if a streak works for them to figure out how can you frame the streak in such a way that a user does feel good about it and it's easier for you know a dualingo but you know I think there's creative ways to to phrase this for users you know the

other thing maybe just on that that I'll say is the streak works really well for dualingo because with language learning it's really hard to see day-to-day progress and becoming more fluent like BEC and influency is is is is not even the right word it's like becoming better at Spanish or whatever it is a yearslong process for someone to get better at a language I mean dualingo makes it easier but you still got to put in thousands of hours if you're going to reach you know C1 or C2 fluency and and that is really hard to

track on a day-to-day and so the streak works really well for us because we not might not be able to tell you hey you now know 0.0 you know 1% more Spanish but we can show you hey like you've gotten your streak a little bit higher and so I think this works particularly well when you're an app that is doing something that's going to be sensed or felt over a longer term to help contextualize that progress in a way that makes more sense or at least feels more tangible to a user great that was a

great context empowering to a lot of companies that aren't necessarily doing language learning okay so I took us on a long tangent away from lessons and experiments you ran along the Journey of iterating on the streak so a few things you've shared so far is just like it started being like started with this XP idea and then went to like a lesson then you iterated on ways uh to make it simpler maybe harder you addit Street goals where you commit to like I'm going to hit a certain goal streak uh what else what else have

you found that has worked didn't work Lessons Learned you know again sticking with this 1 to S days streak you know it is the the idea of a streak particularly probably to this audience is like obvious like oh it's a streak it you know goes it just counts how many days we've realized over time that a lot of users do not understand how a streak works and and it can be as big as you know as small as well I don't understand how streak freezes work or I don't like my mom the other day was

talking to me about she's like my I like didn't use dualingo and I come back and my you know Streak still there so like there's like certain elements of the feature that you know we uh can do better explaining but even like what a streak is it's it's tracking how many days that I've used the app Yeah the more that we can make the feature easily comprehensible to users the more retentive it is and it and and we've run a number of experiments through this you know as simple you talked about you asked about early

easy copy changes that we made actually this is my first first win uh experiment win when I joined dualingo was when you start a streak we Ed we have little copy at the bottom of the screen that just I don't know it's kind of like Flavor copy we use it to celebrate you or give you context and I ran some tests that just like tried to in eight words explain what a streak was that was it and it was a massive win because it really required like it really dumbed down here is exactly how the

streak worked and it really helped users um just understand what they needed to do and I think this is something that's like you kind of constantly got to remind yourself particularly if you work in Tech and you're building Cool Tech features but your user base is not a bunch of tech workers uh to like think about all right who is my audience and for us it's not just like Tech work it's not people in America people all over the world of all ages of all cultures um and so making sure that your your your feature

is uh even something as simple as the streak is understandable as critical what was the actual copy do you remember it was it's still I think it's still on the app uh it's like start a day to extend your streak but miss a day and it resets something like that makes sense to me very clear yeah eight words I love that okay and then when you say massive win by the way just to give people a reference point what do you think what does that look like what is a massive win in this scale well

and it's and it's funny I mean this was and this was four years ago but I think it was like in the order of magnitude of over 10,000 daus for us and actually maybe this a small bit of context so duelingo really cares about the metric CER um current user retention rate and actually our first ever duelingo post uh with Lenny uh was the newsletter that our former head of product wrote Jorge still the single most popular newsletter post of all time in my newsletter across 300 I would highly recommend that if you are interested

in this give it a read um you know to to summarize basically what we found is that if we want to drive daus and do lingo cares you know our growth Northstar is da use the metric that is most uh uh effective you know where a percentage change in that metric is is most effective at driving daus is current user retention rate and this is just users who are not new or resurrected getting them to come back tomorrow and so most of the work that our teams do uh or retention-based teams do is focused on

cerr um and so the retention team that I lead focused on cerr just so happens the streak is the best feature at driving cerr and so this was this experiment was the biggest cwin that we had had I was like a top three Cerwin anyway for us just this little copy right and that's what I say like test copy a thousand different ways sometimes it's not the big beautiful feature you know that's going to drive the hu gain huge gains sometimes it's just some simple as a few words I love this uh okay so when

you you said 10,000 daus I think that references you guys measure incremental impact and absolute numbers of new daily users you're going to drive with attributed to that EXP experiment what it's yeah and we we we do both I mean we'll also look at like a lot of times I'll look at you know for retention day one versus day seven versus day 14 a lot of what I'm looking for is for us to have a better day 14 excuse me better day 14 impact than better day day one impact because it means that users are

retaining better over time this is particularly for users that would see a feature multiple times I just like absolute da numbers because as long as you're controlling for you know different bi like a recency bias or a novelty bias it's a really easy way to just have an absolute comparison you know you start to look at percentage changes and then it's influenced by who your treatment you know how many users saw the experiment and but at least you know uh an absolute number is is easier in my mind to start compar again there are down

you know there are pitfalls with it but um we find that that's a pretty useful way that uh lesson comes up a lot on this podcast in that approach to experiment so yeah you're in good you're in good company quick tangent if there's not an answer to this no problem coming back to the idea of just like experimenting like crazy and not creating a product that nobody wants to use anymore long term is there an example of an experiment that was positive that you all decided no we don't actually think this is what we want

in the product they ended up not shipping you know retention doesn't only work on the streak although you know you would you would think we most of our work is a streak we've touched a lot of different surfaces over the years and there was one experiment that we launched that you know we talked about XP earlier in the lesson the only UI elements are a progress bar at the top and then how many hearts you have right so we keep it really simple this very much speaks to the design you know philosophy of dualingo which

is simpler UI is better and we decided hey let's add XP in there and so let's show your XP ticking up as you're going you know through a lesson that's going to make the user feel good it's going to give you know show you what you've earned you're going to be less likely to quit all of these good reasons to do it and then you finish the lesson and we'll show you've collected all this XP and it and it won I mean the hypothesis was a good one um but we realized and I remember having

this conversation with Louise is like cool this is our most important screen in the app it is our lesson it is where users learn and the focus here is on learning and now you've added this other thing up there to to distract you know that could be distracting for users and I think the question you know we talked about road maps and strategy here the question that he had for me and I didn't have a good answer for at the time was like so what else are you going to do with with us like what's

your iteration ideas where is this going to go is this going to make the You Know lesson experience more gamified or and and and and what we realized is that like honestly it was kind of just a you know an easy engagement win idea but we had touched our most uh you know Sacred Space in the app to to do that and so that was a case where it's like yeah it was a nice win but we had added that UI element and you know at least at the time it was less clear what we

would do within we realized that like longterm it was just going to get in the way and we'd rather for Simplicity sake pause that you know shut it down and keep the lesson you know to be the a little bit of learning sh Sanctuary it was now it's funny nowadays I think we actually have enough XP based mechanics and fun things that we can do that like I think actually a lot of our um you know the the the beliefs about the inuson experience have changed such something like that could work but at the time

you know didn't feel good to keep that around that is an awesome example uh hopefully we have time to talk about how the team operates where my mind goes as like oh but you have all these PMs and teams that want to show impact and the performance reviews and all that stuff and you're shipping you're not shipping something they're like oh look we did a win so I want to chat about that later but let's keep going on things you've learned and things that didn't work along the journey the the other thing that that I'll

call out with the the the streak is you know like we have the the the image of the streak is this flame right and you know we have the streak flame and and it's it's very much quar iconography it's important to acknowledge that that's a metaphor for a retention mechanic like the idea of keeping a flame lit again if you've I think we've we've established the flame as you know for a lot of users as sort of their understanding of a streak which is great but there's a lot of people you know in different cultural

context in different you know uh uh stages of life where the idea of like keeping a flame lit to show your commitment to something makes less sense um we did some uxr in India many years ago and this was something that just like did not resonate at all um there which was which was a really interesting learning and that's something it's like again depending upon what your user base is like the more Global you XR you can do to understand how users are actually understanding and experience your feature the better because you just again encounter

insights like this and so even our screen design you know we used to have a flame it was just it was mostly this flame that would like light up every day but again it was like an indication of a metaphor for a mechanic and when we redesigned it we did this Kurt our one of our animators did this awesome odometer animation where it's like your number would tick every day it like kind of like looked good but from a product perspective it was cool is we actually focused the design on the screen to to show

your number going up and then it would say you know like seven day streak eight day streak and I think that as you're thinking about designing around a streak don't get too bought up you know or or caught up into like what is this you know like the beautiful story that you're trying to tell at the expense of it being a really comprehensible feature and so as you're thinking about product design making that product design a clear distillation of this is what we're actually tracking you know form should follow function here you know was a

was was a learning for us and you'll see that now in a lot of places where we're showing streak we really leading with the number not necessarily the the flame that's a that's a theme that I'm hearing again and again is Clarity don't obsess with making it too like clever and don't ever think it just like Clarity has a big impact Clarity also doesn't have to come at the expense of delight and this is something where you know you hit a milestone and Duo gets you know becomes we call him Phoenix Duo and he becomes

awesome and you know lights on fire and I think there are things that you can do to still make the experience really exciting and delightful and celebratory and and I would I would not lose that but just don't do it at the expense and and I think it's also about figuring out for what you you know you can get away with doing more of this for users who are deeper into their streak experience than users who are starting where it's like your goal for the one day streak user is just to make sure do they

understand how this feature works I mean even something again just like another random experiment at the bottom of the streak screen we have a calendar and over the years it just looks more and more calendar like and that is simply because we find that the more we make it look like a calendar days on top you know little circles the check like the more we make it look like calendar the more that people realize hey this is a day a daily meic and so think about the screen holistically but every single thing that you're doing

on the screen how can you use it to communicate what is the point of this feature how does it work this episode is brought to you by Kota I use Koda every day to coordinate my podcasting and newsletter workflows from collecting questions for guests to storing all my research to managing my newsletter content calendar Kota is my goto app and has been for years Koda combines the best of documents spreadsheets and apps to help me get more done and Coda can help your team to stay aligned and ship Faster by managing your planning cycle in

just one location set and measure okrs with full visibility across teams and stakeholders map dependencies create progress visualizations and identify risk areas you can also access hundreds of pressure tested templates for everything from road map strategy to final decision-making Frameworks see for yourself why companies like door Dash figma and qualtrix run onoda take advantage of this special limited time offer just for startups head over to coda.io Lenny and sign up to get six free months of the team plan that's coda.io Lenny to sign up and get 6 months of the team plan coda.io Lenny I

imagine one of the biggest wins was just giving people flexibility flexibility along the journey like streak freezes and all these things is that is that a big vein of opportunity Discover it it is it is actually I'm going to show you the one of the most thoughtful gifts anybody has ever given to me this is our dualingo Serenity or streak Serenity Prayer my my co-lead Antonia it's like knitted right it's like for me it's amazing and so it says Lis grant me the serenity to accept the flexibility I need the courage to reach Perfection when

I can and the wisdom to celebrate regardless and that actually is like kind of our strategy with the streak so love the show and tell by the way that was great yeah I will we I guess for you know podcast listeners we'll have to you know get an image somewhere um but the uh this idea of like flexibility versus Perfection and then regardless celebration is core to how we think about the streak because you know I think for the streak for us it's very much a Ben not break you know I if you're gonna miss

a day I'd rather you come back you know having not you know having missed that day to an intact streak but if you don't have to miss a day I'd much rather you don't I'd much rather you come back uh and and use the app every day so that that that thing on flexibility though that's almost certainly been the biggest you know from a mechanic perspective the biggest da driver you know one of the earliest experiments we ran was going from you used to only be able to have one streak freeze and then we let

you have either two or three so we tested two different arms um it was an again another huge Dau win this actually is funny it was something that this is again kind of a a call back to that growth model post from from Jorge it actually was really bad for ker because we were basically saying hey you can take a day off you know and that's okay but it was really good or is this going to be like alphabet soup iar inactive weekly active user retention return rate so basically users who had taken a day

off we were getting them to come back more at higher rates and so it made up for our losses and curve but effectively what this meant is that you know why two streak freezes work better than one was I don't know sometimes people just need a little bit more flexibility than one day but one of the again the really interesting inside of this experiment was that three streak freezes was actually no better than two streak freezes and there were two competing things here and I think this is important if you're going to build a streak

you know to figure out what your flexibility mechanic is we are getting more users to return after longer times away to an inact you know to an intact streak but if you start taking three days off from any habit it's just going to be less likely that you return even four days later and so we had these competing things where users might more users might be returning to a streak but a lot of users were also just not coming back we're training them to you know take more time off so that flexibility what's the right

amount of I mean we've again this is another area we've run hundreds of experiments on what is the right number of flexibil what is the right amount of flexibility and we we are constantly surprised here we we don't I I still don't have the answer for at every point in your streak Journey how much flexibility you need one thing that I can say with certainty though is give more flexibility when a user is starting their streak again one of our biggest streak freeze experiment when I feel like I'm constantly saying this one of our biggest

wins but they all were were really really big one of our biggest streak freeze wins was when you start a new streak we'd give you two streak freezes and again it's so funny to think back it's like how are we not doing this to begin with but at the time the streak freeze was kind of an overly gamified mechanic you had to buy them with gems that's our inhab currency you know because we wanted the whole idea of you know this to feel like you like it was really you know something you earned that you

know there was a little bit of pain to getting that streak freeze and so we test Ed though what if we just give users when they start off their streak two streak streak freees and holy smokes to that win it's sort of obvious now you know in retrospect but if you have a one or a two or a three- day streak it's really easy just to let it die and restart you know again you need to get to seven days what we've seen in the data for it to really lock in and so giving users

more flexibility so it's harder to lose their streak initially then conversely and this is what we keep learning eventually once people get on Long streaks you kind of don't want to give them as much flexibility you know because there's a lot of times where yeah users don't and I'm I'm like this I've got like a 400 day streak note that that is a lot less time than I've been at duelingo I have lost and restarted streaks a lot of times at my time at dualingo um but you start getting a l streaks and you really

care about this feature you really care about your streak and most people as long as you're not like backpacking through I don't know you know the the back country of uh Utah you know you'll you'll be in a place where you can get service and so figuring out where's the you you know who is the user that actually could use dualingo and not conditioning them to start taking days off that they didn't otherwise need to do is um is important to figure out sort of where that line is for for your feuture this is fascinating

you can also buy a streak right like with money is that that's a feature right yeah you can it's funny this is this is also something that we or freeze sorry not yeah so you can you can buy a streak freeze and the way it works is you can buy gems and then you can use those gems um to to buy a streak and this is something we wrestle with all we're actively working on an experiment right now that's having a small hit uh Revenue but it's a really nice one for retention and I think

it's actually worth thinking about from day one as you're building a streak do you see this more as a monetization feature or do you see this more as a retention feature what's the role of monetization in this what's the role of retention I think for us it's started out much more organically and so we have a lot of like monetization hooks that again is the retention PM I would love to get rid of um but again it's sort of part of how the streak works right now and so we always have this tension of hey

if we start to make it harder to buy streak freezes then fewer people buy them you know buy gems to buy them and so there's this more convoluted you know series of impacts that happen um yeah no I love that I love that uh people wanting to buy streak freeze is like the ultimate sign of how much streaks matter yeah it's it's stre streak freeze is the other big one that we've that we've recently demonetized or or introduced a free option for is getting back a um a lost streak so us losing streak we had

a feature um in the day you know back in the day called streak repair we' give you your streak back he had to pay gems but we found out worked way better was a feature called earn back and this is basically where you would have to do a certain number of lessons as long you came back within you know a a window soon after losing your streak do a few lessons and we just give you your streak back and it was such a retention winner and again what we thought about was it feels like you've

earned it so much more when you've done like you deserve to have your streak back we haven't cheapened the streak uh because you've done something and in this sense this idea of cheapening the streak is something like from a philosophical philosophy of the streak from a philosophical level we wrestle with all the time of cool we're giving out more streak freezes at what point do we cross the line and users start to realize their streak means nothing um now everything that we've seen users are totally cool with using streak freezes and still thinking about their

streak as this meaningful thing but my colad Antonio who made that awesome cross stitch uh for me she is sort of the keeper for us of the sanctity of the streak and a lot of times as we and I think this is this is really important to have as you're thinking about building your streak you can almost always get engagement wins up to a certain point by just cheapening the streak making it easier to extend letting users have more flexibility but you kind of got to hold the line at some point it's not clear where

that line is and once you like you talk about oneway doors or two-way doors there's a point where you go too far and it's a one-way door and all of a sudden those users those nine million users on onee streaks don't care about their streak anymore and that is I don't know again retention PM perspective that'd be an extinction level event for us I don't I don't want all of these users to stop caring about their streak and so to have somebody who has invested in the the sanctity of the streak and you know for

us it's Antonia and and Lis he's very good about this um is really important just so you make sure you don't go too far that's an awesome Insight uh so to protect and push notifications I think are another example this in general for companies like how much is too much because everyone's just like let's just send another push it's fine just do one more uh and I so your solution to that is uh a person is like the keeper and the gate almost the gatekeeper plus the founder of how far is too far that it's

yeah it's good if you can if you can have that um I think push notifications are also easier because you know there's a lot of things you can do around all right we'll set a budget cap for how many notifications will send you know you can you can like a create a policy yeah policies but I think with a lot of things it's it's hard at least for the streak it's harder to create policies for in the same way a lot of it has to be done based on feel and so you kind of just

got to use your best Judgment at times sweet okay any other maybe like one or two more lessons from this journey of what streaks has become today I mean I can you mentioned notifications I've mentioned this a few times one it's funny you you you tend to think of like exactly as you say ah you can just always send another notification it's going to be some win and at some point you know it'll it'll be a bad experience but it's tough to see that there's actually a notification that we so we send two notifications related

to your core streak each day the first is a practice reminder we send it this is actually an interesting Insight 23 and a half hours after you practice the day before whoa 23 and a half okay so basically if you practice at noon today we'll send it to you at 11:30 am tomorrow and we have done because it's like assuming they were free in that time the day before maybe they'll be free the same time and we actually moved we used to let users set this practice reminder time and you know our thinking was cool

you're going to say 7 PM that's when I really want to extend my streak each day and then you know what I say this is somebody with two kids life gets in the way life always gets in the way and when you think you're going to practice will change your life will change whatever and what we realized is the best indicator of when you should practice was when did you practice the day before we could almost certainly get more detailed we have tried a bunch of ways to have much more complex logic and what always

wins is 23 and a half hours that's so interesting revealed Behavior versus yeah exactly um so so we again we send this practice to miter 23 and a half hours later the other thing that we'll do though is we'll send it what we call a streak saver and this is at 10 o'clock at night if you have not extended your streak we'll send you a message saying hey you know like it's it's your last chance this is this is it if you don't extend your streak and you would think that like that's kind of spammy

that's that's kind of annoying to get a notification from an app at at 10 pm but what we found is because people care about their streak their streak is this good thing that they attach positive emotions to that they don't really want to lose that notification reminding them hey come back and like people see this uh uh by and large as a positive notification and not a negative notification obviously it serves our purpose as well of getting users to come back and cannot lose their streaks but again I think if you can think about your

notification strategy related to what is the feature that it's tied to how do users perceive that you can almost certainly get a not get away with more but you know you could be thoughtful about notification load um and when to send notifications and again for us this like late night message that's also highly impactful super good you know is actually something that could be perceived as spammy but a lot of our users really do I somebody who it's often at night I'm and I work here and I'm like H forgot to do my you know

I was thinking about dualingo all day you know here it is 11:15 and I saw it done it that message is um really powerful yeah it has saved me many times I totally know that message and uh I love that it's like a late night message uh from an app very rarely do you actually are happy about that and I love that this actually is a good example where like it's it's really it's really funny all of the stories that you hear about people extending their streak you know if you look around the dueling party

uh you know where it's like 10 11:30 11:45 all the dualingo employees that are like doing their lesson at the last minute uh you always see these like pictures of people in the club doing or like at a concert doing doal lingo and yeah because it's like otherwise that you're GNA use a streak freezer God forbid you will lose your streak so funny okay anything else and if there's like more than one more definitely share but any other really interesting lessons or wrong turns or insights you know I talked about streak freezes and we've done

a lot with streak freezes but I think I think if you're going to make flexibility a thing it's probably also useful thinking about how do you celebrate Perfection um and so we have a feature that we have it is the simplest thing in the world it's called perfect streak and it's just if you don't use a streak freeze for a few days we make your streak look gold and we make your little progress bar on the calendar um just look a little bit nicer there's no reward for doing it you don't get anything other than

this nice little indication and it is awesome it is it is it is is is a simple feature it is old not complex and it is really powerful not only for getting users to you know as as a bit of a reward they' be hey you know get to seven days you know without using a streak freeze and you know your streak becomes perfect but it's also a really nice indication of users who aren't using streak freezes here is the thing that if you uh you know if you don't use a streak freeze which again

candl I would love for you never to use a streak freeze if you don't use a streak freeze your streak will stay you know perfect it's funny we actually just you know we're constantly you know responding to bug reports uh about the streak you know it is I swear to God we we have that we have the best infrastructure around this feature because it is so important um we had an employee who lost her like a four-month perfect streak uh and you know it was a big deal for her because she you know and because

she like did her lesson as she was crossing International dat line there was like a bunch of stuff going on that was like it was just kind of weird in our back end but like people start to care about perfection as as much as they do their streak and you know for that person it was it was it was a big deal when they lost their perfect streak and so you know this is just an example of look if you're gonna if you're going to go after flexibility um which is good finding a way to

pull users back into Perfection is a really important counterweight to have what I'm imagining is you guys need like a Amazon style chatbot that just gives you the streak back just like okay here you go we have very much so we we have if people lose their streak you know there there there's ways to get in contact with us but we've actually about that where it's like okay we should just like build a self-service feature and if you're if we think that your excuse is good enough whatever we'll just yeah yeah because again it's us

it back i' much rather you be on you know be on a streak than you know have lost it particularly if it but it also can't feel that easy uh I love this I also love this point about just the power of the animation and user experience having impact that's really interesting is that something you find often just like make like celebrating and making it feel really amazing without like copy or you know or like a feature it's just like holy you're awesome this is another thing where it's like when users care about the feature

using not only uh uh animation haptics sound effects using it's funny we don't have sound effects on the streak this will probably something we'll look at in the not too distant future but like haptics are like something we have done a lot of testing on the phone vibrating in exactly your phone you know like there being a really cool haptic pattern as you extend your streak um all of this stuff wins and it's cool because I think it wins there's a few reasons one is it just makes you feel good right you know you you

get some cool moment in your streak and and and we celebrate you and we celebrate you in this visual way and your phone's buzzing it just like feels awesome the other thing it does is it like causes you to pause on that screen and I think there's this desire as you think through a lot of you know as PMS think through like oh how can I get users through this funnel as painlessly as possible you know I talked about Good Fortune earlier there's a lot of times where I don't I want you to stop I

want you to stop and like land on the screen you got to be careful not to do this for too many screens right but like for the one big ones sometimes I just want you to pause there and enjoy the moment um because if I can get you to enjoy the moment more you're going to care more about your streak and you're going to be coming back tomorrow and so animations that are cool and that cause you to like really soak it in haptics you know that feel good all of that comes together to make

you really focus on that moment um all of that just gets users more connected to their streak so animation in the right times works well and it's something we've had win quite a lot who designs the haptic stuff is there like a haptic designer for the longest time it would be a product designer or it initially started as like the engineer would be like all right you know we'd Cobble together haptics um based on what they felt good then it became a product design um role where they would kind of use their best judgment we

actually just recently required an animation studio uh Hobs in in Detroit and now they are the sort of keepers of you know they're do a lot of motion design work haptics very close to that and so they do a lot of that I do remember trying to hire for a while a haptics like contractor like haptics design and it was the saddest hiring I've ever done because it was just like I don't know like it was such a specific like I know I just went through a lot of people who uh you know it's it's

just a really tricky space of like kind of sound effects kind of motion design sort of technical yeah um so such a unique role very unque specific skill set right and there's very few apps that really need this this deeply so you're almost creating this person yeah that is fascinating uh I don't want I'm gonna that's like a whole podcast on its own by the way I was going to say as you're talking about this I love that uh it's a win to celebrate people that don't lose their streak like you introduce this way to

make it flexible and that's a big win and then you could go the opposite direction of if you don't use this feature you also feel even better well and it's funny you like you talked about the the danger of feature blow we sort of talked about the danger feature below this is actually something I'm constantly thinking about with this like we have the streak but then we also have the perfect streak and we can count how many we count how many weeks you've had a perfect streak all of a sudden we have two streak numbers

that are kind of competing with each other it's funny we actually don't introduce the concept of a perfect streak until after you've hit seven days and some of this is just because the cognitive load of additional streak features a lot of our like cooler streak features you got to get on a long enough streak and not to say we have haven't tested it because we have because we test everything introducing these features earlier but what we've found is that pretty universally they lose when we introduce too many things too many Concepts to users too early

in the experience it's just hard for them to manage okay sweet I know that we can go down this track for uh hours and hours there's endless learnings about all the things you all have done along the journey I want to shift to talking about how your team operates so there's a lot of uh threads you touched on of just how a team can do this so well ship 600 experiments as you said uh continue to find opportunity what are some maybe lessons or advice you'd have for folks that are like oh wow I want

to work more like this from your team's experience how to use your team operate that folks can learn from yeah maybe just a little bit of context we're dualingo cares a lot about metric so most of our teams are metric-based teams so we do the most work with streak um but you know the metric what we really about at the end of the day is cerr and daus because we see that daus hit cerr and so when you can be really laser focused on my goal each quarter is to make this metric go up I

think it's much easier to make sure that you're working in the highest Roi thing I think when you think more about like oh I want to make this feature better I think it's easier to get lost in what better means and how you think about better and so I do think that like having a really strong degree of focus you know as as a team on you know what is the metric that I'm caring about and how is that directing my efforts is versus feature Orient so basically your teams are structured around a metric SL

a goal outcome versus we own this feature or this we retention owns streak I guess but that's only because we've seen streak Drive Cur better than any feature but we are not I mean we have this you know I uh hook with our streak freeze purchases there are other teams that work on you know that can and have worked on the streak because it's not ours to say no no no we do all the iterations here we just know that it drives our metric better and the same way that like leaderboards uh you know we

have a team that focuses on how much time you spent we want users to spend more time on dualingo so they're learning more leaderboards the best Vector for doing it so that team does a lot of leaderboards work but every now and then I have an idea that I think will be highly retentive and I will go in and I will pitch to them and then we'll do some change the leaderboard to make it more retentive um and so but I but I do think having that clear metric of we're trying to drive Kerr not

we're trying to just make this feature better helps at least make sure you know give the team clear marching orders and that Focus I think is really good for prioritizing backlog cool this is a really important point this is the same way Airbnb worked when I worked there for a long time is it's here's a a goal that we want your team to be responsible for you can work on any product you need to hit this goal as you said often like various products are most connected to what you're doing but what you're describing is

like even though A team's kind of like I imagine you own it from a bug perspective and you're kind of like are The Shepherd of this part of the feature because it hits your goal helps your girl most but any other team can come in and be like hey Jackson we need to work on some streak stuff to help with learning you're like go for it does that just a tangent there uh do they like work really closely with your team if they want to do some work in the code is it like how does

that work logis yeah if if if you're again this is where I say there's soft ownership we're not against teams doing things to the streak but if we're going to do something given you know we probably have multiple quarters worth of a road map around the street I say probably we do uh multiple quarters of road map uh for what we can do to the streak if other teams want to come and mess with it okay we got to just figure out how is that going to work with you know what what our plans for

the streak were how do we make sure a lot of times when teams are coming in thinking hey let's do this to the streak they're like in context that we might have and so there's as much of like a much simpler version of what we're doing now a bit of a knowledge sharing of saying all right well this is what we think about the streak this is what we've seen work hasn't worked how does that influence some of the hypothesis that you have and so I think getting that really um making sure the juice is

worth the squeeze good old fashion product management work right there uh cool what else is interesting about how you all operate and how youall work to achieve this sort of success again my my team lead runs is Antonio runs like the most you know a really process if you're going to run this many experience you have to be really process oriented and really thoughtful about which experiments am I going to run when how is that going to set up the next one we uh you know uh there's there's heavy Jura automation I think sometimes the

atasan suite makes my eyes bleed but like there's a lot of times where that degree of process helps the team unblock engineers and make them move really fast and so making sure that you have really good process around how are you going to run so many experiments it's you know it's it's it's worth investing in can you follow that threat actually just what do you when you say that what does that look like what are some elements of that process to make this work efficiently all the way down to like know really detailed road maps

around all right we're running this experiment is based on the results of this experiment or my you know uh hook into an element of this feature how do we make sure that we're lining up implementation on this so that as soon as this thing runs and we're ready to go we can start rolling out the next one you know I hate features just sitting around and us not again continuing that thread so it's not just thinking about what's our engineering bandwidth but also what's the design bandwidth to make sure that we have the next iteration

of the feature ready to go you know we're planning months out as we think about these feature iterations even small ones um you know feature iterations because again you just you lose when you lose Cycles you know not pushing on a feature it's just sort of lost opportunity and so everything from manage you know being thoughtful about engineering road maps to design road maps to product road maps um all of that needs to come together in a system so essentially mapping dependencies across function and you're saying in jira you can do this you can do

a lot of it in jira there's there is a nonzero amount of Google docs that we have that sometimes does things yep a little bit I don't know sometimes it just looks a little bit nicer it's a little more flexible um but jira is our it is where the mother load of process is great okay what else like another thing that I'll that I'll just say is like we really resist the urge to do the big V1 um and I think this is you know I shared the the the street goal example where a lot

of depend all when we're exploring something we will say okay well like that's cool how do we strip away a bunch of stuff and figure out what our core hypothesis is and then just ship that thing first as a V1 because it's it's easy and I've found this time and time again it's easy to add things to features that make them win like I've worked in retention engagement long enough I can like add I know enough like things to pull and bells to add and whistles and you know to make something win but you know

there's a lot of times where it's like cool then it w because of all the whistles you added not because of what your core hypothesis was and a lot of times if you can just really simplify what the feature is it's also much easier to ship it's easier to design you're not designing for a whole system you're designing for you know something something much simpler um and so getting everybody to think that way ends up allow you know allows us to end up shipping faster shipping simpler you know designing faster getting faster approval getting insight

and then doing what what I talked about with Street goal being able to run iteration after iteration after iteration add these things iteratively um and then know you you you not only by doing this are you able to move faster but you get confidence at each step of the way that hey my series of hypotheses you know is actually born out or if it's not cool well then we're going to drop that part of the feature and then just ship what actually matters if I can try to summarize kind of uh the broad lesson so

far that I'm hearing and maybe you you would have shared this but I'm just thinking like if I were to try to design a company to operate the way you all operate you essentially map all the levers that drive the business so you have kind of this mapping of all the metrics that drive up to growth and and daily active users you Curr ended up being the biggest specific metric to move to drive growth long term so there's like imagine a tree of all the opportunities you could work on you found like this is what

is most uh connected to or growth long term you basically just start mining uh I don't know if mining is the right metaphor but just like looking for things that move that specific metric you just like look and poke and explore and then once you find one you just go real deep on trying a lot of different you I you could come up with a hypothesis and a strategy of here is how we think we can do this and how we can move this and then you just try a bunch of stuff there's also this

element of uh there the rested development quote there's always money in the banana stand comes to mind where it's just like keep working on see there's more there's going to be more opportunity at this when I joined dualingo um the PM that I took over for Anton who used to lead the retention team I remember saying dude the streak like it just counts up you guys have been testing on for years how much more work can we do on the streak and he was like Jackson you child he didn't say exactly this but this is

how I felt it like Jackson you child there is we're not even 30% of the way optimized and four years later I say that with like such conviction we are so far away from what I mean we've made a ton of strides but we are still so far away and every quarter where we ship a ton of you know winds and you know improvements to the streak it just continues to prove to me that like there is so much more to be done so I I think your framing of it is and I would say

there's a lot of thought that goes into you know again you know I talk about the strength the hypothesis that you have you know you have to have as you start to build that larger you know future strategy I do think it's really important not just do a bunch of random stuff but do it with intent you know with a goal in mind otherwise you do end up in these local you know local Maximas but yeah I mean there's still a bunch of stuff that we haven't tried that you know I think we have high

confidence and you know working out and so we'll we'll keep doing that are there any other say lasting lessons from this journey that if someone were to try to operate this way build streaks into their product anything you'd recommend yeah I really do think it starts with you know it streaks are an engagement hack uh you know you can you can make your app more retentive I'm almost positive almost every app out there can make it more retentive it is loss aversion that is you know again armchair psychologist Jackson like it's just a human a

thing that works on humans um but if if you're if your app is not something that users want to use every day or whatever you know Cadence you want your app to be uh you know to work on it's going to be you're only going to get so much from that streak and honestly it's probably going to distract you from what really should matter which is making your app something that people want to use every day and so if you start focusing on the streak but you haven't made that a enjoyable experiment experience you're just

going to waste a lot of time honestly and so I think making sure that you have your core Loop of your app figured out that that it is giving value to users is something that they want to come back to every day that really sets the stage for something to later uh a streak on so resist the temptation if you haven't you know if you don't think you've reached that point to go too hard down the you know the path of streak that's a really good point just like a streak is not going to solve

your problems if people don't actually care about the core value you're providing no and honestly it'll probably cause more problems if if what you end up focusing on is how do I make the street highly engaging but your app isn't I mean you're just you're you're you're wasting time that could otherwise be you know better spent on solving more more critical problems so that's that's one learning you know the other thing that I'll say is we we met with one of our board members um bing bing Gordon a few weeks or uh a few months

ago rather and he had this comment where he was just like the reason why users care about your streak so much is because you care about your streaks much you being dualingo like the reason why users care about our streak so much because dualingo cares about the streak so much we're like what do you mean well he's like well after every session you see a big streak screen and it's animated cooler than almost any other screen in the app and then sometimes you see some other screens and there's all these other Fe like you don't

let a user forget it you talk about them in messages and and so I think it's worth thinking about look if you're going to build a streak and then you're gonna scond it off into the corner of your app uh you know where users aren't going to see it like they're probably not going to care about it as much and which might be fine because there might be other lovers that you think are more important to pull on but there's a reason why you know we focus as much on the streak you know as as

we do and that's because we want it to be top of mine for for us users and it's not by accident then that users start to care about and so I think just as you're thinking about building the streak making sure that you're giving it the visibility it deserves if you want it to have the kind of impact that duelingo has um you know is it's it's sort of a important hierarchy principle to to think about as you design things that's such a good point like you look for cues to the app of tell me

what I should pay attention to what's important if you're just like fire explosions you made a streak oh maybe I should pay attention to this feature and then the push notifications off L encourage you there too uh anything else along those lines maybe final thing is look we ran so many tests on our dualingo streak to figure out what worked I mean we have we have a we have a philosophy of dualingo of tested first you know we are a lot of times willing to test things I really think that if you're gonna if you're

going to try to introduce a streak or you want to improve on the streak you have like don't get too caught up in the the the philosophy of of everything I mean make your hypotheses feel like they're good but my recommendation is just try things and this is again you said it earlier it's like it's as much human psychology as anything and as soon as that becomes the case like you kind of just got to understand what users respond to and the easiest way to do that is to stop spending time you know batting around

ideas in a in a conference room and just try some stuff um so huge recommendation to like if you're going to invest in a streak try and figure out what works through testing with users um rather than trying to get it perfect on the first try say someone's listening and they're like should we do streak is this worth doing what's your take on just like the chances that a streak feature would be helpful to another consumer I'm I am uh well known for saying the company that I think every streak every team every app could

could benefit from a streak now how you implement it is very different and I think you gota like what is your users use case like if they're going to come use I don't know tax software okay you know you know not that I say this tax software would be a hard one but maybe it's all about you know you know you need users to come back every day during the tax season uh or how many times I don't know you know now that I say this out loud times you upload your your yeah that is

a hard use case but I the the vast majority of of of of companies I think have a good idea of like all right here is my ideal use case I want users to come here three times a month that that would be ideal you know or four times a month you can build a streak to work I mean pelaton has has weekly stre because the idea of doing a pelaton you know workout every single day is hard for a lot of was hard for this user during Co you know it's just like every now

and then you get on the Pelton that was great but the idea of a weekly streak was something that I could keep up and so I think figure out what your usage pattern is as a user and then build your streak around it but you know as long as you're not you know the like a really again the the tax example was probably a good counterfactual but as long as you have some degree of frequency in your in your use um I think almost anything can have a can have a streak so duelingo it's again

A1 14 billion company do feature possibly the most contributing factor other than the core you know product to that level of success and market cap and it's hard to imagine another just feature of a product that has had this much impact on on growth and revenue and building this sort of business so I love that we spent This Much Time on it the mother load the motherload of advice and insights so thank you again for putting very fun with that we reached our very exciting lightning round are you ready I'm I'm I'm ready first question

what are two or three books you've recommended most to other people all right I'll start with uh a guide to Midwest conversation so I'm based in Kansas City I'm a proud midwesterner and us midwesterners talk in a certain way I think you know you hear about Minnesota nice um but we tend not to say what we mean and it is a very funny primer into What Men westerners actually mean when they say what they say um so highly recommend reading that I like that that you give that to people just like here's what I might

be telling you which you may not read but my my wife is German and I gave it to Herman being the opposite of that okay very different um another book uh this is a good one uh fate is the hunters this is a really cool book it it's a memoir of one of the early commercial airline pilots and it is wild to hear the stories about what flying was back in the day you know I'm former Management Consultant I flew every week for uh almost six years and I never once had to worry about am

I going to make it to the other end of this flight alive that was not the case back then and so some of the stories about what it used to be like to to be a pilot on some of these planes um before modern aviation technology is fascinating it makes you really appreciate what we have it's uh feels good to read a book like that being a software or PM or engineer whatever like how different that life is hard Hardware is hard oh man it's not htic design okay next uh unless there's any other books

you're gonna share no okay great what's uh favorite recent movie or TV show you've really enjoyed uh so I have two kids I watch a lot of Bluey um it's really good I swear it's it it brings me no shortage of Joy um but adult show that I or show not meant for four-year-olds that I have watched I just finished the latest season of em in Paris man wonderful uh I realize it's not the highest brow of Television but just like beautiful people and beautiful cities solving problems that are not you know Earth shattering sometimes

it is nice to just tune out also I'm learning French on duelingo uh slight plug for the app I can understand a lot of the French that is being spoken and there is no better Joy than having invested as much time as I have in French and actually being able to use it um so huge fan of Emily that is so funny what a fun vend of interests uh my mother-in-law loves Emily in Paris uh I saw someone tweeting about like what what visa is she on how is she still in Paris yeah yeah you

best just not ask questions there's aot there's a lot of questions to the show that are better left un do you have a favorite product you recently discovered that you really like other than do last week I went to Home Depot and I bought a new ladder and Ladder Innovations you don't think you know uh uh of often but you can make one of the legs go a little bit further than than the other leg and as somebody like myself who has a house that is built on a slight slope uh every time I go

up on my ladder I take my life in my hands but with this ladder I'm always even I cleaned my gutters twice last week just because of how awesome this ladder has has how how much this ladder has changed my life so ladder Innovation I don't think it gets talked enough about and so I'm happy to give it the spotlight it deserves I appreciate you doing that this the first L recommendation we' had on the podcast two more questions do you have a favorite life motto that you really find useful worker in life they share

with folks this probably will not be much of a surprise based on how I've talked about our willingness to test things but you miss 100% of the shots that you don't take um I'm a big fan of just trying things even if your uh possibility of success is not 100 uh because you learn a lot along along the way so final question do you have any fun Traditions at dualingo amongst either the PM team or the company in general that might be delightful to share we have way too many Traditions to count um I will

share the weird tradition that we do at every retention standup and this started during the pandemic you know we obviously used to stand up in person and then um when we went remote we did this thing where whoever's last person to go would count down three two one and then we'd all try to clap at the same time which was kind of fun and dorky but we fell in love with it and four years later we're still doing it recently we've added we all say Yeehaw in unison afterwards I Can't Tell You Why um but

trying to synchronize a clap via zoom and then all shouting Yeehaw I did this in a phone booth the other day and after I came out someone told me you know that those aren't as uh soundproof as you think but you know when you get a good opportunity to give a ye-haw you kind of can't pass up on it so I love these little things like they're so you know they sound so minor but they're such important elements of Team culture and tradition and so for PMS to find ways to just have fun and do

something ridiculous I will say it took a while to get people behind shouting ye-haw um but now that we have people doing it you can't you can't take it away from we all we all love it oh man I uh I called our all hands for a while y'all hands feel free to steal that you get it you get it Lenny yeah I get it Jackson this was incredible I feel like people are going to listen to this with notebooks and just like okay here here's a bunch of ideas we should try with whatever we're

working on whether it's streaks or not thank you so much for being here two final questions work can folks find you online if they want to reach out learn more learn more about dingo I know you're hiring product managers so share more there and finally how can listeners be useful to you yes uh you can find me on on on LinkedIn that is where most of my online social media so jackon shuttle worth um and then how people can be useful to me yes as you said we are hiring we're actually hiring for my team

are you interested in thinking about streaks as much as we do uh you know we might be the right home for you so please uh you know you can apply on our website we're also hiring from a number of other product management roles and they are all as as as thrilling as this work is and then you know I'm always interested about how other companies have implemented streaks in what they've learned and so what I'd say is if you're a company who's implemented a streak uh maybe in a different way than dualingo has or you

know you've found a whole ton of success and you know another Vector another element of the feature that we didn't talk about today I would love to know more I used to catalog basically every streak I found out there um and as it's become more of a a popular feature it's just been hard to keep up on so you know if you have interesting streak insights to to to send my way I would love to hear them I love that the collection of all the best ways of doing streaks uh Jackson I just I want

to say congrats to your team and and you for having so much impact this is like the dream of a lot of PMs and teams is to see this much impact and continue to ship wins and so congrats nice work attention and with that Jackson thanks so much for being here yeah thank you Lenny this was uh this was a lot of fun same bye everyone thank you so much for listening if you found this valuable you can subscribe to the show on Apple podcast Spotify or your favorite podcast app also please consider giving us

a rating or leaving a review as that really helps other listeners find the podcast you can find all past episodes or learn more about the show at lenp podcast.com see you in the next episode