Spotify Engineering Culture - Part 2 (aka the "Spotify Model")

[Music] okay quick recap from part one our culture is based on agile principles all engineering happens in squads and we try to keep them loosely coupled and tightly aligned we like cross-pollination and have an internal open source model for code squads do small and frequent releases which is enabled by decoupling our self-service model minimizes the need for handoffs and we use release trains and feature toggles to get stuff into production early and often and since culture is all about the people we focus on motivation community and trust rather than structure and control that was part

one and now I'd like to talk about failure our founder Daniel put it nicely we aim to make mistakes faster than anyone else the idea is to build something really cool we will inevitably make some mistakes along the way but each failure is also a learning so if we fail fast we learn fast and therefore improve fast it's a strategy for long-term success it's like with kids you can keep a toddler in the crib and she'll be safe but she won't learn much and won't be very happy if you still let her run around and

explore the world she'll fail and fall sometimes but she'll be happier and develop faster and the wounds well they usually heal so Spotify is a failed friendly environment we focus more on failure recovery than failure avoidance our internal blog has articles like celebrate failure and story is like how we shot ourselves in the foot some squads even have a fail wall where people show off their failures and learnings failing without learning is well just failing so when something goes wrong we follow up with a post mortem this is never about whose fault was it it's

about what happened what did we learn what will we change post mortems are actually part of our incident management workflow so an incident ticket isn't closed when the problem is solved it's closed when we've captured the learnings to avoid the same problem in the future fix the process not just the product in addition all squads do retrospectives every few weeks to talk about what's working well and what to improve next all-in-all Spotify has a strong culture of continuous improvement driven from below and supported from above failure must be non-lethal or don't live to fail again

so we promote the concept of limited blast radius the architecture is quite decoupled so if a squad makes a mistake it will usually only impact a small part of the system and not bring everything down and since the squad has enter and responsibility for their stuff without handoffs like you usually fix the problem fast also most new features are rolled out gradually starting with just a tiny percent of all users and closely monitored once the feature proves to be stable we gradually roll on out to the rest of the world so if something goes wrong

it normally only effects a small part of the system for a small number of users for a short period of time this limited blast radius gives squads courage to do lots of small experiments and learn really fast instead of wasting time trying to predict and control all the risk in advanced mario andretti puts it nicely if everything is under control you're going to slow alright let's talk about product development our product development approach is based on Lean Startup principles and is summarized by the mantra thinking bill it should be tweak it the biggest risk is

always building the wrong thing so before building a new product a major feature we try to define a narrative kind of like a press release or elevator pitch showing off the benefits for example radio you can save or follow your favorite artists we also create prototypes to get a sense of what the feature might feel like and define hypotheses how will this feature impact user behavior and our core metrics will they share more music will they log in more often whenever possible we put real prototypes in front of real users once we feel confident this

thing is worth building we go ahead and build an MVP Minimum Viable Product just enough to fulfill the narrative but far from feature complete you might call it the minimum lovable product anyway the real learning happens once we put something into production so we want to get there as quickly as possible again we deploy the MVP to just a few percent of all users and use techniques like a be testing to measure the impact and test our hypotheses the squad monitors data and continues tweaking and redeploying until they see the desired impact then they gradually

roll out to the rest of the world while taking the time needed to sort out practical stuff like operational issues and scaling by the time the product or feature is fully rolled out we already know it's a success because if it isn't we don't roll it out impact is always more important than velocity so a feature isn't considered done until it has achieved the desired impact so with all this experimentation going on how do we actually plan how do we know what's going to be released by which date well the short answer is we mostly

don't we care more about innovation than predictability and 100% predictability means 0% innovation on a scale we'd probably be somewhere around here of course sometimes we do need to make delivery commitments like for partner integrations or marketing events and that sometimes involves standard agile planning techniques like velocity and burn up charts but if we have to promise a date we generally defer that commitment until the feature is already proven and close to ready by minimizing the need for predictability squads can focus on delivering values that have being enslaved as someone's arbitrary plan one product owner

said I think of my squad as a group of volunteers that are here to work on something they are super passionate about an amazing new product always starts with a person and a spark of inspiration but it will only become real if people are allowed to play around and try things out so we encourage everyone to spend about 10 percent of their time doing hack days or hack weeks that's when people get to experiment and build whatever they want no limits like this dial a song product basically a Spotify enabled analog phone just dial the

number of the song you want to listen to is it useful doesn't matter the point is if we try enough ideas we're about to strike gold from time to time and quite often the knowledge gained is worth more than the actual hack itself plus it's fun in addition twice per year we do a Spotify wide hack week hundreds of people hacking away for a whole week the mantra is make cool things real do whatever you want with whoever you want in whatever way you want and then we have a big demo and party on Friday

it's amazing how much cool stuff can be built in just a week with this kind of creative freedom whether it's a helicopter made of lollipop sticks or a whole new way of discovering music turns out that innovation isn't really that hard people are natural innovators so just get out of their way and let them try things out you notice we have an experiment friendly culture Thule or tool B let's try both in compare do we really need sprint planning meetings don't know let's skip a few and see if we miss them should this button be

in the middle or in the corner let's try both an a/b test even the Spotify wide hack week started as an experiment and now it's part of our culture so instead of arguing an issue to death we talked about things like what's the hypothesis what did we learn and what will we try next this gives us more data-driven decisions and less opinion driven ego-driven or Authority driven decisions although we are happy to experiment and try different ways of doing things our culture is very waste repellent or lean if you prefer that means people will quickly

stop doing anything that doesn't add value if it works keep it otherwise dump it for example some things that work for us so far our retrospectives daily stand-ups Google Docs get and guild on conferences and some things that don't work for us our time reports handoffs separate test teams or test phases and task estimates we mostly just don't do these things we are also strongly allergic to useless meetings and anything remotely near corporate bs one common source of waste is what we call big projects basically anything that requires a bunch of squads to work tightly

coordinated for several months big project means big risk so we are organized to minimize the need and instead try to break projects into a series of smaller efforts however sometimes a big project is necessary and in those cases we found some practices to be essential visualize progress using various combinations of physical and electronic boards do a daily sync meeting where all squads involved meet up to resolve dependencies do a demo every week or two where all the pieces come together so we can evaluate the integrated product together with the stakeholders these practices reduce risk and

wastes because of the improved collaboration and short feedback loop we found that a project also needs a small type leadership group to keep an eye on the big picture typically a tech lead product lead and sometimes a design lead no project manager so far but that might change in general we're still experimenting a lot with how to do big projects and we're not so good at it yet one of our big challenges is growth painting as we grow we risk falling into chaos but if we overcompensate and add too much structure and process we risk

getting stuck in bureaucracy instead and that's even worse so the key question is really what is the minimum viable bureaucracy the least amount of structure and process we can get away with to avoid total chaos both sides cause waste but in different ways so the waste repellent culture and agile mindset helps us stay balanced the key thing about reducing waste is to visualize it and talk about it often so in addition to retrospectives and post-mortems many squads and tribes have improvement boards that show things like what's blocking us and what are we doing about it

we also like to talk about definition of awesome for example awesome for this squad means things like really finishing stuff easily ramping up new team members and no recurring tasks or bugs and our definition of awesome architecture includes I can build test and ship my feature within a week I use data to learn from it and my improved version is live in week 2 awesome is a direction not a place so it doesn't always have to be realistic but if we can agree on what awesome would look like it helps focus our improvement efforts and

track progress here's an example of an improvement tracking board inspired by a technique called Toyota improvement Kutta top left shows what is the current situation in this case the squad was having quality problems bottom left shows definition of Awesome in a perfect world we have no quality problems at all top right is a realistic target condition if we were one step closer to awesome what would that look like and finally the bottom right shows the next three concrete actions that will move us towards the target condition as these get done new actions are identified by

the squad boards like this live on the wall in the squad room and are typically followed up at the next retrospective all right I realize that maybe this video makes it seem like everything at Spotify is just great well truth is we have plenty of problems to deal with and I could give you a long list of pain points but I won't because it would go out of date quickly we grow fast and change fast and quite often a seemingly brilliant solution today we'll cause a nasty new problem tomorrow just because we've grown and everything

is different however most problems are short-lived because people actually do something about them this company's pretty good at changing the architecture process organization or whatever is needed to solve a problem and that's really the key point healthy culture heals broken process so the culture is so important we put a lot of effort into strengthening it this video is just one small example no one actually owns culture but we do have quite a lot of people focusing on it groups such as people operations and about 30 or so agile coaches spread across all squads and we

do boot camps where new hires form a temporary squad they get to solve a real problem while also learning about our tech stack and processes and learning to work together as a team all in one week it's like cultural shock therapy they often manage to put code into production in that time which is impressive but again failing is okay as long as they learn mainly though culture spreads through storytelling whether it happens on the blog at a post mortem a demo or at lunch as long as we keep sharing our successes and failures and learnings

with each other I think the culture will stay healthy at the end of the day culture in any organization is really just the sum of everyone's attitudes and actions you are the culture so model the behavior you want to see that's it I hope you enjoyed this story thanks for listening [Music] you