Amazon's Infinite Money Glitch

it was September of 2003 Andy jasse was nervous he had an idea that had been percolating in his mind for over a year simply put Andy thought Amazon should build and sell digital infrastructure to third parties but he was one of the few who understood the business opportunity Andy knew that if he didn't convince Jeff Bezos and the rest of the senior leadership team it wouldn't just be a step backwards for his career at the company Amazon would miss out on a massive Market he made his pitch and just like that Jeff Bezos approved it right then and there neither man knew it at the time but they just started a business which 20 years later is worth more than a trillion dollars this is the story of how Andy ji Jeff Bezos and the rest of Amazon discovered one of the biggest markets in history this is the story of Amazon web services AWS is one of the most revolutionary ideas in the history of tech and it's arguably the most successful second act for any major tech company today the world's biggest companies rely on AWS for everything from their storage and Compu to their payroll in crms and an entire generation of amazing startups that form the fabric of our everyday lives names like uber Netflix Airbnb all owe their existence to AWS it's not just startups either Nasa uses AWS the literal CIA uses AWS the history of AWS is a history of the ways Computing and software development and even capitalism itself have evolved over the last 20 years and that's important so maybe it isn't a surprise that the origin story is pretty complicated the main reason is that it's just plain complex especially for people who don't understand software and that lack of society-wide understanding has allowed some false narratives to emerge for example there's a theory that Jeff Bezos and other senior people at Amazon got the idea for AWS because they wanted to make extra money renting out excess server capacity it's true that Jeff Bezos was annoyed at how much Amazon was paying for Server costs but back then almost 20 years ago it wasn't possible to rent out capacity like it was a storage locker another reason why the origin story of AWS is so complex is that it's hard to get a real sense of what goes on behind the scenes at Amazon so you need to kind of piece it together from secondary sources but the main reason AWS has such complex Origins is because of the nature of innovation itself we love stories about individual Founders who came up with world changing ideas on their own and then built incredible companies but a lot of the time especially within large organizations that's not how it works don't get me wrong Jeff Bezos and Andy jasse are incredibly important people in the history of AWS probably the two most important people it just doesn't happen without either of them but they didn't invent the concept of AWS the truth is no one person did it was the culmination of dozens probably hundreds of memos meetings and random conversations Jeff and Andy's main contributions were in seeing the big picture of what Amazon was trying to do nurturing the good ideas and then maintaining a laser-like focus on execution that doesn't make their contributions less important and it doesn't make the story of AWS any less exciting in fact I think it's the opposite Amazon created a AWS because it was the only company in the world that could there's an irony here throughout the development of AWS Amazon's leaders were obsessed with building something that could be used by kids working on a project in their dorm room but they also wanted it to be so scalable that when those kids turned into Mark Zuckerberg or Patrick hollison they would stick with AWS like I said AWS has a complicated backstory but that doesn't mean we can't tell the story from the beginning to begin let's rewind a little a few years before Andy jasse made his pitch to Amazon's s team it was the late 1990s and Amazon had a problem it didn't look like it from the outside sales for axed between 1997 and 1998 Amazon was adding new features almost every day and hiring tons of Engineers but that insane growth was precisely the problem basically Amazon was suffering from success internally the company was running on what at the time was a traditional two-tier architecture on the one side you had a monolithic application that served the pages on the website then on the other side you had a bunch of different databases That Grew with every new customer product category and Market that architecture was basically fine when Amazon was a small online book seller but as Amazon expanded that system increasingly struggled to cope the end result was like a Jenga Tower one wrong move or one new feature that wasn't properly implemented and everything could topple over it was a big mess that created tons of redundancy and those architectural constraints created real business constraints Jeff Bezos didn't want to just sell books he wanted Amazon to become the everything store this problem came to a head when Amazon was hired by their competitors the large physical retailers like Walmart and Target to build out their e-commerce platforms I'll let Andy Jesse explained there were a few things going on I'll call it between the years 2000 and 2003 that made us think this could be a good idea you know the first was this business in Amazon called merchant. com where we sold our third our e-commerce technology at third party Merchants these are companies like Target marks and Spencer and when we went to deliver that solution to those companies it turned out to be way harder and way more timec consuming than any of us imagined and that's because we had uh really kind of jumbled up a bunch of parts of our platform growing as fast we did the first eight years and to deliver these through apis which is the way that these companies wanted to consume them it took a lot of decoupling and a lot of work that we just hadn't thought about before jasse and Bezos knew that the Legacy architecture put a hard limit on Amazon's future in fact it was already eating into its margins and because Amazon was so far ahead of its competitors there weren't many real life examples of how to solve this problem they couldn't just pick a solution off the shelf they pretty much had to invent one so that's what they proceeded to do back in 1998 a team of Amazon Engineers wrote something they called the distributed computing Manifesto at this stage I should stress something the creation of AWS was a deeply collaborative Endeavor dozens probably hundreds of people were involved but there are memos and moments and people which I'll talk about in this video that really matter the distributed computing Manifesto is one of those memos that really matters and just over 3,000 words the authors did three things firstly they identified the flaws in Amazon's existing two-tiered architecture secondly they proposed a different architecture model to replace it and then last but not least they described the change in mindset that needed to accompany the technical changes for the shift to work the model proposed in the manifesto was what's called a service oriented architecture or S SOA in software a service is something that's designed to complete a specific task like executing an operation or retrieving some information in the case of Amazon it might have been something like fulfilling an order or delivering it to the customer services allow different business functions to be performed remotely and independently in other words they're modular and flexible ideal for a company that wants to scale quickly s SOA is just the name for all those different Services working together if that still doesn't make a whole lot of sense don't worry here's an analogy for Amazon transitioning to service oriented architecture was like going from building on top of a Jenga Tower to building with Lego blocks Jenga was delicate and fragile any new feature or change to an existing feature and the whole thing could collapse but Lego is different the blocks fit together perfectly they're reusable and they're almost impossible to break the distributed computing Manifesto was a blueprint for a different kind of company it would go on to influence Jeff bezos's thinking as he considered how to modernize Amazon's architecture he met with people who urged him to embrace apis as a way of opening Amazon to outside developers he had long conversations with his most trusted employees including Andy jasse he read books about architecture design and then after months of absorbing advice and information Bezos made a call it was time to take a hammer to the existing monolithic architecture it was time to embrace s SOA Bezos knew it would be one of the biggest leadership challenges of his career because it's no exaggeration to say it could make the difference between Amazon being a trillion dooll company or just another internet retailer Lost In The Mists of time the distributed computing Manifesto is an important document in Amazon's Journey from a monolithic architecture to building AWS except it was more of a mandate here's what it said all teams will henceforth expose their data and functionality through service interfaces teams must communicate with each other through these interfaces all service interfaces without exception must be designed from the ground up to be externalizable anyone who doesn't do this will be fired I'm pretty sure that's not the actual text of the memo Amazon has never released it and as far as I know Jeff Bezos has never acknowledged it the reason we know it exists is because of a mistake Steve Y is an engineer who's worked at senior roles across Amazon Google and grab but to the public he's best known for accidentally publishing an internal Google+ memo in October of 2011 in the post which was mostly criticizing Google yay talks about the Bezos memo he talks about how it was a major turning point in Jeff bezos's thinking about Amazon's architecture which therefore makes it a pivotal moment in the history of AWS so let's break it down what does it mean and why does it matter okay so the first thing to understand is that when Bezos was talking about service interfaces he was talking that apis very quick hyper simplified crash course an API or application programming interface is a mechanism that allows two different software components to talk to each other according to set rules for example whenever you open the Weather app on your phone it pulls the current data from the weather bureau's Software System using an API for Amazon customer facing apis help to do things like create rankings offer customer recommendations and retriev save customer details they're like Lego blocks small modular unbreakable Bezos believed apis the way forward for Amazon he' even created a team within Amazon whose sole purpose was creating them to share with thirdparty developers that led directly to incorporating apis into merchant. com almost immediately developers came up with innovative ways to display the product catalog on their sites those advances and the progress that Amazon was making building apis inspired Bezos to go a step further he figured that if Amazon could build a solution like that and transition from its monolithic architecture to service oriented architecture maybe the company itself should follow the same same pattern think about how profound a change this was as yay pointed out in his accidentally public memo organizing into Services taught teams not to trust each other in most of the same ways they're not supposed to trust external developers it sounds completely counterintuitive but it was precisely the lack of trust that made it work all of a sudden every team within Amazon had to interact using these so-called hardened interfaces if you were HR and needed some numbers from the marketing department guess what you had to build an interface to retrieve the numbers it was a totally scalable decentralized system hundreds of individual Lego blocks once you truly understand the implications of what Bezos was trying to do you realized that he was shifting Amazon onto the path that culminated with the creation of AWS one of the best decisions Amazon made right around this time was hiring Alan verin ver Mulan wound up becoming Amazon's CTO but his first job was to build tools that disentangled Amazon's messy code base and separated it into hardened interfaces that individual teams could incorporate into their work without without worrying if they were going to break everything there's a term in software engineering known as Conway's law in its general form it says that the technical infrastructure you build will mirror the structure of your organization I don't know if Jeff Bezos knew about Conway's law he probably did he's a smart guy either way he was enacting Conway's law within Amazon in July 2002 a couple years after Alan verin was hired Amazon held a developer conference today Amazon's reinvent conferences are attended by thousands and watched by millions but just eight people attended the conference where Amazon announced the launch of their new division Amazon web services needless to say it was nowhere near the AWS we know today it was really just a way for outside developers to access the amazon.

com product catalog still it was an indication that something big was beginning to happen developer conferences were cool but Amazon had bigger fish to fry this is the part of the AWS story where Andy jasse becomes more prominent so let's talk about him for for a minute today Andy is Amazon's CEO but despite rising to the top of one of the world's biggest tech companies he's not a software engineer he's a Harvard NBA who's so obsessed with sports he almost became a sports caster in another world he's giving color commentary on ESPN thankfully for everyone except Amazon's competitors he decided on a more conventional career path Andy joined Amazon's marketing department way back in 1997 after the department was downsized in the wake of the dotom crash Bezos appointed Andy to be his shadow a hybrid role which was part technical assistant part Chief of Staff it was a strong signal to everyone that Andy was a rising star now it was his turn to prove it remember Amazon was focused on rapid growth and that presented new challenges for example Amazon wanted to start selling clothes but clothes aren't the same as books books are almost perfect Commodities but clothes aren't they come in different sizes and colors in order to sell clothes Amazon's Engineers people like Alan ver Mulan had to build all kinds of new features a bigger product catalog and more website traffic meant amazon. com needed more Community features like username verification and review ratings but those features weren't being added as quickly as they should have been Andy realized that the reason projects were taking so long was because software Engineers time was constantly being consumed by doing work that wasn't directly related to the feature they were building instead they were spending as much as 2third of their time building databases storage and compute infrastructure it's easy to see that this was a big problem Amazon needed a solution if it didn't find one it risked squandering the advantage it works so hard to build an answer arrived just in time and once again it came in the form of a classic Amazon memo the author was a guy named Matt round a senior engineer who was persistent and outspoken about how hard it had gotten to actually build things within Amazon his memo opened with a bang to many of us Amazon feels more like a tectonic plate than an F-16 that's a great Hook from there Matt set out some principles for maximizing the amount of time that Amazon's Engineers spent building cool things maximizing their autonomy adoption of rest architectures standardization of infrastructure removal of bureaucracy and continuous deployment if you work in software today none of those recommendations sound radical to you but remember this was 20 years ago Matt's memo was probably on the agenda when Amazon's senior leadership team gathered at Jeff bezos's house in Medina for an offsite all the company's biggest hitters were there Jeff Bezos obviously it was his house Andy jasse the CTO Alan veran the CIO Rick delzell and others they were there to brain storm new business ideas and list out Amazon's core strengths they thought it would take half an hour an hour tops but as the meeting went on the people in the room began to realize something the distributed computing Manifesto Amazon's migration to S SOA Amazon launching web services in 2002 and thirdparty developers building more than 100 applications within 2 years Matt round's memo these weren't disperate threads they were all pointing in the same direction a couple of years after AWS properly launched Jeff Bezos gave a talk at why combinator startup school it was a few years before my time but people were still talking about it when I was there it's a good thing and I mean for all of society that the talk was recorded I was in Luxembourg and I took a tour of a 300-year-old brewery their business of course is making beer and about a 100 years ago they were one of the first uh factories in Luxembourg to start using electric power to make to help make beer to help in the manufacturing process and of course they couldn't buy the electric power off of the electric grid because there wasn't an electric grid so they started making their own electric power if you could make your operations more efficient or do new things with electric power the only way to get power was to set up your own generator and become an expert in electric power generation the important thing to notice here is that the fact that they generated their own electric power did not make their beer taste better and what startup companies want actually companies of all sizes is they want to go from their IDE a or their Vision to a successful product as quickly as possible the problem always is that there's a lot of undifferentiated heavy lifting that gets in the middle between your idea vision and that successful product the undifferentiated heavy lifting by the way has to be done at worldclass levels of Excellence or your vision will fail but it's totally undifferentiated and isn't actually making the beer taste better teams inside Amazon had been spending too much time doing things that didn't make their beer taste any better and if that was happening at Amazon then it must have been happening everywhere Amazon already offered limited software tools to developers and businesses through Amazon web services 1. 0 and the company was already building the digital infrastructure it needed to run its expanding operations maybe it was time to take that infrastructure expertise and externalize it so other people could make their beer taste better the idea for AWS didn't emerge fully formed from that offsite in 2003 but it was the day the starting gun was fired we've now arrived at the part of the story where Andy jasse wrote his six-page vision statement for AWS by now you know how high the stakes were but I haven't explained what was actually in that memo to inform his thinking Andy assembled a tiger team of the 10 best technical Minds within Amazon and together they deconstructed the major web services of the day like Google and eBay they reflected on their own expertise within Amazon on Amazon's principles as a company their target audience their competitors and the vision emerged the team came up with a list of Primitives basic highly flexible Services storage compute databases and a Content distribution Network these are the raw materials of what became AWS the Lego blocks in other words the team also decided that in order for AWS to make an impact it had to be flexible and scalable it had to be built from the bottom up and it had to launch with as many services as possible it wouldn't be easy but the opportunity was right there as a result of all that planning and deliberation Andy's memo didn't just suggest that Amazon rent out their unused server capacity he suggested Ed a pricing model he sketched out Technical and implementation details it was a collaboratively produced road map no wonder the s team was impressed let's go back to Alan veran because he wrote the six-page proposal for what became the first core component of AWS the simple storage service or S3 for short he wrote most of it at the six arms a pub in Seattle and because he had already contributed a lot to Amazon his ideas were taken seriously his proposal was approved and he put together a team to build it Jeff Bezos was so closely involved in the development process that Alan verm Mulin called him the product manager of S3 he asked questions suggested ideas and challenged Allan and his team to think big having Jeff Bezos as your product manager would be pretty scary but it showed that he was Allin Amazon never had to raise any Capital to build AWS because the company's senior leaders were committed to the project and took the time to understand it on March 14th 2006 Amazon launched S3 S3 stores objects which is just any text images and other assets that customers create as part of their normal business it's a massive disc drive in the cloud before S3 companies including Amazon itself were paying way too much for data storage the old business model created a tough dilemma for companies that handled lots of data either you bet on yourself in which case you needed to spend massive amounts of money to install fixed amounts of server capacity or you set your sites lower spending Less in the short term but limiting long-term potential it was a l lose situation so many businesses either didn't grow as large as they could have or were never founded in the first place all because of data storage costs S3 totally turned that on its head all of a sudden you could store data in the cloud and you'd only pay for what you used you didn't need to have Deep Pockets to buy space in S3 you could sign up with an email address pay with a credit card and get started that day it was as flexible as you were and according to Andy jasse it was informed by what Amazon's customers were telling them ver Mulan says that just before he retired from Amazon in 2021 he found the original S3 design document the growth projections were off by a factor of thousands within 2 months the number of objects stored on S3 exceeded Amazon's Expectations by a factor of 100 6 years after launching S3 had more than a trillion objects in storage today that number has grown to over a 100 trillion that's what happens when you make infinite data storage convenient and affordable Amazon disrupted the old business model because S3 scaled with its customers in effect it allowed customers to trade upfront costs for variable costs every business will make that trade Amazon's margins on S 3 were never as big as oracles or ibms but that's fine Amazon had years of experience dealing with lower margins from their retail business in addition to being highly cost-effective the S3 pricing model was optimistic because when you offer a solution that lets people focus on making their beer taste better and encourages other people to get into the Brewing business some of them are going to win and when they win they're going to need a lot more storage they're going to need a lot more of everything that requires them to spend more money and if you take care of them they'll spend it on you in August of 2006 just a a few months after the launch of S3 Amazon launched the second core component of AWS the elastic compute Cloud also known as ec2 there's actually some controversy about how ec2 was created the official narrative is that ec2 or something like it was in the memo that Andy presented to Amazon's s team The descenting Narrative is that the ec2 concept was actually developed by a guy named Chris Pinkham who is leading Amazon's network engineering operations at the time Andy jasse actually addressed this in his Infamous one-star review of the book the everything store when he wrote The Vision document proposing the AWS business and outlining the initial set of services for AWS including our compute service ec2 was finished and presented to the executive team in September of 2003 I wrote the document and was lucky to have the help of several people in putting it together you won't find much mention of Chris Pinkham in the official AWS story who is Right does it even matter it matters because it reminds us that a concept as disruptive as AWS could only come from from Amazon itself and it matters because it's a reminder that the idea is just a small fraction of the actual work you've still got to actually build the product and the business and Chris penum had left Amazon before ec2 even launched so what is ec2 it's basically a network of virtual computers that you can use to run applications you log in and configure a virtual machine an instance it's elastic because a user can create launch and terminate instances as they need the same way that S3 blew people's minds by providing infinite storage at low cost ec2 provided a way for everyone to perform intensive computations in the cloud it was truly revolutionary all of a sudden you didn't need to spend months and millions of dollars on data centers you could create an account and let Amazon with their unbeatable scale do the heavy lifting as a value proposition it was hard to resist at this point it's important to differentiate between two types of AWS customers startups and Enterprises because they both tell different stories about its massive success for startups the use case was immediately obvious lowcost storage and compute when you needed it if you were were a developer you could even just build applications on top of it but initially there was some skepticism about whether established Enterprises would adopt it with the same enthusiasm even if the status quo wasn't perfect it seemed like a risk for Enterprises to lift and shift their storage and data center operations from what they had onto AWS enter Netflix we were facing some extraordinary growth so um streaming at that point in 2008 was about a million hours a month and now it's over a billion hours a month so we've had this thousandfold old ramp up in computational resources necessary over just four years with the ad basically Broadband networks and streaming and so you know we had to take some risks you know doing that kind of buildout with your own data centers was going to be challenging um and uh it's worked out great for us and the key is that now we're on a cost curve and on an architecture that you know with as all of this room does more with AWS we benefit by that Collective effect that gets you to scale and brings the prices down Netflix started out by mailing DVDs to subscribers but with the opportunity of streaming ahead of them they realized they needed to change up their business model at first Netflix actually partnered with Microsoft to build out their video streaming capacity it cost millions of dollars switching over to AWS looked risky but Reed Hastings knew that not switching over was a much bigger Risk by the time he spoke to Andy jasse at the first ever reinvent conference in 2012 95% of Netflix's computation and storage was in AWS there's something else worth mentioning here as AWS was getting better it was also getting cheaper one of the major announcements at reinvent was a 28% reduction in S3 storage costs enabled by infrastructure upgrades and economies of scale Amazon didn't really need the credibility of a customer like Netflix and it probably wasn't even the first big Enterprise to switch to AWS but Netflix switching over showed everyone that not only did Enterprises have nothing to fear from AWS they could use it to become a better company and it wasn't just Netflix Airbnb launched in 2008 Uber the year after stripe the year after that all of them are multi-billion dollar companies none of them would look the same without AWS today AWS customers use it for absolutely everything from Storage development and testing to Scientific Computing CRM and maintaining Mission critical Financial systems more than 15 years after launch ec2 and S3 still form the core of AWS but Amazon's Engineers never rested on their Laurels they kept rolling out new features and they just kept on going in late 2008 they released cloudfront a service that speeds up delivery of web content virtual private Cloud a serverless database service a massive data warehouse platform with each new AWS product Amazon was sending a clear message we're listening to our customers and squeezing our competitors S3 originally launched with eight microservices when Amazon's CTO Warner Vogal took to the stage to deliver the keynote address at the 2022 reinvent conference it had more than 235 AWS doesn't get taken offline for maintenance in fact outside of a service disruption in 2011 AWS has pretty much never gone offline AWS customers never have to upgrade to a new version because the service is constantly being updated all of this goes back to that fundamental point I made earlier coming up with the idea of AWS was an incredible achievement that's worth studying but actually executing day in and day out is what's allowed Amazon's leaders to grow it into the world's most successful internal startup today you could easily build an ancestry.

com style family tree of all of the different AWS services but beyond the different products and services is an extremely simple value proposition the reasons we hear people people moving in the cloud you get to trade capex or variable expense that variable expense is lower than what you can do on your own it provides you elasticity so you only pay for what you consume it lets you move much more quickly it lets you use your scarce resources of Engineers on things to differentiate your business and then you can go Global and provide that customer experience in minutes or days as opposed to months the story of how Amazon built AWS is also the story of how its Rivals failed the simple version is that Amazon beat the existing data storage and database providers by providing a better product at lower price and beat Microsoft and Google by being first to Market there's an alternate reality where I'm making this video about Microsoft and it's not too different from this one because to be honest Microsoft should have figured out cloud computing sooner the reason they didn't was because the company was a total mess in the Steve Balmer era its priorities were all wrong Windows teams had too much power internally so anything that looked like it could cannibalize that Revenue stream was watered down or scrapped entirely meanwhile Balmer was too focused on taking down the iPhone he eventually got with the program and of course SAA Adela saw the potential of cloud computing right away but by the time Microsoft took Azure seriously it was already a long way behind AWS it's been playing catchup ever since what about Google the theory I've read which I agree with is that basically Google didn't have the right combination of skills and incentives internally to see the opportunity like Amazon did Google became one of the world's biggest and most cash generative businesses mostly through a single product search but it never really needed to get its hands dirty its leaders never had to learn to sell to consumers or Enterprises they never had to learn how to grow under tight profit margins so they weren't really equipped to build a platform like AWS of course today both Microsoft and Google do have cloud computing platforms of their own recent estimates put Azure at about 20% market share while Google cloud is at about 10% so they're making inroads but both companies should have been competing with AWS years before they actually did but they were too distracted by the money they were making from software and advertising to see the opportunity Andy jasse talks about this all the time everyone in Amazon was stunned that their competitors were so late to cloud computing AWS ran the table for years because it was first to Market and even as real competitors have emerged Amazon's maintained its Advantage because AWS is easy use cost competitive and because it's the default Choice the numbers around AWS are hard to comprehend more than half of Amazon's total operating profits have come from AWS its Revenue curve is absolutely insane and if it switched off the lights on Amazon amazon.