Benedict Evans: AI Eats the World | Slush 2024

[Music] so thank you for that introduction every time I speak at another event I always ask if there'll be lasers but somehow no one else has Managed IT um now I've asked was asked to come and say insightful sensible clever things about Ai and explain everything that's happening in Ai and not talk too fast and and only take half an hour um I'll probably manage two of those things possibly one but I'll see what I can do I think a good place to start in thinking where we are with AI today is this quote from

Bill Gates from 18 months ago saying that in his whole career he'd seen two things that were revolutionary the graphical user interface and chat GPT which is a pretty big statement and then on the other side um a month ago two months ago open a raising at a valuation of $160 billion I went and looked this up and worked out that it took Microsoft about 25 years to get to a valuation of $150 billion and openai did it um 12 months 18 months after launching their product and again that kind of reflects just how quickly

this has all happened um this is a survey of people around the world who've used or at least heard of chat GPT and this is kind of unprecedented speed of adoption for a new technology something between a third and 2third of people have tried this or at least heard about it already um and some of that just reflects the fact that you don't have to wait for everybody to go out and buy a phone and wait for Telos to build Broadband it's just a website and it just runs on the cloud so you can go

and use it but nonetheless it's a pretty amazing um speed growth penetration interest level of excitement around this thing on the other hand if you ask people what they actually mean when they say they've used it you get a slightly different picture the answer is that most people used it once and said well that was very clever and then didn't go back um and a much smaller number of people have worked out a way that they can make this part of their life every single day you see something similar I think if we look at

the Enterprise every big company has got some kind of a pilot or a trial which might just mean that the CIO um is paying for a chat GPT account um but a smaller number of companies have actually got something into deployment and of course this is just one having one thing in deployment it's nothing like kind of moving your entire system over to um over to generative AI so we've got to this point where on the one hand everyone is very excited and interested and thinks this is important but we haven't actually got to the

point that everybody is using it or spending money on it at least not yet that got us this summer to a wave of people writing stories like this saying well wait a minute are we quite sure we want want to spend half a trillion dollars on this thing when we don't have product Market fit yet um and some of this is just kind of the normal hype cycle for anyone who doesn't know the idea of the Garten hype cycle I ask chat GPT to draw it for me um apparently it will take time to get

to the pom of producy um so we'll kind of see um quite how this will evolve but if we step back from that um this is probably the more relevant cycle to be thinking about that every 10 or 15 years to take into goes through a platform shift and that becomes the center of all of The Innovation and investment and Company creation and change in the industry and so that was main frames from the mid 60s to the late '70s and the PC and then the web and then smartphones and now pretty much everybody in

Tech thinks that generative AI is the next big platform shift however that's about the only thing that we really think is clear after that all the questions are wide open in fact we're still trying to work out what the questions are and what I've tried to do in the rest of this presentation is to try and group some of the questions and work out what we might ask if not what the answers might be into three categories like how far is this going to scale which of course is like the foundational question for everything else

how are these models actually useful what are you supposed to do with them and then of course how do you actually deploy this which for people in this room means how do you build a product and a company out of this stuff um how do you actually take it to Market and so looking at the first of these uh how far will this scale as I think we all know or certainly everyone in this room knows we made this stuff work by making it much bigger than any thought was anyone thought was remotely feasible we

made it much bigger with much more data and much more compute and that got us better results and so the question now is we don't actually have any theory of why that works but it does work or it's worked so far so will it keep working and there's one view that says no it's going to scale down inevitably like anything else it will slow down and so this will be just software and then there's another view that says no the scaling will just carry on working and at the extreme that might mean that can just

kind of do the whole thing they can replace all the other software and everything else and you can spend hours of your life watching machine learning scientist arguing about this on YouTube and all you'll really conclude is that they don't know um it's got Kevin Scott saying well it's worked so far and sergy Brin saying well yeah but just cuz it's worked so far that doesn't tell us it'll keep working um we just don't know um in the last week we've had a flurry of stories from inside some of the big Labs saying maybe it's

already stopped working um a lot of argument about quite what that means and seems a little bit premature to just assume it's suddenly all ground to a halt and of course some of this is just that scaling these models has a bunch of practical difficulties um it takes time to build gws more power it takes time to get the gpus it's not clear how much training data is left and if we can use synthetic data or what other kinds of data we could use um and of course there's that foundational Science question if you do

make the model 10x bigger will you get 10x better results we just don't know um however we're going to find out um and so these are both um um Google and meta saying um that the downside of not investing is bigger um and letting somebody else take this over is the downside of spending a bunch of capex that ends up not being useful for a couple of years it's a quote from Sil Silicon Valley I think was I don't want to live in a world where somebody else is making the world a better place faster

than we are um and that's some of what's going on here especially if if like Mark Zuckerberg you've got a controlling stake in the company um the thing of course that drives that fear is that there isn't doesn't seem to be any kind of fundamental Mo there doesn't seem to be any sort of fundamental Network effect any fundamental reason why one company will have a winner takes all effect and so this is an internal Memo from Google from last spring so well we don't have a Moe and neither does neither does open AI um of

course it's not quite true because the moat is capital capital this is the um CEO of anthropic very casually talking about 5 or10 billion of build of model building he's now started talking about1 billion of model building which seems a little bit 1999 and slightly more tangibly Mark Zuckerberg here this summer said that the latest version the next version of llama will need 10x more compute because that's just kind of how you go out and build them and just to put some numbers on that this is not a semiconductor presentation but llama 3.1 was 16,000

Nvidia gpus so about half a half a billion dollars worth of chips plus the data centers on top and the comput the Clusters that people are using today are more like 100,000 gpus so there's quite a lot of money and quite a lot of chips and quite a lot of infrastructure and of course a lot of that money um is going to Nvidia um they updated their numbers last night so the numbers carry on going up and to the right this is kind of an unprecedented chart you have to go back to like 1999 or

2000 to see charts that look like this um and in turn most of that money is coming from the four big platform companies which will spend something over $200 billion of capex this year almost hundred billion increased capex from last year um and they all expect to spend even more in 2025 at least that's what they say at the moment it's kind of interesting um to look at what that means for Microsoft in particular um this is capex to sales for typical Telco Verizon in the USA and we think of Telos as being infrastructure companies

they spend about 15% of their revenue on capex every year Microsoft will spend over 25% of its revenue on capex this year it's gone from being a company that sold you um a $1 CD for $100 in a cardboard box to a company that actually has to build infrastructure in order to sell its products um of course when you have these kinds of surges of money and people talking about hundreds of billions of dollars of construction then private equity and bankers and Wall Street start to get interested um um a little company called andrees and

horz you might have heard of apparently has 20,000 gpus in order to win AI deals um and of all of this of course is coming in the context of a market where we still don't really know how this is going to work um all of the science is still changing all of the engineering is still changing this started working 2 years ago and that was amazing and then what well we have a hugee amount of investment into creating better results agent models um scaling of course that I've talked about multimodal models on the other hand

a huge amount of investment into trying to make this slightly less um expensive as we've applied a huge amount of kind of data center engineering to what was previously a science project and that cost I think is is is really important here because the last time the software industry had marginal cost um was back when this was what software looked like kind of before most of us were born this is the last time that it cost you money every time the user pressed okay and waited for something to happen and the whole consumer internet has

grown up on the premise that you basically don't have marginal cost in s your YouTube or someone um but llms at least for the for for for the moment still kind of do that gets you inter to charts like this so this is um a cost axis horizontal axis is cost vertical axis is quality for the sort of succession of models that have been released by open aai you got a relatively small increase in model quality but a huge increase in model efficiency certainly an order of magnitude maybe more than an order of magnitude increase

in in in in in how cheap these things are depending how on how you model it and in parallel there's an awful lot of other models now um so these are just the models from the other three big companies um I could probably add I could probably double the number of points on this chart if I included everything that's been released in the 2 year in the last 2 years but what's really happening amongst all the noise is we're seeing a convergence on the one hand everybody has got a best model and then everybody's got

a cheap model and the thing is you can have the model that's almost as good for about 5% of the price so you've got a very very steep um cost curve what we also have on this chart um is meta who are giving the model away for free or almost free and there's an old saying in the tech industry that everyone in Tech is giving somebody else's business model away for free that's what meta is trying to do with open source they're trying to make llms into commodity infrastructure that's available from anybody um and that's

sold at marginal cost apple is doing something similar from the opposite direction Apple sells High Performance Edge comput so they want to turn llms into commodity infrastructure that runs on your iPhone and it becomes just another API call and so we've had in the last 2 years a sort of model boom in which you have can choose better faster cheaper you can have the best model you can have the best cheap model you can have the best model that fits on the edge and that's kind of a big story for the next year and of

course there's another model that's more and better every couple of weeks now if kind of step back from this if you're not a machine learning scientist or a data data center engineer however I think there's a couple of fairly simple observations one can make um here we have Michael Corone telling us that if anything life is simple that you can kill anyone and I think the technology industry equivalent of this is that semiconductors are a cyclical industry um and commodity technology tends to go to marginal cost and it tends to become even more of a

commodity um and meanwhile every new technology tends to produce a bubble um and it's very easy to call a bubble it's much more difficult to say when when the bubble will pop but it's relatively straightforward to point to the trends that we've seen in the past now if we set all of that aside then we can ask well okay this is great there's going to be lots of models and they're going to get cheaper but what are we going to do with them how exactly is this useful and to answer that question I think it's

kind of useful to go back and think about the last wave of machine learning that started now almost exactly a decade ago with um image recognition and I would go to events like this and I would show people demos like that and I would say look isn't this cool and the B companies in the audience would say uh well done we're very happy for you you can recognize a dog we are a very large German reinsurance company we don't have pictures of dogs what are we supposed to do with this and it took a while

to work out that the right level of abstraction the right way to understand this was that this is patent recognition and once you understand that this is patent recognition then you can start thinking well what can we turn into patent recognition and every software company for the last decade has basically been saying we can turn that problem deep inside this industry or that department or that Corporation into patent recognition and this is how we get the data and we'll build a company around that um we all kind of understand this now um we're kind of

at the same point now I think which lar with with with generative AI with large language models okay that's a great demo and I can kind of see why that's cool but I'm not quite sure what how I should understand this at a kind of a conceptual level what is this how do we think about what we could do with this we also of course have to remember the things that we can't do with this you can't do this use this as a database this is still my favorite example of hallucinations um Air Canada used

an llm to build a customer support bot a customer asked about the return policy the support bot gave them a return policy it was a very good return policy unfortunately it wasn't Air Canada's returns policy um and the customer had to go to court to get a refund because Air Canada said well you should have read the other part of our website not that part and the judge did not find it's very impressive um and so there's a sort of challenge here in understanding what it is that these things are doing um and in thinking

about how it is that you would build products around something that can automate a task but might not get it right um one answer is well just make the models better which would be nice but we don't really know if we can do that um and it's not particularly clear that a probabilistic model can produce deterministic answers anyway that seems a little bit like Zeno's Paradox um so you ask well on the one hand what are use cases where there isn't a wrong answer or use cases where the errors are easy to see on the

other hand how do you abstract this away how do you build product design around the fact that it might be wrong how do you manage that provide tooling analytics metrics user experience to manage those kinds of questions um the big place of course at the moment this is interesting is asking can you use this for General search can you use this to replace Google and it there's a lot of sort of of unknown answers to that what kinds of queries is this good for what kinds of queries does it matter if it's wrong where does

it matter if you can't tell that it's wrong how much filtering do you need on this we don't really know neither does anyone at open AI but alphabet made so much money from search that it seems like it's worth trying to find out and so if we go back to my question like how do we work out the right level of abstraction to understand this we can sort of say well it's reasoning and synthesis and summary or it might be um we can step back and say okay this automates a class of things that we

couldn't automate before and we're kind of trying to work out what things we can automate one of the ways I used to talk about the last wave of machine learning was to say that this gives you infinite interns so you would like to listen to every call coming into the call center and tell me if the customer's angry you don't need an expert for that you could just use a 10-year-old um except you could never automate it and machine learning lets you automate kind of a whole class of thing that you just kind of need

a mamal brain to do um you can kind of look at generative Ai and try and work out from kind of conceptual pivot Points so you can say what does it mean that creating certain kinds of content is going to be free what does it mean that translation will now be more or less perfect and free what does that do to science or pop culture or the nature of the web you can also try and do it top down as a kind of macroeconomist and you can say which Industries have low labor productivity which Industries

seem to be full of lots of people doing very boring work that's very repetitive that somehow we couldn't automate before we'll probably be able to automate all of that and then you can kind of go across Industries and kind of produce a plot like this I think the challenge with this kind of exercise is imagine that youve done this for the internet in 1995 what would you have got right and which things would you have completely missed can you really kind of model the evolution of a trans of of a transformative technology like that I'm

not sure that you can and meanwhile if you're actually sitting in a big company saying why should I double my Microsoft spending you get to quotes like this um Chevron CEO employs two or 300,000 people Microsoft is saying you should give 200,000 people um a license for co-pilot and double your Microsoft ban and he's thinking yeah but why what value is it exactly that we're going to get from that what are they going to do with this I think perhaps kind of a slightly more useful way to look at this is to try and unbundle

it a little bit um this is um a wonderful advertisement from the 1970s for visal which was the first soft successful software spreadsheet and so what you do with this is you type in a grid of numbers and then you can change a number here and all the other numbers on the screen change and we look at this today and we think yes but if you're an accountant in 1978 that was your entire life you could have spent weeks doing that and now this software just did it for you in like 10 minutes he so

the founder of this company has all these stories of accountants who would do a a one month project in two or three days and then go and play golf for 3 weeks and then come back and say I finished now so if you're an accountant and you saw this you had to have it but if you were a lawyer and you saw it you would think well that's very clever and maybe my accountant should see this but I don't have that use case that's not what I do and I think that's the way the other

90% of people react when they see chat GPT they say well that's very clever but I don't do that all day so if we ask who has that use case today for generative AI particularly for chat gbt there's some things that very clearly do have that use case so software development works right now and companies are already talking about 20 and 30% efficiency gains people in marketing are saying something very similar don't really have wrong answers and the error if there are wrong answers they're kind of easy to see a lot of people in Customer

Support very interested in this although as I said earlier you kind of have to be careful about that and then you've kind of got early adopters across every industry who are starting to play with it and try and work out what they would do with it and that gets you to some numbers like this how many employed people in the USA are using generative Ai and how much are they using it and you can see in some Fields something like 20% of people are willing at least to claim that they're using generative AI now of

course you could suggest here um which fields are people most likely to and is anyone in management going to admit that they're not using chat GPT yet um maybe not um on the other hand over in something like law it kind of matters um if this thing says something that isn't quite right so you have this kind of widespread of adoption um for everybody else I think though you've got this problem that we have a technology um and we're asking the user to work out what to do with it which gets us to this great

quote from Steve Jobs that's kind of the not normally the way we go about deploying technology we normally start from the other end we start from working out what the customer experience should be and work back to the technology when right now we're kind of trying to do the opposite and so that gets me to my my third section how is it that we deploy this how do we build products with this how do we build companies with this um and as we ask that question there's a bunch of very common patterns for every technology

how do we always do this well first of all you try and absorb it the incumbents try and make it a feature we bolt it into the existing business we automate the things we already do then over time you start changing the way you do your business you start creating new products you start creating new ideas and maybe unbundling the incumbents you unbundle that of something out of sap or Google or Oracle and then every now and then someone will come along and actually fundamentally redefine what the market is Airbnb comes along and changes what

you mean when you say a hotel that's a bit that's kind of hard to predict and so you can kind of ask as you look at this are you asking an accenta kind of a question or a bane BCG McKenzie kind of a question is this a question for your CIO or is this a question for the CEO is this Topline Innovation or bottom line Innovation and the answer is it might be all of those depending on which part of the company you're talking to which part of the industry which kind of consumer which kind

of the problem it might be all of those at once the first answer of course is that you call up McKenzie um sorry you call up accenta and you give them an RFP and so accenta reported that they're now doing a billion dollars a quarter of generative AI although I'm not quite sure what they're calling generative AI but whatever exenta calls generative AI they're doing a billion dollars a quarter in that and that gets you to some very kind of boring traditional Enterprise procurement questions do you buy versus build do you use the big companies

or small companies will Google own the whole thing and the most important question of all what does this do to our EPS um and that kind those kinds of questions get you charts like this um for most people in this room cloud is old and boring and done and that's like what your parents worked on and it's finished but in actual big companies cloud is still only sort of a quarter to a third of Enterprise workflows even though cios of course always desperately hope that they'll be able to get the budget to double that in

three years time but somehow never managed to do so and we see something very similar here um for what CI think is going to happen to generative AI maybe a quarter of them think they'll have something deployed by the end of this year um another quarter of them think probably not until sorry they should say until 2026 or later um so it takes time to deploy this stuff it takes time to work out what you do with it it takes time to build it into products when you do that you have kind of a tidle

wave of new products so the SAS which as I said is still only sort of a quarter of Enterprise workflows now means that the typical big company department has got 50 to 75 applications and there the typical big company overall has four 500 SAS applications what all of those are doing is unbundling Excel or email or sap or Salesforce or Google and turning that into some workflow that they've managed to automate and we saw that with machine learning over the last 10 years we'll see it again with llms as people use this to unbundle some

task and automate it again hence classic quote from Jim barkdale there's two ways you can make money bundling and unbundling and that's what startups do now the counterargument to all of this what you might call the AI maximalist view is to say yes but what about the scaling so let's come back to this slide is this stuff going to keep scaling or not if the scaling keeps working then what happens well then it might be that the llms just become the whole thing and the llm sits on top and runs everything else and makes everything

else an API and you might have radically more stuff being done with software but without needing radically more more apps or more companies or more products you could just ask chat DPG to do it for you on the other hand if we've got we what we've got now is kind of ually where we're going to be then the llms are probably just going to be another API call you will build software you will use an llm to do a thing for it and that will be an API call just like storage or calculation or indeed

image recognition and that gets me to another Steve Jobs quote which is it's not actually the customer's job to know what they want it's not the customer's job to work out what to do with this and there's a kind of a challenge I think in giving everybody at Chevron and llm today which is that the classic way that we work out what to do with technology is that startups pick up the technology and invent use cases and invent problems and invent ways that you could do things that you could do with this but when you

give everybody in llm you're forcing the users to invent the use cases and work out what they're supposed to do with it which isn't really how Innovation Works now I think if you look at charts like this this is um startups going through y combinator this is a bet on there being thousands more companies and thousands more use cases and on llms being another API call because if chat GPT could do whole thing then you wouldn't need all these AI companies um now the extreme case here of course is that this just becomes a feature

and so on the right here we have a screenshot of Apple's um new AI powered writing tools um I look at this and I think this is spell check or this will become spell check rewrite this proofread it summarize it check the spelling check the grammar these just become features that disappear into the background and there's a progression here I think with AI that to begin with it's Ai and its amazing sexy and exciting and cool and then a little bit later it's just smart so it's smart suggest smart format smart recommend smart layout smart

summarize and a little bit later it's Auto so it's auto correct or Auto format and then it becomes just software it's just what's always been there so I think if we try and imagine some models for what products we do with this stuff might look like again I suggest kind of three categories the stuff that will be new features it will rewrite my email it will summarize the reviews it will suggest something it will make something there will be completely new tools and completely new capabilities and features just as we always have had for the

last 50 years and then there's this kind of maximalist view that says I can just go to chat gpt7 and say yeah so I'm moving to Singapore can you buy me a house and sort out the Visa and it will just do it and that's not quite science fiction but it's still definitely kind of an outlying view um hence if I kind of pull all of these threads together um I've spent a lot of the last 18 months talking to big companies and groups of people about Ai and got a lot of questions and it

seems to me that all questions about generative AI have one of two answers um the answer is either it will work just like every other platform shift or we don't know so should we buy from Google will Google go control the whole thing will there be any startups do we need our own National strategy for this um do we need our own National strategy for AI to me that's like saying do we need our own National strategy for SAS do we need our own National strategy for SQL that doesn't seem like the right way of

understanding it this is just another platform shift the other view set of questions are okay how much will the model scale what happens to error rates is there enough data how much energy will this need will we will we be able to have models that can train themselves continuously so all sorts of kind of science questions as to how good this will get where the answer is I don't know but neither does anybody else no one knows and we'll just have to kind of wait and find out in a couple of years meanwhile it is

kind of worth pointing out that all the stuff that we were excited about 2 years ago before chat GPT 3.5 launched is kind of still there and still happening and one of the ways I I tend to think about this is that the tech industry is always very excited by things that are going to happen in 2025 or 2030 so um two or 3 years ago that would have been crypto obviously no one here believed in crypto but there were other people who thought that crypto was going to be a big thing um and it

was going to be metaverse and it was going to be ar and VR now of course it's all generative AI um meanwhile most of the actual software industry is deploying ideas from 2010 and 2015 SAS Cloud automation workflows business Process Management collaboration Google docbot for video Google docbot for this we found this problem deep inside the HR departments in the finance industry and we' built a SAS company to solve it and that will be a billion dollar company that's what most actual software companies are doing um and then the rest of the economy is being

overturned by ideas from 2000 like kind of wild crazy bizarre ideas like maybe people watch TV on the internet um which was definitely a crazy idea in 2000 um and took 25 years to happen um so if we think about what some of those things look like firstly and I I did this chart last year metaverse actually is still doing metaverse meta is still doing metaverse um invested um$ 17.5 billion dollar in the last 12 months in the the reality Labs division which is massively more than Apple spent to get the iPhone out of the

door um but meanwhile there's a bunch of stuff other stuff that's still going on so this is kind of the the most boring chart in the tech industry like it went up one percentage point every year for 20 years except for that that little bit in the middle but that's just now 2025 per of retail um it's trillions of dollars of value um and when that happens all sorts of other things change so Shen is now almost certainly the world's largest apparel retailer aggregating Chinese manufacturers and shipping directly to the West $45 billion of gmv

last year their IPO prospectives will come out any day now and then we'll find out where they are today um Shopify um did over $150 billion of gmv last year which makes it about a third of the size of Amazon um and that's enabling T of thousands of small businesses to produce a best-in-class e-commerce experience Amazon itself now makes more money from advertising than almost anything else certainly more money than it makes from retail it did over $50 billion selling advertising on the amazon.com site in the last 12 months um the software industry is eating

television YouTube did $50 billion of advertising and subscription Revenue in the last 12 months which is bigger than any Global Media Company except Disney and if you were to subtract sport from this it would probably be bigger than Disney as well um software is also eating cars I'm old enough to remember when we were excited about autonomous cars um weo is actually doing 300,000 Robo track taxi trips a month um here so weo is actually kind of got something working maybe which is more than some other companies we we could mention are talking about the

interesting question though as we look at cars is that the car IND is going to go to electric and that's going to be powered by software but who is it that's going to do that is that going to be software companies or is it going to be a whole new generation of car companies how is this going to work and I think there's a kind of a generalized question here which is that technology changes all the parameters technology changes all your assumptions about how these industries work but then all the questions for that industry are

no longer Tech questions all the questions for TV now a TV industry questions all the questions for Shen are apparel questions is the Chinese car industry going to do what the Japanese car industry did in the ' 80s the this is a car industry question this is not a software industry question and so coming back to a quote I used earlier um say Larry Tesla is a pioneer of computer science and of AI said intelligence is whatever machines haven't done yet AI is whatever doesn't work yet and he's partly kind of making a philosophical Point

here that we always redefine intelligence as anything that machines do because if machines can do it it can't be intelligent but what he was also saying here is that as soon as something works people say well that's not AI That's Just software in the 1970s databases were AI now they're just software 10 years ago image recognition was AI I don't think image recognition is AI anymore I think image recognition is now just software the same with voice recognition the same with patent recognition and so by default the process we're going to go through in the

next couple of years is that large language models again will just be software they won't be intelligence they'll just be something that computers do um and so I think you could kind of propose a more General version of this C is that that technology indeed is whatever machines have haven't done yet and so I'll finish with one final chart which I will use a great deal because it took me hours to type this data in and I couldn't get chat GPT to do it for me um this is the number of people employed in the

USA as elevator attendant um to begin with there are more elevators so you need more elevator attendance then the elevators get automated and you don't need that and so I'll ask you a question when is the last time you used an automatic elevator when's the last time you got into an elevator and pressed a button and said ah I'm using an AI elevator today in fact an electronic automatic elevator or was it just an elevator and that's just the way the world's always been that's probably how all of this is going to evolve as well

and with that I will say thank you