Mustafa Suleyman & Yuval Noah Harari -FULL DEBATE- What does the AI revolution mean for our future?

[Music] historian Yuval Noah Harari and entrepreneur Mustafa Suleiman are two of the most important voices in the increasingly contentious debate over AI good to be here joining us thanks for having us the economists got them together to discuss what this technology means for our future from employment and geopolitics to the survival of liberal democracy if the economic system has fundamentally changed will liberal democracy as we know it survive [Music] you well know Harare welcome you are a best-selling author historian I think a global public intellectual if not the global public intellectual your books from sapiens

to 21 lessons from the 21st century have sold huge numbers of copies around the world thank you for joining us it's good to be here Solomon wonderful that you can join us too you're a friend of The Economist a fellow director on The Economist board you are a man at The Cutting Edge of creating the AI Revolution you are a co-founder of deepmind you're now a co-founder and CEO of inflection AI you are building this future but you've also just published a book called The Coming wave which makes us a little concerned about this revolution

that is being Unleashed you're both coming from different backgrounds you are a historian a commenter a man who I believe doesn't use smartphones very much not very much no Mustafa as I know from our board meetings is right at The Cutting Edge of this pushing everyone to go faster so two very different perspectives so but I thought it would be really interesting to bring the two of you together to have a conversation about what is happening what is going to happen what are the opportunities but also what is at stake and what are the risks

so let's start Mustafa with you um and you are building this future so paint us a picture of what the future is going to be like and I'm going to give you a time frame to keep it specific so let's say I think you wrote in your book that within three to five years that you thought it was plausible that AIS could have human level capability across a whole range of things so let's take five years 2028 what does the world look like how will I interact with AIS what will we all be doing and

not doing well let's just look back over the last 10 years to get a sense of the trajectory that we're on and the incredible momentum that I think everybody can now see with the generative AI Revolution over the last 10 years we've become very very good at classifying information we can understand it we sort it label it organize it and that classification has been critical to enabling this next wave because we can now read the content of images we can understand text pretty well we can classify audio and transcribe it into text the machines can

now have a pretty good sense of the conceptual representations in those ideas the next phase of that is what we're seeing now with the generative AI Revolution we can now produce new images new videos new audio and of course new language and in the last year or so with the rise of chat GPT and other AI models it's pretty incredible to see how plausible and accurate and very finesse to these new language models are in the next five years the Frontier Model companies those of us at the very Cutting Edge who are training the very

largest AI models are going to train models that are over a thousand times larger than what you currently see today in GPT 4 all and with each new order of magnitude and compute that is 10x more compute used we tend to see really new capabilities emerge and we predict that the new capabilities that it will come this time over the next five years will be the ability to plan over multiple time Horizons instead of just generate new text in a one shot the model will be able to generate a sequence of actions over time and

I think that that's really the character of AI that we'll see in the next five years artificial capable AIS AIS that can't just say things they can also do things but what does that actually mean in practice just just use your imagination tell me what my life will be like in 2028 how will I interact with them what will I do what will be different so I've actually proposed a modern Turing test which tries to evaluate for exactly this point right the last Turing test simply evaluated for what a machine could say assuming that what

it could say represented its intelligence now that we're kind of approaching that moment where these AI models are pretty good arguably they've passed the Turing test or they maybe they will in the next few years the real question is how can we measure what they can do so I've proposed a test which involves them going off and taking a hundred thousand dollar investment and over the course of three months trying to set about creating a new product researching the market seeing what consumers might like generating some new images some blueprints of how to manufacture that

product contacting a manufacturer getting it made negotiating the price drop shipping it and then ultimately correct collecting the revenue and I think that over a five-year period it's quite likely that we will have an ACI an artificial capable intelligence that can do the majority of that task autonomously it won't be able to do the whole thing there are many tricky steps along the way but significant portions of that it will be able to make phone calls to other humans to negotiate it'll be able to call other AIS in order to establish the right sequence in

a supply chain for example and of course it will learn to use apis application programming interfaces so other websites or other knowledge bases or other information stores and so you know the world is your oyster you can imagine that being applied to many many different parts of our economy so you vote a man who doesn't use a smartphone very much you listen to this does this fill you with horror or and do you agree with it do you think that's the kind of thing that is likely to happen in the next five years I will

take it very seriously I don't know I'm not coming from within the industry so I cannot comment on how How likely it is to happen but when I hear this as a historian for me what we just heard this is the end of human history not the end of History the end of human dominated history history will continue with somebody else in control because what we just heard is basically Mustafa telling us that in five years they'll be a technology that can make decisions independently and that can create new ideas independently this is the first

time in history we confronted something like this every previous technology in history from a stone knife to nuclear bombs it could not make decisions like the decision to drop the bomb on Hiroshima was not made by the atom bomb it was made by President Truman and similarly it can every previous technology in history It could only replicate our ideas like radio of the printing press it could make copies and disseminate the music or the poems or the novels that some human wrote now we have a technology that can create completely new ideas and it can

do it at a scale far beyond what humans are capable of so it can create new ideas and in important areas within five years we'll be able to enact them and that is a profound shift before we go on to the many ways in which this could be the end of human history as you put it and the the potential downsides and risks of this can we just for a second just indulge me I'm an optimist at heart can we talk about the possibilities what are the potential upsides of this because there are many and

they are really substantial I think you you wrote that it that there are there is the potential that this technology can help us deal with incredibly difficult problems and and create tremendous honestly positive outcomes so can we just briefly start with that before we go down down the road wasn't the end of human history again I'm not I'm not talking necessarily about the destruction of humankind or anything like that there are many positive potential it's just that control Power is Shifting away from human beings to an alien intelligence to a non-human intelligence we'll also get

to that because there's a question of how much power but let's stick with the potential upsides first the opportunities Mustafa everything that we have created in human history is a product of our intelligence our ability to make predictions and then intervene on those predictions to change the course of the world is in a very abstract way the way we have produced our companies and our products and all the value that has changed our Century I mean if you think about it just a century ago a kilo of grain would have taken 50 times more labor

to produce than it does today that efficiency which is the trajectory you have seen in agriculture is likely to be the same trajectory that we will see in intelligence everything around us is a product of intelligence and so everything that we touch with these new tools is likely to produce far more value than we've ever seen before and I think it's important to say these are not autonomous tools by default these these capabilities don't just naturally emerge from the models we attempt to engineer capabilities and the challenge for us is to be very deliberate and

precise and careful about those capabilities that we want to emerge from the model that we want to build into the model and the constraints that we build around it it's super important not to anthropomorphically project ideas and you know potential intentions or potential agency or potential autonomy into these models the governance challenge for us over the next couple of decades to ensure that we contain this wave is to ensure that we always get to impose our constraints on the development of this traject the the trajectory of this development but the capabilities that will arise will

mean for example potentially transformative improvements in human health speeding up the process of innovation dramatic changes in the way scientific discovery is done tough problems whether it's climate change a lot of the big challenges that we Face could be much more easily addressed with this capability everybody is going to have a personal intelligence in their pocket a smart and capable Aid a chief of staff a research assistant constantly prioritizing information for you putting together the right synthesized nugget of knowledge that you need to take action on at any given moment and that for sure is

going to make us all much much smarter and more capable does that part of it sound appealing to you absolutely I mean again if there was no positive potential we wouldn't be sitting here nobody would develop it nobody would invest in it it's again it's so appealing the positive potential is so enormous in everything again from much better Healthcare higher living standards solving things like climate change this is why it's so tempting this is why we are willing to take the enormous risks involved I I'm just worried that uh in the end the deal will

not be worth it and I would comment especially on again the notion of intelligence um I think it's overrated I mean Homo sapiens at present is the most intelligent entity on the planet it simultaneously also the most destructive entity on the planet and in some ways also the most stupid entity on the planet the only entity that that puts the very survival of the ecosystem in danger so you think we are trading off more intelligence with more destructive risk Yes again it's it's it's not uh it's not deterministic I I don't think that we are

doomed I mean if I thought that what's the point of talking about it if we can't prevent the worst case scenario well I was hoping you thought you'd have some agency in actually effectively we still have agency there are a few more years I don't know how many 5 10 30 we still have agency we are still the ones in the driver's seat shaping the direction this is taking no technology is deterministic this is something again we learned from history you can use the same technology in different ways you can decide which way to develop

it so we still have agency this is why you have to think very very carefully about what we are developing well thinking very carefully about it is something that Mustafa has been doing in this book um and I want to now go through some of the most commonly discussed risks and I I was trying to work out how I would go in sort of order of Badness so I'm starting with one that is discussed a lot but relative to human extinction is perhaps less bad which is the question of jobs and will you know artificial

intelligence essentially destroy all jobs because AIS will be better than humans and everything you know I'm an economist by training I you know history suggests to me that that has never happened before that the lump of Labor fallacy indeed is a fallacy but tell me what you think about that do you think there is a risk to jobs it depends on the time frame so over a 10 to 20 year period my intuition and you're right that so far the evidence doesn't support this is that there isn't really going to be a significant threat to

jobs there's plenty of demand there will be plenty of work right over a 30 to 50 year time Horizon is very difficult to speculate I mean at the very least we can say that two years ago we thought that these models could never do empathy we said that we humans were always going to preserve kindness and understanding and care for one another as a special skill that humans have four years ago we said while AIS will never be creative you know humans will always be the creative ones inventing new things making these amazing leaps between

new ideas is self-evident now that both of those two capabilities are things that these models do incredibly well and so I think for a period of time ai's augment our skills they make us faster more efficient more accurate more creative more empathetic and so on and so forth over a many decade period it's much harder to say what are the set of skills that are the permanent Preserve of the human species given that these models are clearly very very capable and that's where the containment challenge really comes in we have to make decisions we have

to decide as a species what is and what isn't acceptable over a 30-year period and that means politics and governance with regard to jobs I agree that like the the scenario that there just won't be any jobs this is an unlikely scenario right in in the at least next few decades but we have to look more carefully at time and space I mean in terms of time the transition period is is is the danger I mean some jobs disappear some jobs appear people have to transition just remember that Hitler Rose to power in Germany because

of three years of 25 unemployment so we are not talking about say no jobs at all but if because of the upheavals caused in the job market by AI we have like I don't know three years of 25 unemployed unemployment this could cause huge social and political disruptions and then the even bigger issue is one of space that uh The Disappearance of jobs and the new jobs will be created in different parts of the world so we might see a situation when there is immense demand for more jobs in California or Texas or China whereas

entire countries lose their uh their economic basis so you need a lot more computer engineers and yoga trainers and whatever in California but you don't need any textile workers at all in Guatemala or Pakistan because this has all been automated so it's not just the total number of jobs on the planet it's the distribution between different countries and let's also try to remember that work is not the goal work is not not our desired end State we did not create civilization so that we could have full employment we created civilization so that we could reduce

suffering for everybody and the Quest for abundance is a real one we have to produce more with less there is no way of getting rid of the fact that population growth is set to explode over the next Century there are practical realities about the demographic and Geographic and climate trajectories that we're on which are going to drive forward our need to produce exactly these kinds of tools and I think that that should be an aspiration many many people do work that is judging us and exhausting and tiring and they don't find flow they don't find

their identity and it's pretty awful so I think that we have to focus on the prize here which is one of a question of capturing the value that these models will produce and then thinking about redistribution and ultimately the transition is exactly what's at stake we have to manage that transition with taxation but just with redistribution I would say that the difficulty again the political historical difficulty I think there will be immense New Wealth created by by these Technologies I'm less sure that the governments will be able to redistribute this wealth in a fair way

on a global level like I just don't see the US government raising taxes on corporations in California and sending the money to help unemployed textile workers in Pakistan or Guatemala kind of retrain to for the new job market well that actually gets us to the second potential risk which is the risk of AI to the political system as a whole and you made a very um good point you are in one of your writings where you reminded us that liberal democracy was really born of the Industrial Revolution and that today's political system is really a

product of the economic system that we are in and so there is I think a very good fair question of if the economic system is fundally fundamentally changed will liberal democracy as we know it survive yeah and on top of that it's not just the Industrial Revolution it's the new information Technologies of the 19th and 20th Century before the 19th century you don't have any example in history of a large-scale democracy I mean you have examples on a very small scale like in hunter gatherer tribes or in city-states like ancient Athens but you don't have

any example that I know of of millions of people spread over a large territory an entire country which managed to uh build and maintain a democratic system why because democracy is a conversation and there was no information technology in communication technology that enabled a conversation between millions of people over an entire country only when first newspapers and then Telegraph and radio and television came along this was this became possible so modern democracy as we know it it's built on top specific information technology once the information technology changes it's an open question whether the market obviously

can survive and the biggest danger now is the opposite than what we face in the Middle Ages in the Middle Ages it was impossible to have a conversation between millions of people because they just couldn't communicate but in the 21st century something else might make the conversation impossible if trust between people collapses again if AI if you go online which is now the main uh way we converse on the level of a country and the online space is flooded by non-human entities that maybe masquerade as human beings you talk with someone you have no idea

if it's even human you see something you see a video you hear an audio you have no idea if this is really a is this true is this fake is this a human it's not a human I mean in this situation unless we have some guard rails again conversation collapses is that what you mean when you say AI risks hacking the operating system this is one of the things again if if if Bots can impersonate people it's it's basically like what happens in in the financial system like people invented money and it was possible to

counterfeit money to create fake money the only way to save the financial system from collapse was to have very strict regulations against fake money because the technology to create fake money was always there so but there was very strict regulation against it because everybody knew if you allow fake money to spread the financial system the Trust In money collapses and now we are in the analogous situation with uh the political conversation that now it's possible to create fake people and if we don't ban that then trust will collapse we'll get to the Banning or not

Banning in a minute democratizing access to the right to broadcast has been the story of the last 30 years hundreds of millions of people can now create podcasts and blogs and they're free to broadcast their thoughts on Twitter and the internet broadly speaking I think that has been an incredibly positive development you no longer have to get access to the top newspaper or you get the skills necessary to be part of that institution many people at the time feared that this would destroy our credibility and Trust in the big news outlets and institutions I think

that we've adapted incredibly well yes it has been a lot of turmoil and unstable but with every one of these new waves I think we adjust our ability to discern truth to dismiss nonsense and there are both Technical and governance mechanisms which will emerge in the next wave which we can talk about to address things like bot impersonation I mean I'm completely with you I mean we should have a ban on impersonation of digital people it shouldn't be possible to create a digital zany and have that be platformed on Twitter talking all kinds of nonsense

enough with the real world so I think that there are technical mechanisms that we can do to prevent those kinds of things and that's why we're talking about them there are mechanisms we just need to employ them I I would say two two things first of all it's it's a very good thing that more people were given a voice it's diff different with Bots Bots don't have freedom of speech so Banning Bots because they shouldn't have freedom of speech they shouldn't have that's very important yes uh there have been some wonderful developments in the last

30 years still I'm very concerned that when you look at countries like the United States like the UK to some extent like my home country of Israel I'm struck by the fact that we have the most sophisticated information technology in history and we are no longer able to talk to each other that my impression maybe your impression of American politics or politics in other democracies is different my impression is that trust is collapsing the conversation is collapsing that people can no longer agree who won the last elections like the most basic fact in a democracy

who won the last it's it's we had huge disagreements before but I feel that now it's different that really the conversation is breaking down I'm not sure why but it's it's really troubling that at the same time that we have the really the most powerful information technology in history and people have no longer can talk with each other it's a very good point we we actually had a you may have seen it we had a big cover package on looking at what the impact might be in the short term on elections and on the political

system and we concluded actually AI was likely to have a relatively small impact in the short term because there was already so little trust um so it was a sort of double-edged uh answer you know it was it was not going to make a huge difference but only because things were pretty bad as they were but you both said there needs to be regulation um before we get to the precisely how the unit that we have that would do that is the nation-state and National governments yet you Mustafa in your book worry that actually one

of the potential um dangers is that the powers of the nation-state are eroded could you talk through that as the sort of the third in my escalating sense of risks the challenge is that at the very moment when we need the nation state to hold us accountable the nation-state is struggling under the burden of a lack of trust and huge polarization and a breakdown in our political process and so combined with that the latest models are being developed by the private companies and by the open source it's important to recognize it isn't just the biggest

AI developers there's a huge proliferation of these techniques widely available on open source code that people can download from the web for free and they're probably about a year or a year and a half behind the absolute Cutting Edge of the big models and so we have this dual challenge like how do you hold centralized power accountable when the existing mechanism is basically a little bit broken and how do you address this Mass proliferation issue when it's unclear how to stop anything in Mass proliferation on the web that's a really big challenge what we've started

to see is self-organizing initiatives on the part of the companies right so getting together and agreeing to sign up proactively to self oversight both in terms of audit in terms of capabilities that we won't explore etc etc now I think that's only partially reassuring to people clearly maybe not even reassuring at all but the reality is I think it's the right first step given that we haven't actually demonstrated the large-scale harms to arise from AIS just yet I mean this is one of the first occasions I think in general purpose waves of technology that we're

actually starting to adopt a precautionary principle I'm a big advocate of that I think that we should be approaching a Do no harm principle and that may mean that we have to leave some of the benefits on the tree and some fruit may just not be picked for a while and we might lose some gains over a couple of years where we may look back in hindsight and think oh well we could have actually gone a little bit faster there I think that's the right trade-off this is a moment of caution things are accelerating extremely

quickly and we can't yet do the balance between the harms and benefits perfectly well until we see how this wave unfolds a little bit so I like the fact that our company inflection Ai and the other big developers are trying to take a little bit more of a cautious approach I think that's a really interesting point because you know we are having this conversation you have written both of you you extensively about the challenges posed by this technology there's now a parlor game amongst you know practitioners in this world about you know what is the

risk of extinction level events where there's a huge amount of talk about this and I don't know in fact I should probably ask you what percentage of your time probably right now it's you know close to 100 of your time is focused on the risk since you're promoting your book but it's it is there's a lot of attention on this which is which is good um we are thinking about it early so that gets us I think now to the most important part of our conversation which is what do we do and you Mustafa you

lay out a 10-point plan which is you know the kind of action do kind of thing that uh that someone who doesn't just comment like you and I do but actually does things we do so tell us what do we need to do as as Humanity as governments as societies to ensure that we capture the gains from this technology but we minimize the risks there are some very practical things I mean so for example red teaming these models means adversarially testing them and trying to put them under as much pressure as possible to push them

to generate advice for example on how to generate a biological or chemical weapon how to create a bomb for example or even push them to be very sexist racist biased in some way and that already is pretty significant we can see their weaknesses I mean part of the release of these models in the last year has given everybody I think the opportunity to see not just how good they are but also their weaknesses and that is reassuring we need to do this out in the open that's why I'm a huge fan of the open source

Community as it is at the moment because real developers get to play with the models and actually see how hard it is to produce the capabilities that sometimes I think we fear that they're just going to be super manipulative and persuasive and you know destined to be awful so that's the first thing is doing it out in the open the second thing is that we have to share the best practices and so there's a competitive tension there because safety is going to be an asset you know I'm gonna deliver a better product to my consumers

if I have a safer model but of course there's got to be a requirement that if I discover a vulnerability a weakness in the model then I should share that just as we have done for actually decades in many waves of Technology not just in software security for example but in Flight Aviation you know the Black Box recorder for example if there's a significant incident not only does it record all the Telemetry on board the aircraft but also everything that the pilots say in the cockpit and if there's a significant safety incident then that's shared

all around the world with all of the competitors which is great aircrafts are one of the safest ways to get around despite you know on the face of it if you described it to an alien being 40 000 feet in the sky is a very strange thing to do so I think there's precedent there that we can we can follow um I do also agree that is probably time for us to explicitly declare that we should not allow these tools to be used for electioneering I mean we cannot trust them yet we cannot trust them

to be stable and reliable we cannot allow people to be using them for counterfeit digital people and clearly we've talked about that already so there are some capabilities which we can start to take off the table another one would be autonomy right right now I think autonomy is a pretty dangerous set of methods it's exciting it represents a possibility that could be truly incredible but we haven't wrapped our hands around what the risks and limitations are likewise training an AI to update and improve its own code this notion of recursive self-improvement right closing the loop

so that the AI is in charge of defining its own goals acquiring more resources updating its own code with respect to some objective these are pretty dangerous capabilities just as we have kyc know your customer or just as we license development developers of nuclear technologies and all the component involved in that supply chain there'll be a moment where if some of the big technology you know providers want to experiment with those capabilities then they should expect there to be robust audits you know they should expect them to be licensed and there should be independent oversight

so how do you get that done and there seem to be there is there are several challenges in doing it one is the division between the relatively few Leading Edge models of which you have won and then the larger tale of Open Source models where the you know the ability to build the model is decentralized lots of people have access to it my sense is that the capabilities of the latter are a little bit behind the capabilities of the former but they are growing all the time and so if you have really considerable open source

capability what is not to stop the angry teenager in some small town developing capabilities that could shut down the local hospital and how do you in your regulatory framework guard against that well look part of the challenge is that these models are getting smaller and more efficient and we know that from the history of Technologies anything that is useful and valuable to us gets cheaper easier to use and it proliferates far and wide so the destiny of this technology over a two three four decade period has to be proliferation and we have to confront that

reality it isn't a contradiction to name the fact that proliferation seems to be inevitable but containing centralized power is an equivalent challenge so there is no easy answer to that I mean Beyond surveilling the internet it is pretty clear that in 30 years time like you say garage tinkerers will be able to experiment if you look at the trajectory on synthetic biology we now have have desk desktop synthesizers that is the ability to engineer new synthetic compounds they cost about twenty thousand dollars and they basically enable you to create potentially molecules which are you know

more transmissible or more lethal than we had with covid you can basically experiment and the challenge there is that there's no oversight you buy it off the shelf you don't need a great deal of training probably an undergraduate in biology today and you'll be able to experiment now of course they're going to get smaller easier to use and spread far and wide and so my book I'm really trying to popularize the idea that this is the defining containment challenge of the next few decades so you use the word containment which is interesting because you know

I'm sure the word containment with you brings immediately you know inspires images of George Cannon and and you know the post-war Cold War Dynamic and we're now you know we're in a geopolitical world now that whether or not you call it a new cold war is one of great tension between the US and China can this kind of containment as as Mustafa calls it be done when you have the sort of tensions you've got between the world's big players are the you know is the right Paradigm thinking about the arms control treaties of the Cold

War like how do we go about doing this at a kind of international level I think this is the biggest problem that if it was a question of you know humankind versus a common threat of these new intelligent alien agents here on Earth then yes I think there are ways we can contain them but if the humans are divided among themselves and are in an arms race then it's because it becomes almost impossible to contain this alien intelligence and and there is I I'm tending to think of it more in in terms of of really

an alien invasion that like somebody coming and telling us that you know there is a fleet an alien Fleet of spaceships coming from planet Zircon or whatever with with highly intelligent beings they'll be here in five years and take over the planet maybe they'll be nice maybe they'll solve cancer and climate change but we are not sure this is what we are facing except that the aliens are not coming in spaceships from planet Zircon that are coming from the Laboratories the actual characterization of the nature of the technology an alien has by default agency these

are going to be tools that we can apply we have narrow settings yes but let's say they have they potentially have agency we can try to prevent them from having agency but we know that they are going to be highly intelligent and at least potentially have agency and this is a very very frightening mix something we never confronted before again atom bombs didn't have a potential for agency printing presses did not have a potential for agency this thing again unless we contain it and the problem of content is very difficult because potentially they'll be more

intelligent than us how do you prevent something more intelligent than you from become from developing the agency it has I'm not saying it's impossible I'm just saying it it's very very difficult I think our best bet is not to kind of think in terms of some kind of rigid regulation you should do this you shouldn't do that it's in developing new institutions living institutions that are capable of understanding the very fast developments and reacting on the fly at present the problem is that the only institutions who really understand what is happening are the institutions who

develop the technology the governments most of them seem quite clueless about what's happening also universities I mean the amount of talent and the amount of the the economic resources in the private sector is far far higher than in the universities so and again I'm I appreciate that there are actors in the private sector like Mustafa who are thinking very seriously about regulation and containment but we must have an external entity in in the game and for that we need to develop new institutions that will have the human resources that will have the the economic and

technological resources and also will have the public trust because without public trust it won't work are we capable of creating such new institutions I don't know I do think Eva Rays is an important point which is as we started this conversation and you were painting the picture of five years time and you were saying that the AIS would be ubiquitous we'd all have our own ones but that they would have the capability to act not just to process information they would have the creativity they have now and the ability to act but already from these

generative AI models the power that we've seen in the last year two three four years has been that they have been able to act in ways that you and your other your fellow technologists didn't anticipate they they reached you know you didn't anticipate you know the the speed with which they would you would Win It Go or so forth there was a the Striking thing about them is that they have developed in unanticipatedly fast ways so if you combine that with capability you don't have to go as far as Yuval is saying and saying that

they're all more intelligent than humans but there is an unpredictability there that I think does raise the concerns that Uval raises which is you their creators can't quite predict what powers they will have they may not be fully autonomous but they will be moving some ways towards there and so how do you guard against that or how do you you know red teaming you use the phrase which is that I understand it is that you know you keep checking what's happening and tweak them when you've seen what's when you pressure test them you try to

make them fit you can't pressure test for everything in advance so there is a I think a very real point that Yuval is making about as the capabilities increase so the risks increase of relying on you and other Creator companies to to make I mean it's a very fair question and that's why I've long been calling for the precautionary principle we should both take some capabilities off the table and classify those as high risk I mean frankly the EU AI act which has been in draft for three and a half years is very sensible has

a risk-based framework that applies to each application domain whether it's Healthcare or self-driving or facial recognition and it basically takes certain capabilities off the table when that threshold is exceeded I listed a few earlier autonomy for example it's clearly a capability that it has the potential to be high risk recursive self-improvement the same story so this is the moment when we have to adopt a precautionary principle not through any fear-mongering but just as a logical sensible way to proceed another model which I think is very sensible is to take an ipcc style approach an international

consensus around an investigatory power to establish the scientific fact basis for where we are with respect to capabilities and that has been an incredibly valuable process set aside the negotiation and the policy making just the evidence observing where are we you don't have to take it from me you should be able to take an independent panel of experts who I would personally Grant access to everything in my company if they were a trusted true impartial actor without question we would Grant complete access and I know that many of the other companies would do the same

again people are drawn towards the kind of of scenario of the AI creates a lethal virus Ebola plus kovid and kills everybody let's go in the more Economist Direction Financial systems like you gave as a new touring test the idea of AI making money what's wrong with making money wonderful thing so let's say that you have an AI which has a better understanding of the financial system than most humans most politicians maybe most Bankers and uh let's think back to the 2007-2008 financial crisis it started with this I was about they called CDO cdus this

is exactly something that these genius mathematicians invented nobody understood them except for a handful of Genius mathematicians in Wall Street which is why nobody regulated them and almost nobody saw the financial crash coming what happens again this kind of of apocalyptic scenario which you don't see in Hollywood science fiction movies the AI invents a new class of financial devices that nobody understands it's beyond human capability to understand it's such complicated math so much data nobody understands it it makes billions of dollars billions and billions of dollars and then it brings down the world economy and

no human being understand what the hell is happening like the prime ministers the presidents the the financial ministers what what is happening and again this is not fantastic I mean we saw it with human mathematicians in 2007-8 I think that's that look that's one you know you you can easily paint paint pictures here that make you want to jump off the nearest cliff and you know that's that's one but actually my other response to mustafa's laying out of where you say well we just need to rule out certain actions is to go back to the

geopolitics is it sensible for a country to rule out certain capabilities if the other side is not going to rule them out so you have a you have a kind of political economy problem going down the road that you learn we this is a moment when we uh collectively in the west have to establish our values and stand behind them what we cannot have is a race to the bottom that says just because they're doing it we should take the same risk if we adopt that approach and cut Corners left right and Center we'll ultimately

pay the price and that's not an answer to well they're going to go off and do it anyway we've said only seen that with lethal autonomous weapons I mean there's been a negotiation in the U.N to regulate lethal autonomous weapons for over 20 years and they barely reached agreement on the definition the definition of lethal autonomous weapons let alone any consensus so that's not great but we do have to accept that it's the inevitable trajectory and from our own perspective we have to decide what we're prepared to tolerate in society with respect to free acting

AIS facial surveillance facial recognition and you know generally autonomous systems I mean so far we've taken a pretty cautious approach when we don't have drones flying around everywhere we can already it's totally possible technically to autonomously fly a drone to navigate around London we've we've ruled it out right we don't yet have autonomous self-driving cars even though you know with some degree of harm they are actually pretty well functioning so the regulatory process is also a cultural process of what we think is socially and politically acceptable at any given moment and I think an appropriate

level of caution is is what we're seeing much but I completely agree on that that we need in many fields the Coalition of the willing and if some actors in the world don't want to join it's it's in our interest so again something like Banning Bots impersonating people so some countries will not agree but that doesn't matter to protect our societies it's still a very good idea to have these kinds of regulations so that area of agreement is one to bring us to a close but I want to end by asking both of you and

use first Mustafa you are you know both raising alarms but you are heavily involved in creating this future why do you carry on I personally believe that it is possible to get the upsides and minimize the downsides in the AI that we have created Pi which stands for personal intelligence is one of the safest in the world today it doesn't produce the racist toxic bias greeds that they did two years ago it doesn't fall victim to any of the jailbreaks The Prompt hacks the adversarial red teams none of those work and we've made safety an

absolute number one priority in the design of our product so my goal has been to do my very best to demonstrate a path forward in the best possible way this is an inevitable unfolding over multiple decades this really is happening the coming wave is coming and I think my contribution is to try to demonstrate in the best way that I can a manifestation of a personal intelligence which really does adhere to the best safety constraints that we could possibly think of so you've all you've you've heard mustafa's explanation for why he continues you look back

over human history now as you look forward is this a technology and a pace of innovation that Humanity will come to regret or should Mustafa carry on it could be gonna I can't predict the future I would say that we invest so much in developing artificial intelligence and we haven't seen anything yet like it's it's still the very first baby steps of artificial intelligence in terms of like you think about I don't know the evolution of organic life this is like the amoeba of artificial intelligence and it won't take millions of years to get to

T-Rex maybe it will take 20 years to get to T-Rex and but one thing to remember is that we also our own minds have a huge scope for development uh also with Humanity we haven't seen our full potential yet and if we invest for every dollar and minute that we invest in artificial intelligence we invest another dollar a minute in developing our own Consciousness our own mind I think we'll be okay but but I don't see it happening I don't see this kind of investment in in human beings that we are seeing in in the

machine well for me this conversation with the two of you has been just that investment thank you both very much indeed thank you thank you thank you foreign foreign foreign