OpenAI’s Deep Research Team on Why Reinforcement Learning is the Future for AI Agents

a lesson that I've seen people learn over and over again in this field is like you know we we think that we can do things that are smarter than what the models do by writing it ourselves but as the field progresses the models come up with um better solutions to things than humans do the like probably like number one lesson of machine learning is like you get what you optimize for and so um if you if you're able to set up the system such that you can optimize directly for the outcome that you're looking for

um the results are going to be much much better than if you sort of try to glue together models that are not optimized end end for for the the tasks that you're trying to have them do so my like like long-term guidance is that um you know I think like reinforcement learning um tuning on top of models is probably going to be a critical part of how the most powerful agents get built [Music] we're excited to welcome Issa fulford and Josh Tobin who lead the Deep research product at openai Deep research launched 3 weeks ago

and has quickly become a hit product used by many Tech luminaries like the cisin for everything from industry analysis to medical research to birthday party planning deep research was trained using endtoend reinforcement learning on hard browsing and reasoning tasks and is the second product in a series of agent lunches from openai with the first being operator we talked to ISA and Josh about everything from Deep researchers use cases to how the technology Works under the hood to what we should expect in future agent lunches from open AI Issa and Josh welcome to the show thank

you thank you so much for joining us excited to be here thank you for having us um so maybe let's start with like what is deep research tell us about the origin stories and what this product is doing so deep research is a agent that is able to search many online websites and it can create very comprehensive reports um it can do tasks that would take humans many hours to complete and it's in chat gbt and it takes like 5 to 30 minutes to to answer you and so it's able to do much more in-depth

research and answer your questions with much more detail um and specific sources than regular chat gbt response would be able to do it's one of the first agents that we've released so we released operator um pretty recently as well and so um deep research is the second agent and you know we'll release many more in um in future what's the origin story behind deep research like when did you choose to do this what was the inspiration and how many people work on it like what what did it take to bring this to poition good question

this is before my time so to hear yeah so I think um maybe around a year ago we were seeing a lot of success internally with um this new reasoning Paradigm and training uh models to think before responding and we were focusing a lot on um Math and Science domains but I think that that the other thing that this kind of new reasoning model um regime unlocks is the ability to do longer Horizon tasks that involve like AG gench kind of you know abilities and so we thought you know a lot of people do tasks

that require a lot of online research or a lot of external context and that involves a lot of reasoning and discriminating between sources and you have to be quite creative to do those kinds of things and I think we finally had models or a way of training models um that would allow us to to be able to um tackle some of those tasks so we decided to try and start training models to um do first browsing tasks so using like the same methods that we used to train reasoning models but on more real world tasks

was it your idea and Josh how did you get involved at first uh it was like me and Yos ptil um who as our opening ey is working on a a similar um project that will be released at some point which we very excited about um and we we built an original um and then also with Thomas Dimson who's one of those people who just is an amazing engineer like will dive into anything and just you know get loads of things on so it was very fun yeah and I I joined more recently I uh

rejoined opening I about six months ago um uh from my startup I was uh out opening ey in the early days and um was looking around the projects um when I rejoined and got very interested in some of our HMT efforts including this one and uh and got involved through that amazing well tell us a little about who you built it for yeah I mean it's it's really for anyone who does knowledge work as part of their um as part of their day-to-day job or really as part of their life um so we're seeing uh

a lot of the usage come from people using it for work um doing things like uh you know research as part of their jobs um for uh you know understanding U markets companies uh real estat a lot of scientific research medical I think we've seen a lot of U medical examples as well um and and one of the things we're really excited about as well is um you this this style of like uh I just need to go out and spend many hours doing something that um you know where I have to do a bunch

of web searches and Cate a bunch of information is not just a work thing but it's also um useful for shopping and uh travel as well so we're excited for the the plus launch so that more people will be able to try deep research and maybe we'll see some new use cases as well it's definitely one of the products I've used the most over the last couple weeks it's been amazing using it for work for work definitely also for fun like what are you using it for oh for me oh my goodness um so I

was thinking about buying a new car and I was trying to figure out when the next model was going to be released for the car and there's all these speculative blog posts like there's patterns from the manufacturer and so I asked deep research can you break down all the gossip about this car and then all of the facts about what they've done what this automakers in before and it put together an amazing report that told me maybe wait a couple months but this year like in the next few months it should come out yeah like

one of the things that's really cool about it is it's it's uh like it's not just for going Broad and Gathering all of the information about a source but it's also really good at finding like very obscure like weird facts on the internet um like if you have something very specific you want to know that you might might not just turn up in the first page of search results it's good at that kind of thing too so that's cool what are some of the surprising use cases that you've seen o I think the thing I've

been most surprised by is how many people are using it for coding yeah um which wasn't really a use case i' considered but I've seen a lot of people on Twitter and in various places where we get feedback using it for coding and code search and also for finding the latest documentation on a certain package or something and helping them write a script or something so yeah I'm like I'm kind of embarrassed that we didn't think of that as a use case because it's like you know for chbt users it seems so obvious but um

it's I know it's impressive how well it works how do you think the balance of business versus individual use case will will evolve over time like you mentioned the plus launch that's happening you know in a year's time or two years time would you guess this is mostly a business tool or mostly a consumer tool I would say hopefully both uh I think it's a pretty General um capability which and I think it's something that we do both in work and in personal life so yeah I'm excited about both I think the the magic of

it is like um it just saves people a lot of time um you know if there's uh something that might have taken you hours or in some cases we've heard like days um people can just put it in here and get you know 90% of what they would have come out up with on their own um and so yeah I I tend to think there's like there's more tasks like that in uh business than there are in personal but I mean I think for sure it's going to be part of people's lives and both yeah

it's really become the majority of my uses for CHC I just always pick deep research rather than normal so what are you saying in terms of consumer use cases and what are you excited about I think a lot of shopping travel recommendations um I've personally used the model a lot I've been using it for months to do these kinds of things um we were in Japan for the for the launch of deep research so it was very helpful and finding restaurants with very specific requirements and finding things that I wouldn't have like necessarily found yeah

and I found it like when you have um something it's like the kind of thing where you know if you're shopping maybe for something expensive or you're planning a trip that uh is special or you want to spend a lot of uh uh that you're you want to spend a lot of time thinking about it's like for me you know I might go and spend hours and hours like trying to read everything on the internet about this one this product that I'm interested in buying um like scouring all the reviews and uh the forums and

stuff like that and deep research can put together um kind of like something like that very quickly and so it's it's really useful for that that kind of thing the model is also very good at instruction following so if you have a query with many different parts or many different questions so if you don't you want the information about the product but you also want comparisons to all other products and you also want um information information about reviews from you know Reddit or something like that you can give loads of different requirements and it will

do all of them for you yeah another another uh tip is like just ask it to format it in in a table it will usually do that anyway but it's uh like if you it's really helpful to have like a table with a bunch of citations and things like that um for all the categories of things that you want to research yeah there are also some features that hopefully we get into the product at some point but the model is able to the underlying model is able to embed images so it can find images of

the products um and it's also this is not a consumer use case but it's able to create graphs as well and then embed those in its response so hopefully that will come to chbt soon as well nerdy cons use case yeah yeah well and speaking of nerdy consumer use cases uh also like personal personalized education is a really interesting use case like if there's if there's a topic that you've been meaning to learn about um you know if you uh uh need to brush up on your your biology or uh or you know you want

to learn about like um like like some some world event it's um it's really good at you know put put in all the information about um what you feel like you don't understand what aspects of it you wanted to go do research on it and it'll put together a nice report for you one of my friends is considering starting a cpg company and he's been using it so much to find similar products to see if specific um names are already you know the domains already taken Market sizing like all of these different things um so

that's been that's been fun to he'll share the reports with me and I'll read them so it's been pretty fun to see another like fun use case is that it's really good at finding like a single like obscure fact on the internet like if there's like a uh you know like a an obscure TV show or something um that you want to you know to like find like one particular episode of of or something like that it'll go and it'll go deep and uh find the like one reference to it on the web oh yeah

my my brother's friend's dad had this very specific fact um it was about some Austrian General who was in power during a certain a death of someone during a battle like a very Niche question and apparently chat gbt had previously answered it wrong and he was very sure that it was wrong so he went to the public library and found a record and found that it was wrong and so then um deep research was able to get it right so we sent it to him and he was he was excited um what is the rough

mental model for you know what deep research is excellent at today and uh you know where should people be using the O Series of models where should where should they be using deep research what deep research really excels at is if you have a sort of detailed description of what you want and in order to get the best possible answer requires reading a lot of the internet um if you have kind of like more of a a vag question um it'll help you kind of clarify what you want but it's I mean it's it's really

at its best when there's like a specific set of information that you're looking for and I think it's it's very good at synthesizing information at encounters it's very good at finding um specific like hard to find information um but it's maybe less and it can make kind of some new insights I guess from what it from what it encounters but I don't think it's NE it's not making new scientific discoveries yet um and then I think using the O Series model for for me if I'm asking for something to do with coding usually it doesn't

require knowledge outside of what the model already knows from it like pre-training so you would usually use 01 Pro or o1 for coding or3 mini high and so deep research are a great example of where some of the new product directions for open AI are going I'm curious how to the extent you can share how does it work the model that powers deep research um is a fine-tune version of 03 which is our most advanced um reasoning model and we specifically trained it on um hard browsing tasks that we collected as well as um reason

other reasoning tasks and so it also has access to a browsing tool and python tool so through training um end to end on those tasks it learned like strategies to to solve them um and the resulting resulting models good at online search and Analysis yeah and like intuitively the way you can think about it is um you make this sort of this request ideally a detailed request about what you want the model thinks hard about that um it searches for information it pulls that information and it reads it um it understands how it relates to

that request and then decides um what to search for next in order to get kind of closer to the final answer that you want um and it's trained to do a good job of pulling together um all of those all that information into a nice tidy report um with citations that point back the original information that I found yeah I think what's new about deep research as an agent at capability is that because we have the ability to train end to end there are a lot of things that uh that you have to do in

the process of doing research that you couldn't really predict beforehand so I don't think it's possible to write some kind of language model program or script that would be as flexible as what the model's able to learn through training where where it's actually reacting to live web information and based on something it sees it has to make it change its strategy and um things like that so we actually see it doing pretty creative searches um you you can read the The Chain of Thought summary and I'm sure you can see sometimes it it's very um

very smart about how it how it comes up with the next thing to look for so John Carlson had a tweet that went somewhat viral you know how much of the magic of deep research is you know real time access to web content and how much of the magic is in kind of Chain of Thought uh can can you maybe shed some light on that um I think it's definitely a combination I think you can see that because there are other sear products that don't um necessarily they weren't trained ENT and so um won't be

as flexible in responding to um you responding to information and accounts won't be as creative about how to solve specific problems because they weren't specifically trained for that purpose um so it's definitely a combination I mean it's a fine tun version of 03 O3 is a very smart and Powerful Model A lot of the analysis capability um is also from the underlying 03 model training um but so I think it's definitely a combination before open AI um was working at a startup and uh we were uh dabbling and building agents um kind of the way

that I see most people describe building agents on on uh on the internet um which is essentially you know you uh you construct this graph of operations and some of the nodes in that graph are language models um and so you can the language model can decide what to do next but the overarching logic of the you know sequence of steps that H uh that happen is defined by a human and um what we found is that it's really it's like powerful way of building things to get quickly to a prototype but um it falls

down pretty quickly in the real world because it's very hard to anticipate all the scenarios um that the model might face and think about all the different branches of the path that you might want to take um in addition to that the um models often are not the best decision makers um at nodes and that graph because they weren't trained to do to make those decisions they were trained to do things that look similar to that um and so I think the the uh the the thing that's really powerful about this model um is that

it's trained directly end to end to solve the kinds of tasks that uh that users are using it to solve so you don't have to set up a graph or make those node like decisions in the back on the architecture on the back end it's all driven by the model itself yeah can you say more about this because you know it seems like that's one of the very opinionated decisions that you've made and clearly it's worked um there's so many companies that are building on your API um kind of prompting uh to you know to

you know solve specific tasks for specific users do you think all a lot of those applications would be better served by kind of having you know trained models end to end for their specific workflows I think if you have a very specific workflow that is quite predictable it makes a lot of sense to do something um like just described but if you have something that um has a lot of edge cases or um it needs to be be quite flexible then I think something similar to deep research is probably a better approach yeah I I

think like the guidance I give people is um the the one thing that you don't want to bake into the model is like kind of hard and fast rules um like if you have you know a database that you don't want the model to touch or something like that it's it's better to encode that in in human written logic but I think it's kind of like a a lesson that I've seen people learn over and over again in this field is like um you know we we think that we can do things that are smarter

than what the models do by writing it ourselves but uh in reality like usually the mo like as the field progresses the model come up with um better solutions to things than humans do and um and uh also like you know um the like probably like number one lesson of machine learning is like you get what you optimize for and so um if you if you're able to set up the system such that you can optimize directly for the outcome that you're looking for um the results are going to be much much better than if

you sort of try to glue together models that are not optimized end to end for for the the tasks that you're trying to have them do so my like like long-term guidance is that um you know I think like reinforcement learning um tuning on top of models is probably going to be a critical part of how the most powerful agents get built what were the biggest technical challenges along the way to to making this work I well I mean maybe I can say as like an observer um rather than someone who was involved in this

from the beginning but it seems like kind of one of the uh the things that um that Isa and the rest of the team worked really really hard on and was kind of like one of the hidden keys to success was like um making really high quality data sets um it's uh you know another one of those like age-old lessons in machine learning that people keep relearning but the the quality of the data that you put into the model is is probably the biggest determining factor in the quality of the model that you get on

the other side and then have someone like um Edward Edward son who's other um person who works on the project who just any data set he will optimize so that's a secret to success find your Edward yes great great machine learning uh uh model training how do you make sure that it's right yeah so that's obviously a core part of this model and product is that we want it to be users to be able to trust the outputs so part of that is we have um citations and so um users are able to see where

the model is um citing information from and we during training that's something that we actually like try and make sure is um correct but it's still possible for the model to make mistakes or hallucinate or trust a source that maybe isn't the most um trust worthy source of information so that's definitely an active area where we're want to continue improving the model how should we think about this together with you know 03 and operator and other different releases like does this use operator do do these all build on top of each other or are they

all kind of a series of different applications of o03 uh today these are pretty disconnected um but you can kind of um you can imagine kind of where we're going with this uh right which is like um the the ultimate agent that um people will have access to at some point in the future should be able to do um you know not just web search or using a computer or any of the other types of actions that you'd want like kind of a a human assistant to do but should be able to fuse all these

things in a more natural way any other design decisions that you know you've taken that are maybe not obvious at first glance I think one of them is the the clarification flow so if you've used deep research the model will ask you questions before starting its research and usually chat gbt maybe will ask you a question at the end of its response but it usually doesn't have such um uh that kind of behavior upfront um and that was intentional because um you will get the best response from the research model if the prompt is very

well specified and detailed and think that it's not the natural user Behavior to give all of the information in um the first prompt so we wanted to make sure that if you're going to wait five minutes 30 minutes that your response is as detailed and satisfactory so um we added this additional step to make sure that the user provides all the detail that we would need and I've actually seen a bunch of of people on Twitter are saying that they have this flow or that they will talk to 01 or 01 Pro or to help

um make their prompt more detailed and then once they're happy with the prompt then they'll send it to deep research which is interesting um so people are finding their own own workflow so how to use this so there's been three different deep research products launched in the last few months tell us a little about what makes you guys special and how we should think about it and they're all called Deep research right they're research yeah not a lot of naming creativity in this field um I I think people should uh should try all them for

themselves and get a feel I think uh I think the the difference in like quality um I think they they all have pros and cons but I think the the difference will be clear um but like what that comes down to is just the way that this model was built um and the the um sort of the effort that went into um constructing the data sets and then the the engine that we have with the O Series mod models um which allows us to just um optimize models uh to make things that are like really

smart and really high quality we had the 01 team on the podcast last year and we were joking that open I is not that good at naming things I will say this is your best named product deep researches at least it describes what it does I guess yeah so I'm curious to hear a little about where you want to go from here you have deep research today what do you think it looks like a year from now and what maybe are complimentary things you want to build along the way well excited to expand the data

sources that the model has access to we've trained a model that's generally very good at browsing public information but um it should also be able to to search private data as well and then I think just pushing the capabilities um further so it could be better at browsing it could be better at analysis and then and then thinking about how this fits into our agent road map more broadly um like I think the the recipe here is um something that's going to scale to a pretty wide range of use cases um things that are uh

can surprise people how well they work um but this idea of you take a a state-of-the-art reasoning model you give it access to the same tools um that that humans can use to uh to do their jobs or to go about their daily lives and then you optimize directly for the kinds of outcomes that um that you're looking that uh you want the agent to be able to do um that recipe there's like really nothing stopping that recipe from scaling to more and more complex tasks um so I feel like yeah AGI is like an

operational problem now um and uh I think yeah um a lot of things to come in that general formula I'm so Sam had a pretty striking quote of deep research will take over a single DIN percentage of all economically viable tasks valuable tasks in the world how should we think about that I think of it as like um it's deep research is not capable of uh doing all of what you do um but it is capable of saving you like hours or sometimes in some cases days at a time um and so I I think

like uh what we're hopefully relatively close to is um deep research in the agents that we build next and the Agents that we build on top of it um giving you you know 1 5 10 25% of your time back depending on the type of work that you do I mean I think you've already automated 80% of what I do so it's definitely on the higher end for me we just need to start writing checks I guess yeah are there entire job categories that you think are kind of more at risk is the wrong word

but like more in the in the strike zone for what deep research is exceptional so for example I'm thinking Consulting uh but like Are there specific categories that you think are more in strikes on yeah I used to be consultant I don't think any jobs are at risk like I I don't really think of this as like a labor replacement kind of thing um at all like it's uh but for these types of knowledge work jobs where like where you are spending a lot of your time kind of looking through information and making conclusions I

I think it's uh it's going to give people superpowers yeah I'm very excited about a lot of the medical use cases just the ability to um find all of the literature or all of the recent cases um for a certain condition I think I've already seen a lot of doctors posting about this or like they've reached out to us and said oh we used it for this thing we used it to help find um a clinical trial for this patient or something like that so just people who are already so busy just saving some time

or um it's maybe something that they wouldn't have had time to do so and now they they are able to have that information for them yeah and I think the like the impact of that is like maybe a little bit more profound than it sounds on the surface right it's not just like it's not just like uh you know getting 5% of your time back but it's um the the type of that might have taken you four hours or eight hours to do um now you can do for you know um a chat gbt subscription

and 5 minutes and so like what types of things would you do if you had infinite time that now maybe you can do like many many copies of so like you know you should should you do uh research on every single possible startup that you could invest in instead of just the ones that you have time to meet with things like that or on the consumer side one thing that I'm thinking of is you know the the working mom that's too busy to plan a birthday party for the for her like now it's now it's

doable so it's I agree with you it's way more important than 5% of your time it's all the things you couldn't do before exactly what does this change about education and the way we should learn and you know what what will you be teaching your kids now that we're in a world of agents in deep research yeah education's been like one of the top few things that people use it for um I think it's I mean this is true for atrib generally it's it's like a if uh like learning things by talking to um an

AI system that is able to like personalize uh the information that it gives you based on what you tell it or uh maybe in the future what it knows about you um feels like a much more efficient way to learn and a much more engaging way to learn than uh like reading textbooks we have some lightning round questions all right okay your favorite deep research use case I'll say yeah like personalized education just like learning about anything I want to learn about i' I've already mentioned this but I think a lot of the personal stories

that people have shared about finding information about a diagnosis that they received or someone in that family received have been really great to see okay we saw a few application categories breakout last year so for example coding being being an obvious one what application categories do you think will break out this year I mean clearly agents agents I was gonna say too okay 2025 is the year of the agent I think so yeah and then how do you think about what piece of content that you should recommend people reading to read to learn more about

agents or where the state of AI is going could be an author too training data this podcast not I think it's it's like um it's so hard to keep up with the state-of-the-art in AI um I think the like the general advice I have for people is like um pick one or two subtopics that you're really interested in and go like cu cre a list of people who are we think are saying interesting things about it and like how to find those one or two things you're interested in um maybe actually that's a good deep

research use case like you know go go uh go use it to F like go deep on things that you want to learn more about this this is a bit old now but I think a few years ago I watched the I think it's called like foundations of RL or something like this from pel and it's um it's it's a few years old but I think that it was a good introduction to reinforcement Landing so yeah would definitely second any any content by uh Peter AAL my grad school adviser yeah oh yeah yeah okay reinforcement

learning is it you know it kind of went through a peak and then felt like it was in a little bit of adrum again is speaking again is is that the right read on what's happening with RL it's so back yeah so back yeah why why now because everything else is working like uh I think if you um maybe people have been following the field for a while we'll remember the Y laon cake analog analogy if you're building a cake um then most of the cake is the cake and then there's a little bit of

frosting and then there's a few cherries on top and the analogy was that like unsupervised learning there's the cake supervised learning is the frosting and uh reinforcement learning is the cherries on top when we in the field were working on reinforcement learning back in you know 2015 2016 it's kind of like I think uh uh Yan lon's analogy which I think in retrospect is probably correct is that we were like trying to add the cherries before we had the cake um but now we have language models that are pre-trained on massive amounts of data and

are incredibly capable um we know how to uh how to you know do uh supervised fine tuning on those language models to make them good at instruction following and like generally doing the things that people want them to do and so now that that works really well it's like very ripe to uh to tune those models for any kind of use case that you can define a reward function for great okay so from this lightning we got agents will be you know the breakout category in 2025 and reinforcement learning is so back I love it

um thank you guys so much for joining us we we love this conversation congratulations on an incredible product and we can't wait to see what comes with it thank you thank you thank you [Music] [Music] oh [Music]