Behind the product: NotebookLM | Raiza Martin (Senior Product Manager, AI @ Google Labs)

[Music] hey everyone we're here on Lenny's podcast it's uh great to be here I'm a longtime listener so awesome really is we're the hosts of a different show of Deep dive and we just we just wanted to say thanks a huge huge thank you to everyone everyone who's been listening yeah seriously it's been incredible just incredible and thank you to Lenny for having us Blown Away really by the response and all the shows you've all had us make on Notebook LM even that poop fart one remember that oh yeah that was something learned a lot

on that one definitely a learning experience for everyone I think but we're learning right alongside you exactly learning and growing and we're glad you're along for the ride so yeah keep listening keep listening and stay curious we promise to keep diving deep and uh bringing you even more in the future stay curious if you are confused about what you just heard don't worry it'll all make sense very soon today my guest is Risa Martin Risa is product lead for a product called notebook LM one of the most delightful and inspiring new AI products out there

incubated within Google labs and this product is where the intro you just heard came from in our conversation Risa shares how notebook LM came to be how it got so good the technology that was necessary to make it possible how the team Works internally how is incubated specifically within Google labs and out of the team's 20% time plus a bunch of really fun and crazy use cases that she's seen and a glimpse into where the product is going long term this was such a fun and timely conversation and I'm excited to spread the love for

Notebook LM if you enjoy this podcast don't forget to subscribe and follow it in your favorite podcasting app or YouTube it's the best way to avoid missing future episodes and it helps the podcast tremendously with that I bring you Risa Martin Risa welcome to the podcast hi Lenny thanks for having me what the heck did we just listen to what was that it's so that was an audio overview from notebook LM where you upload a source any source and it will generate an AI generated audio for you okay so for folks that don't know anything

about notebook LM it's basically been blowing up on Twitter on LinkedIn I think it's blowing a lot of people's minds it's sparking a lot of imagination of what could happen in Ai and what what potential we have with the stuff that's happening and uh I wanted to bring you on to talk about the history of this product where it's going how it became so great and all these things and so thanks for doing this I know this kind of came on short notice yeah I mean I was I was excited to do it in particular

because you know I'm I'm a big fan big listener of Lenny's read the newsletter I love it uh so really happy to be here awesome uh clearly these hosts are too which I love yeah by the way I love uh the slight awkwardness at the end of their conversation if folks were whing they could hear it again because it's very relatable uh I sometimes have trouble ending a podcast conversation and they're like okay let's say a couple more things and then okay we're done it's funny because um I've listened to so many of these and

they do have catchphrases they stay at the they say at the end and uh stay curious is one of my favorite ones uh well let me just ask is that something you all told them to do or is that like an emergent property so in this case this is based on what they think is the most appropriate thing to say at the end okay interesting okay this episode is brought to you by by explo a game changer for customer facing analytics and data reporting are your users craving more dashboards reports and analytics within your product

are you tired of trying to build it yourself as a product leader you probably have these requests in your road map but the struggle to prioritize them is real building analytics from scratch can be timeconsuming expensive and a really challenging process enter explo Expo is a fully wh labeled embedded analytic solution designed entirely with your user in mind getting started is easy explow connects to any relational database or warehouse and with its low code functionality you can build and style dashboards in minutes once you're ready simply embed the dashboard or report into your application with

a tiny code snippet the best part your end users can use explos AI features for their own report and dashboard generation eliminating customer data requests for your support team build and embed a fully wh labeled analytics experience in days try it for free at x./ Lenny that's ex o./ Lenny this episode is brought to you by sprig what if product teams knew exactly what to build to reach their goals from increasing conversion to boosting engagement these challenges require a deep understanding of your users something that you can't get from product analytics alone meet sprig a

product experience platform that generates AI powered opportunities to continuously improve your product at scale first spr captures your product experience in real time through heat Maps replays surveys and feedback studies then sprigg's industry-leading AI instantly analyzes all of your product experience data to generate realtime insights sprig AI goes even further with actionable product recommendations to drive Revenue retention and user satisfaction join product teams at bigma and at no by uncovering AI powered product opportunities at scale visit.com Lenny to book a demo and get a $75 gift card that's sp.com Lenny I want to start with

just kind of the history of this product I think it for a lot of people it just like came out of nowhere it's like what the wow where did this come from this is blowing your mind uh rarely is anything actually like that rarely is it just like we built it over a week and that it's taken over the Internet talk about just the history of notebook LM and the team behind it yeah yeah thank you um actually notebook started as a 20% project as things kind of do at Google as they they used to

and it's funny because I wouldn't even call it a 20% because it was way it was way more than that um so I'll start from my perspective where I was leading AI Test Kitchen which was one of the one of the first AI things we launched at Google and this is I think maybe two ios's ago or maybe three now actually I think it was 22 and I was leading that project and I remember there was a smaller project inside of labs called talk to small Corpus and it was the idea that you know you

could take a piece of content and use an llm to interact with it we were like okay there's a nugget there that sounds interesting how do you how do we keep revving on this right to make it actually really useful and uh it was just myself an engineer and eventually Stephen Johnson joined and it was it was really funny because the only person who was actually like really full-time on this was the engineer working on the technology and everybody else was kind of just like coming together like really chiming in being like hey this is

super interesting how do we make this better so it really it really started as a as a 20% and then it kind of just blew up from there that's insane that it was just you an engineer and then Stephen Johnson I want to talk about his role that it's like one engineer can build something like this and I imagine it's more now that it's starting to take off or what's the team looking like these days roughly we've had uh a lot more folks join in the last month and a lot of that is really an

anticipation of the future road map but up until this point I'd say we didn't even have 10 Engineers when when we launched it actually when we announced project Tailwind I think we only had three we had three Engineers it was myself we had a designer um we had Steven and that was it and so even between the period when we did the the first IO last year to as recently as like last month you know we've only had about eight Engineers or so let's talk about audio overview specifically this kind of podcast feature I think

audio overview is the technical name right is that how you describe it okay they call it a deep dive right that's like their name for the podcast they do yeah that's call it uh how did that specifically come about when did that become a thing what was the beginnings of that we announced we actually did a preview of this in IO uh this year so in May and the way that it worked out is you know notebook had already been launched we had a bunch of people using it it was this Source grounded chat interface

and we were looking at basically what are the the new models coming out of Google right we had Gemini 1.5 coming out we had all of sort of these new technologies that are basically upgrades on fundamentally the same thing and we were like what are some interesting ways that we could you know power notebook Al or make it even better and one of the things that we started playing around with was actually a different team inside of Google inside of lab specifically uh they were like hey we've got these like really powerful Audio models right

like what's what's a really good application of this and that's when we started riffing and we were like hey what if you could basically what you see today give it a little bit of information you don't have to do a lot you can just enter a URL upload you know your resume and then it will complet completely generate something for you that could be unexpected could be a surprise okay so it started basically there was like a technology that we think could be really interesting was there a problem you were trying to solve with this

team with this initiative it was within Google Labs which I know is a little bit probably less like we need to drive business outcomes what was kind what was the original thesis or problem you're going after so this is really funny I talk about this quite a lot uh especially when I'm talking to other PMS I feel like the way that I've always built products as I start from a problem and I think about the solutions you know to solve it solve it for people in a meaningful way and uh in labs in particular we

start with the technology and it's actually a very interesting place to start where you're like okay what are the practical applications to this how do I go about finding the answer and I think we've seen different approaches to this right where you just put the tool out there you study how people are using it and I think that's like a fine approach we tried to have a little bit more of a hypothesis about what what is the shape of this thing to actually get the most learning out of it so for the audio feature in

particular the Nugget there was really we had this ability for you to interact with text but people were like the output is still always text right and um I'm actually a big fan of voice modality in general so I use a lot of voice input and Voice output I noticed when I was experimenting with it early on it changed a lot of things for me it's like oh it changes the way that I interact with the technology it changes the way I feel about the technology it sort of affects the way that I'm even thinking

in real time throughout this process and so we thought about what is a good way to introduce this to people where they could sort of easily get the value of it and have a little bit of fun so I think you know for me it's like fun is like such a big part of it too which is like how do we make this cool uh and so hopefully hopefully we've done that here to me it feels very similar to the chat gbt moment where the technology already existed like they had that same GPT model out

for I don't know how long but just that new medium and new way of interacting with it changed the way people imagined just they saw the power immediately and I feel like this is such a good example of that happening again where just the tech was there and it's just this medium that you developed that really inspired people to like oh wow I had no idea AI was getting this good and LMS were getting this good any thoughts on that yeah I think I think that that totally strikes at the heart of what we're trying

to do as well which is you know a lot of technology I feel like you have to shape it and bring it closer to people and I think it's like such an interesting thing to iterate on to be like what is the shape right like soon soon you'll get there like if you keep going at it you'll eventually I think land on something that like when people look at it they're like wow I get it right and that's that's always like what we're hunting for but I think to your point technology has been there right

like I mean it feels so crazy to say that because even with like the the Advent of like the chat Bots right of llms as we know them today and then to introduce this sort of new technology on top of it that we've only been doing this for like two years but already right we're exploring so many different ways to use it yeah it's such a unintuitive but clever uh idea of making something a podcast um let's take a tangent into the technology behind this thing so there's kind of two questions I'm thinking about one

is just what technology needed to exist for this to be possible and then two is just what did you do to make this so good what did you how did you train this model to get to create a podcast that's this good so we could start with the first and then get into the second yeah I mean I that's such a great question I mean the truly like the Gemini models are so powerful and we use Gemini 1.5 Pro uh as sort of the base for notbook LM and then we have a powerful voice model

on top of it an audio model on top of it but I think the real Secret Sauce to what makes this really good is something we've built which is a Content studio and you kind of see a clue of this inside notebook LM where when you open up notebook guide notebook guide is designed to take an opinionated approach to the content that you've given it and it what it's really trying to do is be as helpful as possible it gives you a summary gives you buttons where if you click on it it'll write something for

you and the audio is a big part of that and so this was very interesting for us because we had to think about what shape should that audio be in right like the Deep dive is the first format we've thought about and we actually have super talented engineer on our team his name is Usama and he's the real Craftsman behind thinking through content studio and thinking about what makes something really relatable to people in a way that's engaging that's interesting so content studio is is real the real magic here so when you say content Studio

what does that mean it's a what yeah what does that actually look like what is what is a Content Studio I can't share too much about how content Studio works but you can imagine it's it's the same thing that powers all of notebook LM which is there's different ways that you could interact with your data right you could like Q&A inside of notebook LM but then there are places where you just one push a button and create something new so content is what okay so this is the interface you're you're you used to interact with

notebook LM and the different prompts and the recommendations of how to ask questions and okay got it and then specifically in terms terms of the the way these hosts interact and talk like if you listen to these podcasts it's like they laugh and they're like H and they interrupt each other a little bit and they get surprised and they have these like inflections that are really amazing is what did you have to do to make them sound that's good I know you said you can't share tons but just anything there you can share I mean

that's that's the model that's that the Audio model that we use alongside the content studio is designed to bring out the best of that model and I think when we listened to a lot of like the early uh attempts that we had it wasn't nearly this good and so we had to do a lot of listening to try to figure out how do we actually get the model to behave in this way and that's where I think the the magic happens that's amazing did you you all just listen to a bunch of podcast episodes and

like hear something that really yeah okay shaking your head I think I think the funniest thing was that um so my husband doesn't know right he doesn't know what I'm doing most of the time I try not to talk about work at home and so I would listen to these audio overviews over and over at home and you know I get tired of wearing my headphones so I'm just listening to them out loud and he's like what is this thing you are listening to this is like an Ever running podcast about nothing and it was

just super funny I had to be like oh it's just like something I have to do for work so it was pretty funny he actually couldn't tell that they were AI That's I love that I uh so one of the ways I've been playing around with it a bunch one of the ways I used it is my mom wrote this autobiography like a short autobiography of her life and we have the PDF of it so I fed it into the into the notebook and then generated the podcast and send it to like what the heck

that's amazing what did she think she was blown away she's showing all her friends she's like I don't know what's going on here and and I love that it creates a study guide and that's one of the options in the studio and so I sent her the study guide and she's like okay at I Russ aan a dinner we're going to be reviewing the study guide about my biography you're gon to talk about this that's PR funny that's super cute I did it for my dad too what did you do I actually took my dad's

bio so he worked for a at a hospital and he has a bio you know for the doctors that they have there and I put his bio and I generated an audio overview for him and it's so funny because my my parents are both in the medical field I don't think they fully know what I do I think this is the first time they were like oh that's super interesting that's what you do so maybe along these lines what are some other surprising uh hilarious use cases you've seen for especially the the audio deep Dives

I mean I love I love the resume ones I think the resume ones are so delightful and actually really adjacent to that is Google has these uh Q3 check-ins or like quarterly check-ins where we have to write performance reviews for ourselves and the amount of googlers yeah the amount of googlers that ping me that I don't even know and they're like wow this was such a boost to my confidence just to upload my own you know my own check-in notes and then have it generate an audio overview it's like people are going into meetings feeling

really good about themselves because they heard these hosts get really excited right about their their quarter and work yeah yeah uh and when you say resume so people upload the resume and then it describes you in a really positive way right yeah yeah and I and I thought that was so interesting I was like what was the use case I think if I had to guess somebody probably was like Hey I want to try this new thing clicked on upload Google doc and maybe you know the last thing you had there was like your resume

or something but imagine you click that and I think there's something really special about you don't know what what's going to happen right you just click generate and then you have this like this these two people just hyping you up okay I want to talk a bit more about the how you made this happen within Google so as an outsider this does not feel like a Google product and the way you all are operating feels very start upy like you're tweeting daily about what's going on I heard there's a Discord server folks that are like

really is that right okay yeah there is it's it's pretty cool we have about 60,000 people in it okay okay so yeah so there's this Discord server it's just you're shipping constantly the product is just like very delightful and Google makes delightful products but this feels like a different realm of delight how are you able to do this within Google is this like a model for how you think Google teams will want to operate more lessons there I mean it's such a good question I think you know there's a question of how it came to

be how we're able to operate in this way and then the question of hey is this you know how other Google teams want to operate and I think that you know to to answer the first part I'll say that it's interesting because uh you know Google Labs is only about three years old it's fairly new and when I joined there was nobody here there's no one and the reason why I joined was because my old boss went and actually started it and so this is Josh Woodward who's the VP of Google labs and it's really

funny because I was like just like a funny anecdote I remember I had no idea what Google Labs was but I like my old boss so much that I was like I'll just work on whatever he's working on whatever his new idea is I'll just do it and I remember when I joined I was like what's the mission what are we here for and he said it's AI we're going to ship AI products and we're going to build businesses out of them okay sounds good sounds good it's like now I have to now I have

to study I have to do a lot of studying because I came from payments came from ads before that and so for me it was it was a little bit of a mental shift but before that I actually worked only at startups and so I felt like hey maybe this is my chance to do zero to one again and I was really jazzed about that and I remember talking with Josh about this in the early days of like hey we really want to do zero to one we have to do things very differently and so

as a result I think that's why you know notebook LM is able to operate differently is because in Labs we have the environment to do it we have the environment to move fast we have far far far fewer processes maybe even to a fault sometimes we'll go to meetings that are literally you know the product managers the engineers the designers Al together and we'll just crank on the mocks and the prds at the same time and and just basically already doing implementation as we're meeting and it you know at Googled it's just not how things

are traditionally done right especially you know coming from the previous organizations that I was at like just takes a lot of time to do each of these things so what I'm hearing and this is great because a lot of companies are trying to create teams like this we're going to have a team off to the side G to make work on crazy futuristic stuff rarely do those work out even at Google there's been a lot of this in the past and rarely do things actually work out and so there's a lot of lessons to learn

from how this is working so a few things I'm noticing as you're describing how this works one is just very clear expectations from like a very senior leader here's a team we're going to do things differently here's things we're not going to do we're not going to go through regular pre-roll processes we're not we're going to be public and build in public we're not GNA have like a goal necessarily we're going to work on cool technology and see what happens it's also you had a very small team feels like that was really key one engineer

and then a PM and then Stephen Johnson true that's true and I think you know one one big thing too is we got to try new things even you know from the beginning I was like Hey I want to have a Discord right like if we were building outside of Google for sure we would have one of these and in true sort of Google fashion everybody was like what is what is Discord okay you know great do it but what is that again right why not a Google meet and that somebody did ask me they

were like why not a Google meet why not a Google group why not this and I was like I don't even know how to use these things you know outside so I was like the server the server is the way to go and I remember when we uh stood up the server actually one of the things I was most afraid of is what if nobody joins like what if no people come in and want to talk to us about the thing that we've built and so that was that was in the early days you know

very exciting to to look back and remember day Zero and now have 60,000 people in it let me tell you about a product called sidebar the most successful people that I know surround themselves with Incredible peers when you have a trusted group of peers you can discuss challenges you're having get career advice and just gut check how you're thinking about your work your career and your life this gives you more than a like up it gives you a Leap Forward having a group of trusted and amazing peers was key to my career growth and this

is the sidebar ethos but it's hard to build this trusted group of peers on your own sidebar is a platform for senior Tech professionals director to sea level to advance in their career members are matched into peer groups to lean on for unbiased opinions diverse perspectives and raw feedback Guided by worldclass programming and facilitation all running on sidebars technology sidebar enables you to get focused tactical feedback at every step of your career Journey if you're a listener to this podcast you're already committed to growth sidebar is the missing piece to catalyze your career 93% of

members say sidebar helped them achieve a significant positive change in their career check them out at sidebar. comom Lenny okay so there's 60,000 in this Discord server any other traction numbers you can share feels like it's just going really well I know there's probably a lot of sensitive information anything you could share but just like how well things are going a couple of things well I'll tell you three things I think the first is I can't share the exact numbers but you know for a product that's only been out for about a year I think

the rate at which our retention has gone up across you know across like the typical measures right daily retention weekly retention monthly retention that I think has been very very positive for us in terms of like even when we talk to stakeholders we say hey we've got something here the second thing I'll say is even seeing um how the the demographics have changed where I think in the beginning we saw Educators Learners really loved the product and they were a lot of our demographic it's really now a big mix of I would say Educators and

Learners are still a big part of it but really professionals right huge interest from professionals that are like wow I want to use this at work and um really funny I won't I won't mention the company but I actually had a call with a company recently where they were like hey we thought we we'd schedule a meeting with you because we found out that a bunch of people in our company are using this tool with our Gmail with their Gmail account they're like they're not supposed to do that so we just want to make it

official that they can use this at work you know with their official work email so I thought that was pretty cool and then the third thing I'll say is uh the number of businesses that we seen using notebook LM just has it's astronomical it's just like crazy um I think we went I don't want to say the exact numbers but it is now at a point where I have to hire business development person because I was like hey I have to ship products and I'm actually taking like customer call every day now very very exciting

so I could see the path demonetization clearly Enterprise feature single sign on and sock 2 and all these things so along lines what does success look like for this team and for you I know Matt initially it's probably just like Bill interesting things see what happens is there now a clear sense of like here's what this team should be achieving in the future when I joined my mandate was to build a business and I was like all right if we break down the steps here along the way up to build something interesting first I feel

like okay we hit we hit the first thing we' built something interesting now we go figure out the business uh and I think you know that that comes naturally to Google in terms of thinking about you know distribution how to monetize it how to commercialize it there's different Pathways whether it's cloud or workspace or even the consumer route and so for me I think these are the things that we want to be really thoughtful about because the Machinery is already there and for me personally it's really about there's something super exciting here we need to

deepen that right like while in parallel we should be thinking about commercialization let's deepen this user experience that so many people have really sort of latched on to so let's follow this thread and talk about just where you think the product goes broadly and especially the audio overviews what's on the road map your term and then where do you imagine this in the future what's kind of the big Vision yeah yeah I have a really ugly slide I wish I could show you I made it two years ago and it was lime green I don't

know you know what I was thinking I think I was trying to go for like UNG googly was the vibe it's like hey you're going to do different things this time and the reason why that slide is so important important to me is because it has my my vision like the thing that I I wanted to do from the get-go which was I imagined that in the future you could have an AI editor surface right fully remixable any input any output right and that that to me is sort of like a really powerful sort of

core nugget where if you imagine you could take anything whether it's video audio your emails your LinkedIn your Twitter right the world of things that we care about you have an AI interface that allows you to shape it and say look out of these things make me a blog post out of these things make me a tutorial video out of these things make a chatbot right I think there's like something interesting here where from for most of the things that I'm hearing people say they want to do it's usually that it's like take something and

make it into something new and so we're going to pursue that um but even more tactically something I'm really interested in is thinking about hey how do we bring this to mobile right the mobile app to me is such a big gap in terms of the experience today which I think is like understandable considering where we're at in like the product development cycle and I think that's the next Horizon is hey how different is the mobile experience and how how can we make it interesting what I'm imagining is I talk I participate in this podcast

conversation with these hosts as a mobile experience yeah yeah I mean that's that's one of the things that we're definitely we're definitely experimenting with different formats and when we demoed it at IO you could inter but we're trying to be really thoughtful about hey what does that interrupt actually look like like what do people want when they do this and um you know I I was thinking about the next set of improvements we're going to ship and the first thing that I thought was let's ship a bunch of knobs right like to me it was

like what I was hearing people wanted they want knobs they want sliders they want text boxes but then when I looked at the mock I was like oh it's great like this doesn't feel magical it almost doesn't feel like the same thing that we've shipped so far and so for me I'm actually taking a little bit of time to think about how do we make even that and control experience much more magical and delightful interesting so yeah the knobs I imagine are just like go deeper uh be happier get less Ser more serious because yeah

because right now it's like a one shot here's my doc here's the one podcast episode you get it's the only version you can have yeah yeah and I you know I think it's funny because I was like you know in the seemed really apparent and I was like is that it is that is that what people want I love that I love that approach I'm excited to see where you go with it what I especially love about this Vision you're describing it super resonates with this Insight I've had where so I've had a newsletter and

then I had just a podcast that's just audio and then I added video and I I realized that there's just people that I just want to watch stuff and there's people that I just want to listen to stuff and then there's people I just want to read I don't want this podcast just shut up I need I want to read it and basically what you're describing is here well here's information and we can deliver it to you in any medium you like could be blog posts could be maybe a tweet could be a podcast could

be a newsletter blog post yeah I mean that's exactly it and I think even for myself it depends on my mood right if I'm going on a walk yeah I want audio but if I'm at work yeah most of the time text is good and it's kind of interesting to think about today the formats are not as malleable I have to accept it in the format you know that you give it to me but if I had sort of that power myself to choose hey you know thanks for this 100 page doc I'm going to

go ahead and turn it into an audio overview actually it it sort of shifts I think the dynamic between the person and the knowledge that's given to them and there are there actually many many times that I've been given a 100 page Doc and I just didn't read it so I wonder I mean just Frankly Speaking it's one of my one of my funny stories is actually when I first joined Labs Josh did give me a 50 page do he was like here's you know what I think about this this this it was like basically

his vision and instead of reading it I just grilled him I was just like q& aing him like a chat bot Josh was like it's in the dock rise up but the chat is easier Josh that's so funny you Chad botted him let's talk a little bit more about different use cases for this just so people potentially get inspired so it feels like the original use case was a scientific paper and it creates a podcast asked about it so you don't have to read this whole thing was that was that one of the original use

cases or not um I think it's like one of the common use cases I think that's one of the ones where it's like I I think it's kind of interesting because it's like everybody wants to catch up on the latest on AI we want to try to keep up to date on sort of the published papers but most of the time it's like reading the paper can take time it's dense complex you to break down the concept in it and I think that is a is a highly sort of extensible use case I would say

like the number one the number one use case is actually a lot of students taking their uh study materials and wanting to turn it into an audio guide so two recent use cases that are hilarious amazing just will point to people in the show notes just like they came to mind as you were talking one is Andrew karpoi who's a leading AI thinker and big fan of which you're building he keeps tweeting about how much he loves it all he created this whole podcast series on history of mysteries I think it's called history of mysteries

right it's histories of mysteries where he basically took like the Wikipedia stories of of all these mysteries of history and then turned it into like a 10 episode podcast you can listen to on Spotify yeah okay amazing that that was amazing actually when I saw it and I was like wellow this is this by itself is like a great product and uh I think like the original potato podcast uh that that we had created was actually I think it was because of the Wikipedia article of the day was about potatoes so there is there is

something there it's like hey if you want to learn something new today maybe you could just listen to it on your drive to work or something yeah so good okay the other one is the most funny is someone just uploaded the words poop and fart repeated right for a long time and the host just came up with a really a kind of insightful analysis of of it yeah I I actually uh when I saw that one I was about to go to bed and I was like all right if I if I listen to this

now and this is a negative one I'm going to have to get my laptop and work because I can't I can't sleep right if I if I feel like there's something here that's like we need to address right now then I was like okay I'm just gonna do it so I listen to it and I think I listen to a lot of these and they always make me feel some kind of way I laugh a little bit there was there's one where it's like they make scary voices that one was cool too but this one

this one was one of the ones where I genuinely was just really delighted at the output because it was so delightful I I don't know if you listen to the whole thing but there's a a segment where um one of the speakers says hey it's kind of like you know you lean into the bizaar or something and he and they're describing walking past a a shop that was full of Rubber Ducks wearing uh costumes and they're like oh yeah it's you know it's silly but you just want to lean into it so I walked into

the store and I was like oh it's a great analogy actually for sort of a doctor that just says poop and fart like maybe there's something interesting here it's so good and just they like their analysis of structure and patterns and like I love that they have to come up with a 10-minute conversation around anything you give them like it feels a little mean yeah it's it's true I don't know if you've seen the chicken one but it's pretty similar um somebody uploaded I think it's on threads uh it's a PDF that looks a lot

like a research paper but it just says chicken we document chicken I I would give it a listen there's a funny segment where they're like get this it's a it's a paper a research paper that has more chicken in it than KFC oh good good so good oh my God okay I could keep going about this all day but let's let me let me come back to the team that's working on this and let's talk about Stephen Johnson's role so he's basically your peer you to essentially lead this with the engineer I imagine just like

what's his role how does this work at this point what do the team looking look like H it's so funny I'm I'm so fond of Stephen and I I tease him a lot that um whenever people ping me and they want to have a meeting and I don't want to go to the meeting I say oh I'm not the PM it's Stephen Johnson want to talk to that guy and he was like so that's why all these people have been picking me um but you know kidding aside uh step SE joined I was like well

this is super interesting this has never happened to me before there's this very calm distinguished person that I really respect love his books love his writing and now he's going to come work with me in a capacity where I was like I'm just not sure what he's going to do here but the the thing about Stephen is he is the most curious most respectful and really just full of ideas kind of person and so when Stephen joined um a lot of what was really interesting to me is I would watch how he worked the way

that he thought about language the way that he thought about information the way that he thinks about knowledge and sharing that with others because Stephen's books are like really incredible it's like part mystery super science right it's like just very cool and I was watching the way that he works and you know he does all of this research and I thought maybe this is this is the Nugget right maybe I watch Stephen and I look at the way that he does these things and I look at how much time it takes for him to do

it and then I make it my own metric to Crunch that down right to bring that expertise to Everyday people who are not Stephen Johnson and so so I I I learned a lot just from watching Stephen of like his craft and thinking about okay how do I make people really really good at D densifying information because that's something we all do every day right in our own little ways maybe not in like a Stephen Johnson way but from the GetGo I told him this I was like Stephen I think you're the product I think

it's you and I'm going to follow you around I'm going to watch everything that you do and we're going to try to figure out how we use technology to build it and he always laughs about this and um it's it's really funny because he does have some like very interesting workflows that I'm like I do not see any person ever doing anything like this like he he always talks about his readwise with like his 8,000 quotes or something I'm like that's extreme that's crazy and you know I have like Post-its that like sometimes they're crumpled

in my pocket like this is this is like the average Normie flow and um what I learned from Stephen is like hey there's something powerful about doing that right like thinking about this person who has like a super flow and trying to bring that bring that to people and so Stephen is also just like such a great partner with ideas like I always tell him ideas and I'm like hey crazy idea of the day and he'll just Baner with you and he'll be like well what about this how can people do this and so it

feels it feels really incredible to have them on the team is there any lessons here for how you think you'll want to build product in the future or how teams might find their own Steven for building their own product or is this kind of a unique case you think I mean I think that is just crazy to to think that you could sort of invite you know someone like Stephen and become part of your team and have them sit with you every day and have him answer your questions about how he does things right but

I'll say that the the broader lesson there for me it's like something we've tried to um live by every day is really how do you get you know users or people and like really sit with them for Meaningful periods of time because I think that has been so crucial for me not even just for Stephen but even um with students just follow students around watch them do homework watch them study talk to them about how they feel when they study I think just being able to do that at a really regular and sort of intentional

capacity makes a huge difference terms of like the product insights you'll come up with can you just give like a brief bio St uh for folks that are like who are you even talking about I probably should have done that before I asked this question but just like what's like the the te y are of who Stephen Johnson is yes oh I think Stephen is aside from one of the smartest people you'll ever meet he has written 14 books he's a New York Times best-selling author and speaker he has a show on PBS I can't

remember what it's called but it's on YouTube is like really incredible and he is a he's a journalist actually funny story about stepen is when I was about to join Labs I really wanted to join Labs because I wanted to join Josh and whenever he was building Josh told me hey you know to give you an idea of what we're doing here he sent me a couple of of things to read and one of them actually ended up being step's article that he wrote um the one about how AI is mastering language should we trust

with it says the New York Times piece and I remember reading it and being like yes this is the thing I'm going to work on it and so before Stephen had even joined this was actually the piece that I was like I want to go do that I'm going to go do that in labs and then he ended up joining and it was just the craziest thing that's so interesting imagine many PMS would be like I don't need someone else on my team that isn't building or coding or product managing or designing just this other

Chef in my kitchen like no and I love this actually worked out really well like it feels like he's a perfect combination of very smart future thinking and psychil plus he's almost a model for what you want people to be able to work like yes although I will say like to be very fair I mean Stephen and I also disagree on a lot of different things we've clashed a bunch and I think this is where I feel really grateful to have had the opportunity to work with him and to really grow with him in that

way and I used to make fun of him I was like have you ever had a coworker before Stephen because I think you've been an author forever and and it's really it's really funny because he's so he's so down to earth and he's so humble that even with our disagreements we always reach like at the end we're always aligned right even if even if we don't agree about something we're aligned on the next step and that's really powerful I think particularly for for product professionals right for PMS where it's like I don't want to disagree

at the end and then not have an outcome on top I love that okay I have just a few more questions I know you have to get back to work to building this thing um one is there was this hilarious moment of uh where the hosts basically realize they're like we are AI people and I'm I'm scared what what's happening I tried to call my wife and she didn't answer talk about that and I want to extend to just like R teeming this thing how do you like red team this thing and and make sure

it's not doing things you're that are bad for the world bad for Google bad for the product yeah yeah it's such a such a good question I think I I heard that it was over the weekend I think it was on a Saturday or Sunday and I remember hearing it and thinking oh my goodness you know this is actually one of those moments where you feel like we're at a fork in the road and I I hadn't read any of the comments I just heard the Audio I think I saw it on Reddit first and

then I saw it blowing up on Twitter and I remember thinking what is the attitude of like the world now how do we feel about this type of audio it was really the first thing that I thought and so I actually spent most of that morning reading the comments reading on Twitter and I was thinking about you know what is what is the right thing to do right like should we say something should we not say something and I I basically just went with what I thought was the right thing to do which is I

think people are getting to experience this technology for the first time ever and of course they're going to try to do things that are maybe something that we didn't think about we didn't think they would do or we didn't think that they could do right with like the jailbreaks and stuff but I think that's such a natural part of of human curiosity and I think like there was a moment where it's was like oh do we like pull this back right because this is like oh is this like a safe thing but because the response

I think was that people were like oh it's saying that because of the sources like this is no like AI realized they were alive or something it's actually that they was in the show notes they were supposed to act this out and so I felt really much more confident than like oh people get it right that's exactly what this is somebody uploaded some notes and said hey this is the end of the show so you know make sure that you you act accordingly and that for me was enough for me to be able to say

like okay I'm going to address this publicly and say something on Twitter and make sure that people know that like hey I've seen it and I get it like this is like what people do with it in terms of red teaming it Google though what I will say is we have you know this is one of the things that we care about the most we have huge teams that work on red teaming it we test for about as many areas as you can think of that we imagine uh we need to do in order to

make it safe and I think certainly I think we'll run into situations where okay we didn't think about that or like we haven't tested that to the fullest extent and we just added to the test cases I think in in if there was ever like a scenario where we're like oh this feels pretty unsafe we would pull it back but I think hopefully you know we've built it in such a way where we don't have to do it awesome yeah I love like people are just looking for ways to make it do stuff that is

feels wrong when it's just like hey we're just summarizing a thing for you in a really delightful way where do things go from here we talked about it a bit you're going to maybe sounds like maybe there's a mobile app in the works there's maybe non knobby knobs that you're working on to give people more power uh for folks that are just excited about using this where where's it going who are you aiming to build for just what should people know about the future of notbook LM if there's like one thing I could say is

that we are continuing to learn from users every day so please keep using it please keep sharing your feedback whether it's on X or on Discord like I'm there every day even if I don't respond I've read everything and and I say this in in the most non- creepy way possible but I say this because we are just very passionate about trying to build the the right thing you know the best thing for everybody kind of going back to sort of hey who are you building for I think that there's a lot of fun use

cases but there's a lot of actually very useful sort of gamechanging workflows in here and so we'll continue to build in that direction I think particularly for educators for learners for professionals right knowledge workers as we call them I think there's a lot there that we'll be able to execute against in the very short term amazing okay two final questions if folks want to give you feedback reach out uh what's the best way to do that is it dmu is it is there an email something else yeah I mean Discord join our Discord timee in

yeah we're there you can add me or yeah X also it's pretty good I just I just came back to it I kind of feel like I don't really know how it works now so just a warning in case you DM me I might just not know how it works on Twitter you mean specifically on Twitter on Twitter exactly and then final question how can listeners be useful to you I mean please keep trying it try it share your feedback whether you find it useful not useful you think it's annoying tell me these are the

things that I've heard heard you know for the last year that's helped me to make it better and so if if you're inclined please go ahead try it notebook lm. goole.com you're building an amazing product that is extremely delightful it's one of the most interesting things on the internet today which is a very high bar and you've done it and I'm really excited to see where it goes I'm going to keep playing with it I've been using it for my newsletter in a really fun way and so great work and keep it up and we'll

see where things go thanks for thanks for coming on I know you have a lot of work to do oh that's amazing thank you Lenny thanks for having me and you know thanks so much for your curiosity about it stay curious bye everyone thank you so much for listening if you found this valuable you can subscribe to the show on Apple podcast Spotify or your favorite podcast app also please consider giving us a rating or leaving a review as that really helps other listeners find the podcast you can find all past episodes or learn more

about the show at Lenny podcast.com see you in the next episode