Whitepaper Companion Podcast - Prompt Engineering

all right let's dive into prompt engineering it's like the key to unlocking the power of llms and luckily we've got Lee boer's white paper to guide us on this deep dive it's full of practical techniques it's awesome how Boonstra really makes it clear that you don't need to be a coding genius to do this anyone can become a prompt engineer that's great to hear so if it's not all about coding then what is it what's the secret sauce of this prompt engineering thing well it all comes down to like understanding how these llms think you

know they're like super Advanced prediction machines constantly guessing the next word based on what they're given okay I'm starting to see the picture so prompt engineering it's like giving these llms instructions right yeah to get them to give you the results you want exactly you're crafting prompts that guide them toward what you're looking for so it's kind of like giving directions but to a super smart but maybe easily distracted friend you can't just say take me to the best pizza place you got to be specific that's a spot-on analogy and just like with directions you

have to think about a bunch of things which llm you're using because they all have their quirks how long your prompt is your writing style even how you structure your request it's all about experimenting and refining as you go trial and error classic so once you've got the basics of giving clu instructions down what are some other tricks we can use to fine-tune these llms well booner gets into some really cool ways to tweak the output you get from an llm one of them is called temperature temperature now that sounds interesting think give it as

a dial that controls how creative the llm gets with its responses you ever see those AI chat Bots that go completely off the rails spouting nonsense or just weird stuff oh yeah I've definitely seen a few of those like they took a random word generator and just let it loose on the internet that's high temperature in action it encourages the llm to be unpredictable explore those less likely word choices on the flip side low temperature it makes things more predictable sticking to Safe factual answers so it's like choosing between like a wild experimental Chef who

throws ingredients together randomly and a meticulous Baker who follows the recipe exactly you nailed it and there are other settings too like top K and top P that kind of act like filters on the ll's vocabulary okay those sound a little technical break it down for me imagine the llm has this giant dictionary in its head right with top K you're basically saying hey only use the top K most likely words for each step of your prediction so it helps the llm focus on a smaller more relevant set of words so instead of trying to

juggle thousands of words it Narrows down the choices makes sense yep and top p is similar but instead of a fixed number of words it uses a probability threshold the llm will only consider words that like together add up to a certain probability it's a bit more nuanced but the goal is the same make the output more coherent less likely to wander off into weird tangents so it's like gently nudging the llm to stay on topic yeah keep it from going off on those wild high temperature ramblings precisely now let's dive into some actual prompting

techniques the simplest one is called zero shot prompting pretty much what it sounds like you give the llm a task no examples just straight to the point straight to the point I like it what kind of tasks can we throw at an llm with this zero shot approach well Boonstra uses a fun example of classifying movie reviews he gives the llm a review that calls a film a disturbing Masterpiece and asks it to say is this a positive or negative review ooh tricky even a human might have trouble with that one is disturbing good or

bad in this context who knows exactly it shows how llms can get tripped up by language they don't have that nuanced understanding of human emotions or sarcasm you know things we kind of take for granted that's a good point we forget sometimes that llms are machines even though they can produce human quality text so if zero shot prompting is about giving no examples what's next well that would be one shot and F shot prompting you give the llm a small number of examples to learn from so instead of just telling the LM what to do

we show it exactly it's like teaching by demonstration booner goes through a cool scenario of parsing pizza orders into Json using just a few examples okay now that sounds incredibly useful so instead of trying to explain like all the rules of Json syntax you just show the llm some correctly formatted orders and it figures it out exactly it's fuse shot learning it's powerful stuff and it mimics how we learn a lot of things think about learning a new board game do you read the whole rule book or do you watch someone play a round or

two and jump in oh definitely the latter seeing a few examples makes way more sense than trying to decipher a wall of text right and that same principle applies to llms now let's get into the realm of more advanced techniques bring on the advanced stuff I'm ready to level up my prompt engineering skills one powerful technique is system prompts it's all about setting the overall context for the llm so instead of just giving a specific task you're giving it broader instructions like to guide Its Behavior you got it you might tell it you are a

helpful assistant or you are a factual encyclopedia it's like setting the stage for the interaction that makes sense it's like giving the llm a Persona to play exactly and booner shows how you can even use a system prompt to force the llm to Output its responses in specific formats like Json ooh I can see the value in that having that structured data would make it so much easier to use it responses in other applications absolutely and it can help reduce those hallucinations too those moments where the llm makes things up goes often to inaccuracies hallucinations

that's such a great term for it it's like the llm is having a vivid dream and can't tell it's not real it's a great analogy and giving the llm that structure that format like Json can help keep it grounded in the real world so system prompts set the context what other Advanced Techniques should we know about there's one that's even more fun role prompts you give the llm a personality to take on this sounds like we're getting into acting an improv now what kind of roles are we talking about here honestly the possibilities are endless

you could have it be a travel guide a code reviewer a standup comedian even a historical figure Boonstra even suggests messing around with different tones of voice so you could have a sarcastic travel guide a brutally honest code reviewer Shakespearean stand of comedian I love it exactly it's like giving the llm a script which can lead to some really creative and entertaining results okay but wouldn't there be a risk of the llm getting too caught up in the role like forgetting about the actual task that's a valid concern which is why even when you're giving

it a role you still need to provide clear instructions and context you wouldn't want a travel guide llm giving you New York City tips when you asked for Amsterdam that would be a disaster so yeah context is key even when we're encouraging the llm to be creative absolutely and that leads us to the third type of advanced prompting contextual prompts all about giving the llm specific background information for the conversation or task so giving the llm just enough backstory to understand what we're asking exactly imagine asking a friend for movie recommendations you wouldn't just say

recommend a movie you tell them what genres you like what you've seen recently maybe your mood that's true context helps people understand your needs and preferences and it's the same with llms giving them that context helps them give more accurate helpful responses so I've got three advanced prompting techniques system prompts for setting the scene role prompts for playing a part and contextual prompts for giving background info got it you got it now are you ready for some even more mind-bending techniques hit me with it I'm ready to see what other prompt engineering magic we can

unlock well there's a technique called step back prompting it encourages the llm to think about a broader question before tackling the main task so instead of jumping right to the solution we're asking the llm to take a step back and see the bigger picture exactly Boonstra has this great example where he's trying to generate a storyline for a video game level but instead of asking for the story line directly he first prompts the llm to think about engaging settings for video games in general that's clever it's like priming the pump for a more creative response

precisely by encouraging that brainstorming beforehand you can get a richer more nuanced storyline for the specific level this is so fascinating we're actually guiding the llm thought process in a strategic way what other Advanced Techniques are out there okay get ready for this one it's called Chain of Thought prompting or Coe for short this is where things get really meta because you're guiding the llm to explain its reasoning step by step like how a human would solve a problem this is sounding like some serious cognitive science stuff it is it's about mimicking how we humans

think think about solving a math problem we don't just leap to the answer we break it down into into steps explain our logic as we go that's so true show your work not just give the final result exactly and this approach can be super effective for llms especially with tasks that need logical reasoning you see llms they're trained on tons of text but they don't understand the logic of say math in the same way we do right so even though they can produce human quality text they don't always grasp the concepts behind the words exactly

that's why kipo is so cool it makes the llm think like a human mathematician breaking the problem down explaining the steps as it goes so we're giving the llm a crash course in human logic precisely and by doing that we can help it overcome some of its limitations and Achieve better accuracy especially with things like math problems wow this is mindblowing so we're not just asking for the answer we're asking the llm to show its work you got it and by making its thinking visible not only do we improve its accuracy but we also learn

more about how it processes information and solves problems this is amazing stuff any other Advanced Techniques that use this idea of guiding the thought process yes self-consistency which Builds on that Chain of Thought idea instead of just running the pumpt once you run it multiple times each time generating a different Chain of Thought So you're getting multiple solutions from the same llm exactly and then you choose the answer that pops up most frequently across those runs it's like taking a pull inside the lm's brain that's a great way to put it and this can actually

help reduce bias in the output I can see that if the lln keeps coming up with the same answer through different thought processes it's probably a more reliable unbiased solution exactly it gives us more confidence in the results now Boonstra also has a whole section on using prompts for things like writing code explaining it translating it between languages even debugging and reviewing code so it's not just about text we can use prompt engineering to work with code that's amazing it is imagine being able to generate code understand a complex next piece of code or even

find and fix bugs using an llm it's a game Cher incredible and Bo even shares a story about using llm generated code to rename hundreds of files and it actually worked that it actually worked moment is going to resonate with a lot of listeners especially anyone who likes automating those tedious tasks and having more time for Creative work totally you know we've covered a lot of ground here but it feels like we're just scratching the surface of what's possible with prompt engineering you're right there's so much more to explore Boer also lays out some great

best practices for making sure your prompt engineering is successful which I think we should definitely discuss next time let's say that for part two my brain is already overflowing with all this information we'll be back after a short break to dive into those best practices and see what else we can uncover welcome back let's get into some of those best practices bster talks about he really stresses the importance of giving the llm clear examples you know especially when we're asking it to do something more complex right like we're giving them some training wheels before letting

them loose on the big stuff exactly remember that F shot learning thing we talked about giving those examples is super effective but even then you got to keep things simple so no using jargon or assuming the llm knows as much as like a human expert nope and be super clear about what you want as the output tell the llm exactly don't leave anything up in the air so instead of saying give me some takeway I should say I need a bulleted list of the five most important takeaways something like that you got it Boonstra also

says it's all about experimenting there's no magic formula impr prompt engineering so try different things see what works absolutely try different input formats different writing styles even different llms like we're scientists running experiments exactly what works great with one llm might totally flop with another which brings us to another key point do mentation keep track of what works what doesn't so it's not enough to just play around we got to be organized right down our observations yep it's like keeping a lab notebook for all your prompt engineering experiments Boonstra actually provides a table template which

is super helpful for keeping track of everything it's so easy to get lost in all the experimenting but yeah if you don't document it it's tough to learn and improve and speaking of improving stay up to date on those model updates llms are constantly changing it's like the rules of the game are always changing can be frustrating in but exciting too for sure and this is where tools like vertex AI Studio are so helpful you can store test and document all the different versions of your prompts keeps everything organized so instead of having prompts scattered

everywhere you have a nice clean view of all your work exactly now Boonstra encourages us to think Beyond just getting text output especially for things like extracting data or categorizing stuff trying different output formats can be super valuable like that example we talked about with forcing the llm to Output in Json format exactly structured data it's just easier to work with it cuts down on those hallucinations keeps the llm on track like using the right tool for the job sometimes you need a simple answer sometimes you need something more structured right on and lastly booner

makes it clear that prompt engineering shouldn't be a solo thing get other people involved two heads are better than one especially when it comes to brainstorming those creative prompts you said it imagine having a bunch of people come up with prompts compared the results you learn so much from that like a prompt engineering party I love that before we move on let's just zoom out for a sec remember prompt engineering is an ongoing process always learning refining trying new things exactly create those prompts test them see what happens refine them and keep experimenting that's how

you get the most out of these amazing llms you know I'm starting to get it not just the technical stuff but the mindset too that Curiosity that willingness to experiment I love that and speaking of new from te remember those Advanced Techniques well Boonstra gets into how llms can actually help us with things like writing code explaining it even debugging it it's like having a coding buddy right there in the computer yeah think of it generating code understanding complex code finding bugs before they become a problem it's revolutionary it is and while we've been focused

on text prompts there's this whole other world emerging multimodal prompting multimodal prompting that's a new one tell me more it's like this instead of just text you use a mix of things to guide the llm wait so you're talking about using text pictures audio even video all together yep it's like giving the llm multiple senses it can perceive and respond in a much richer way this is blowing my mind it's like we're stepping into the future of how we interact with computers we are and while Gemini you know the model we've been talking about it's

mainly text focused there's another one called Gemini Vision that takes us to a whole new level Gemini Vision that sounds like something straight out of sci-fi what can it do well Gemini Vision it builds on what Gemini can do but it can take in input from images and videos too it can do crazy things like read text from a photo describe what's happening in an image even answer questions based on what's happening in a video whoa that is impressive it's like the lmm has eyes now exactly and this opens up so many possibilities for what

we can do with llms I can already imagine all kinds of cool applications like helping someone who can't see well understand what's around them or creating Interactive Learning experiences the possibilities are endless and this just shows how prompt engineering is constantly evolving pushing the boundaries of what we can do with llms it's an exciting time to be learning about this stuff that's for sure what new break shoes are coming next who knows but I think it's time for us to take a quick break and then wrap up our Deep dive in part three we'll leave

you with some final thoughts to ponder sounds good I definitely need a minute to digest all this amazing information okay we're back and I'm still trying to wrap my head around all the amazing things we've talked about prompt engineering has come so far it's mind-blowing to think about what the future holds it really is it's like we've been given these tools to unlock new levels of problem solving and creativity it feels that way yeah but as we wrap up this deep dive I think it's important to step back for a moment and reflect on everything

we've learned and what it means for us moving forward I agree we've covered so much from like the basic stuff how to write a good prompt all the way to those Advanced Techniques the ones that almost mimic how we think it's been a wild ride it really has yeah I feel like I have a whole new appreciation for llms now but I also have this sense of like responsibility you know we need to approach this technology thoughtfully and carefully absolutely prompt engineering is more than just getting a machine to spit out an answer it's about

understanding what these llms can and can't do and using them in a way that benefits everyone that's such a good point it's easy to get carried away with all the exciting things we can do but we can't forget that llms are still tools we need to use them ethically and responsibly exactly any powerful technology can be misused as prompt Engineers we have a responsibility to think about the consequences of what we're doing we need to build things that are beneficial not harmful well said so as we close out our Deep dive into this amazing world

of prompt engineering I want to leave our listeners with a challenge now that you have a better understanding of this technology how are you going to use it to shape the future of llms will you be a force for good creating things that solve real problems and make the world a better place or will you just focus on personal gain without thinking about the bigger picture the choice is yours the future of llms is in the hands of the people who are shaping this technology The Prompt Engineers use your knowledge wisely and let's work together

to make sure llms reach their full potential for good