I Was Prompting AI Agents Wrong... Until I Learned This (Live Prompting Examples)

building AI agents and hooking up different tools is fun and all but the quality and consistency of the performance of your agents directly ties back to the quality of the system prompt that you put in there if you guys don't know me my name is Nate I've been building agents for a while now and I also run a school Community that's all about building AI agents with NN we've got different Deep dive Topics in there like building AI agents Vector databases apis and HTTP requests and also doing step-by-step builds of all the videos shown on my channel and if it's something you're interested in check out the link in the description anyways today what we're going to be talking about is what actually goes into creating an effective prompt so that your agents perform as you want them to I'm going to be going over the most important thing that I've learned while building out agents and prompting them that I don't think a lot of people are doing so let's not waste any time and get straight into this one all right so I've got a document here if you want to download this one to follow along or just have it for later you can do so by joining my fre School Community the link for that's down in the description you'll just click on YouTube resources and find the post associated with this video and you'll be able to download the PDF right there anyways what we're looking at today is how we can Master reactive prompting for AI agents in naden and the objective of this document here is to understand what prompting is why it matters develop a structured approach to reactive prompting when building out AI agents and then learn about the essential prompt components so let's get straight into it and start off with just a brief introduction What is prompting make sure you stick around for this one because once we get through this doc we're going to hop into NN and do some live prompting examples So within our agents we're giving them a system prompt and this is basically just coding them on how to act but don't be scared of the word code because we're just using natural language instead of something like python or JavaScript a good system prompt is going to ensure that your agent is behaving in a very clear very specific and a very repeatable way so instead of us programming some sort of python agent what we're doing is we're just typing in you're an email agent your job is to assist the user by using your tools to take the correct action exactly as if we were instructing an intern and why does prompting matter I'm sure by now you guys already have a good reason in your head of why prompting matters and it's pretty intuitive but let's think about it like this as well agents are meant to be running autonomously and they don't allow that back and forth interaction like Chachi BT now yes there can be some human in the loop within your sort of agentic workflows but ideally you put in an input it triggers the automation triggers the agent to do something and then we're getting an output unlike chat gbt where you ask it to help you write an email and you can say hey make that shorter or you can say make it more professional we don't have that um luxury here we just need to trust that it's going to work consistently and high quality so our goal as prompters is to get the prompt right the first time so that the agent functions correctly every single time it's triggered so the key role here is to keep the prompts clear simple and actionable you don't want to leave any room for misinterpretation um and also less is more sometimes I'll see people just throw in a novel and that's just obviously going to be more expensive for you and also just more room to confuse the agent so less is more so now let's get into the biggest lesson that I've learned while prompting AI agents which is prompting needs to be done reactively I see way too many people doing this proactively throwing in a huge system message and then just testing things out this is just not the way to go so let's dive into into what that actually means to be prompting reactively first of all what is proactive prompting this is just writing a long detailed prompt up front after you have all your tools configured and all of the sort of you know standard operating procedures configured and then you start testing it out the problem here is that you don't know all the possible edge cases and errors in advance and debugging is going to be a lot more difficult because if something breaks you don't know which part of the prompt is causing the issue you may try to fix something in there and then the issue originally you were having is fixed but now you cause a new issue and it's just going to be really messy as you continue to add more and more and you end up just confusing both yourself and the agent now reactive prompting on the other hand is just starting with absolutely nothing and adding a tool testing it out and then slowly adding sentence by sentence and as you've seen in some of my demos we're able to get like six tools hooked up have no prompt in there and the agent's still working pretty well at that point we're able to start adding more lines to make the system more robust but the benefits here of reactive prompting are pretty clear the first one is easier debugging you know exactly what bro the agent whether that's I added this sentence and then the automation broke all I have to do is take out that sentence or I added this tool and I didn't prompt the tool yet so that's what cause the automation to break so I'm just going to add a sentence in right here about the tool this is also going to lead to more efficient testing because you can see exactly what happens before you hard prompt in fixes and essentially you know I'll talk about hard prompting more later but essentially what it is is um you're basically seeing an error and then you're hard prompting in the error within the system prompt and saying hey like you just did this that was wrong don't do that again and we can only do that reactively because we don't know how the agent's going to react before we test it out finally we have the benefit that it prevents over complicated prompts that are hard to modify later if you have a whole novel in there and you're getting errors you're not going to know where to start you're going to be overwhelmed so taking it step by step starting with nothing and adding on things slowly is the way to go and so if it still isn't clicking yet let's look at a real world example let's say you're teaching a kid to ride a bike if you took a proactive approach you'd be trying to correct the child's behavior before you know what he or she is going to do so if you're telling the kid to keep your back straight lean forward you know don't tilt a certain way that's going to be confusing because now the kid is trying to adjust to all these things you've said and it doesn't even know what it was going to do what he or she was going to do in the in the beginning but if you're taking a reactive approach and obviously maybe this wasn't the best example CU you don't want your kid to fall but you let them ride you see what they're doing you know if they're leaning too much to the left you're going to say okay well maybe you need to lean a little more to the right to center yourself up um and only correct what they actually need to have corrected this is going to be more effective fewer unnecessary instructions and just more simple and less overwhelming so the moral of story here is to start small observe errors and fix one problem at a time so let's take a look at some examples of reactive prompting that I've done in my ultimate assistant workflow as you can see right here I'm sure you guys have seen that video by now if you haven't I'll link it right up here but I did a ton of reactive prompting in here because I have one main agent calling four different agents and then within those sub agents they all have different tools that they need to call so this was very very reactive when I was prompting this workflow or this system of Agents I started with no persistent prompt at all I just connected a tool and I tested it out to see what would happened so an example would be I hooked up an email agent but I didn't give it in any instructions and I running the AI to see if it will call the tool automatically a lot of times it will and then it only comes to when you add another different agent that you need to prompt in hey these are the two agents you have here's when to use each one so anyways adding prompts based on errors here I have my system prompts you you guys want to pause it and read through you can take a look but you can see it's very very simple I've got one example I've got basically one brief Rule and then I just have all the tools that has and when to use them and it's very very concise and not overwhelming and so what I want you guys to pay attention to real quick is in the overview right here I said you know you're the ultimate personal assistant your job is to send the user's query to the correct tool that's all I had at first and then I was getting this error where I was saying hey write an email to Bob and what was happening is it wasn't sending that query to the email tool which is supposed to do it itself was trying to write an email even though it has no tool to write an email so then I reactively came in here and said you should never be writing emails or creating event summaries you just need to call the correct tool and that's not something I could have proactively put in there because I didn't really expect the agent to be doing that so I saw the error and then I basically hardcoded in what it should not be doing and what it should be doing so another cool example of hardcoding stuff in is using examples you know we all understand that examples are going to help the agent understand what it needs to do based on certain inputs and how to use different tools and so right here you can see I added this example but we'll also look at it down here because I basically copied it in what happened was the AI failed in a very specific scenario so I added a concrete example where I gave it an input I showed the actions it should take and then I gave it the output so in this case what happened was I asked it to write an email to Bob and it tried to send an email or Tred it tried to hit the send email agent but it didn't actually have Bob's email address so the email didn't get sent so what I did here was I put in the input which was send an email to Bob asking him what time he wants to leave I then showed the two actions it needs to take the first one was use the contact agent to get Bob's email send this email address to the email agent tool and then the second action is use the email agent to send the email and then finally the output that we want the personal assistant to say back to the human is the email has been sent to Bob anything else I can help you with the idea here is you don't need to put examples in there that are pretty intuitive and that the agent's going to get right already you only want to put in examples where you're noticing common themes of the agents failing to do this every time I may as well hardcode in this example input and output and Tool calls so step four is to debug one error at a time always change one thing and one thing only at a time so you know exactly what you change that broke the automation too too often I'll see people just get rid of an entire section and then start running things and now it's like okay well we're back at square one because we don't know exactly what happened so you want to get to the point where you're adding one sentence you're hitting run and it's either fixing it or it's not fixing it and then you know exactly what to do you know exactly what broke or fixed your Automation and so one thing honestly I want to admit here is I created that system prompt generator on my fre School Community um and really the idea there was just to help you with the formatting because I don't really use that thing anymore because the fact that doing that is very proactive in the sense that we're dropping in a sort of a query into chat GPT the custom GPT I built it's giving us a system prompt and then we're putting that whole thing in the agent and then just running it and testing it and in that case you don't know exactly what you should change to fix all issues so just wanted to throw that out there I don't really use that system prompt generator anymore I now always like handcraft my promp anyways from there what you want to do is scale up slowly so once you confirm that the agent is consistently working with its first tool and its first rule and its prompt then you can slowly add more tools and more prompt rules so here's an example you'll add a tool you'll add a sentence in the prompt about the tool test out a few scenarios if it's working well you can then add another tool and keep testing out and slowly adding pieces but if it's not then obviously you'll just hard prompt in the changes of what it's doing wrong and how to fix that from there you'll just test out a few more scenarios um and then you can just kind of rinse and repeat until you have all the functionality that you're looking for all right now let's look at the core components of an effective prompt each agent you design should follow a structured prompt to ensure Clarity consistency and efficiency now there's a ton of different types of prompting you can do based on the role of agent ultimately they're going to fall under one of these three buckets which is toolbase prompting conversational prompting or like content creation type prompting and then categorization SL evaluation prompting and the reason I wanted to highlight that is because obviously if we're creating like a Content creation a agent we're not going to say what tools it has if it has no tools but um yeah just wanted to throw that out there and another thing to keep in mind is I really like using markdown formatting for my prompts as you can see these examples we've got like different headers with pound signs and we're able to specify like different sections we can use bulleted lists we can use numbered list I've seen some people talk about using XML for prompting I'm not a huge fan of it because um as far as human readability I think markdown just makes a lot more sense so that's what I do anyways now let's talk about the main sections that I include in my prompts the first one is always a background so whether this is a role or a purpose or a context I typically call it something like an overview but anyways just giving it some sort of background that defines who the agent is what its overall goal is and this really sets the foundation of you know sort of identifying their Persona their behavior and if you don't have the section the agent is kind of going to lack Direction and it's going to generate really generic or unfocused outputs so set its role and this could be really simple you can kind of follow this template of you are a blank agent designed to do blank your goal is blank so you are a travel planning AI assistant that helps users plan their vacations your goal is to provide detailed personalized travel itery based on the user's input then we have tools this is obviously super super important when we're doing sort of non-deterministic agent workflows where they're going to have a bunch of different tools and they have to use their brain their chat model to understand which tool does what and when to use each one so this section tells the agent what tools it has access to and when to use them it ensures the AI selects the right tool for the right task and a well structured tool section prevents confusion and obviously makes the AI more efficient so here's an example of what it could look like we have like the markdown header of tools and then we have like a numbered list we're also showing that the tools are in bold this doesn't have to be the way you do it but sometimes I like to show them in bold um and it's you can see it's really simple it's it's not too much it's not overwhelming it's not too um you know it's just very clear Google search use this tool when the user asks for real- time information email sender use this tool when the user wants to send a message super simple and what else you can do is you can Define when to use each tool so right here we say we have a contact database use this tool to get contact information you must use this before using the email generator tool because otherwise it won't know who to send the email to so you can actually Define these little rules keep it very clear within the actual tool layer of the prompt and then we have instructions I usually call them rules as you can see um you could maybe even call it like a standard operating procedure but what this does it outlines specific rules for the agent to follow it dictates the order of operations at a high level just keep in mind you don't want to say do this in this order every time because then it's like why are you even using an agent the whole point of an agent is that it's you know it's taking an input and something happens in this black boox where it's calling different tools it may call this one twice it may call this one three times it may call them none at all um the idea is that it's variable it's not deterministic so if you're saying do it and this every time then you should just be using a sequential workflow it shouldn't even be an agent but obviously the rule section helps prevent misunderstanding so here's like a high Lev instruction right you're greeting the user politely if the user provides incomplete information you ask follow-up questions use the available tools only when necessary structure your response in clear concise sentences so this isn't saying like you do this in this order every time it's just saying when this happens do this if this happens do that so here's an example for AI task manager when a task is added you confirm with the user if a deadline is missing ask the user to specify one if a task priority is high send a notification store all tasks in the task management system so it's very clear too um we don't need all these extra filler words because remember the AI can understand what you're saying as long as it has like the actual context words that have meaning you don't need all these little fillers um you don't need these long sentences so moving on to examples which You Know sample inputs and outputs and also actions within those between the inputs and outputs but this helps the understand Expectations by showing real examples and these are the things that I love to hard code in there hard prompt in there because like I said there's no point in showing an example if the AI was already going to get that input and output right every time you just want to see what it's messing up on and then put put an example in and show it how to fix itself so more clear guidance and it's going to give you more accurate and consistent outputs here's an example where we get the input that says can you generate a trip plan for Paris for 5 days the action you're going to take is first call the trip Planner tool to get X Y and Z then you're going to take another action which is calling the email tool to send the itinerary and then finally the output should look something like this here's a 5day Paris itinerary day one day two day three day four day five and then I typically end my prompts with like a final notes or important reminder say section which just has like some miscellaneous but important reminders it could be current date and time it could be rate limits it could be um something as simple as like don't put any emojis in the output um and sometimes why I do this is because something can get lost within your prompt and sometimes like I I've thrown the today's date up top but then it only actually realizes it when it's in the bottom so playing around with the actual like location of your things can be sometimes help it out um and so having a final notes section at the bottom not with too many notes but just some quick things to remember like always format responses is markdown here's today's date if unsure about an answer say I don't have that information so just little miscellaneous things like that now I wanted to quickly talk about some honorable mentions because like I said earlier The Prompt sections and components varies based on the actual type of agent you're building so in the case of like a content creator agent that has no tools um you wouldn't give it a tool section but you may want to give it an output section so here's an output section that I had recently done for my voice travel agent um which if you want to see that video I'll drop a link right here but what I did was I just basically included rules for the output because the output was very specific with HTML format and it had to be very structured and I wanted horizontal lines so I created a whole section dedicated towards output format as you can see and because I Ed three pound signs for these subsections the agent was able to understand that all this rolled up into the format of the output section right here so anyways I said the email should be structured as HTML that will be sent through email use headers to separate each section add a horizontal line to each section um I said what it what should be in the subject I said what should be in the introduction section I said how you should list these departure dates return dates flights for the flight section um here's something where I basically gave it like the HTML image tag and I showed how to put the image in there I showed to make I said like make a inline image rather than um an attachment I said to have each resort with a clickable link I also was able to adjust the actual width percentage of the image by specifying that here in the prompt um so yeah this was just getting really detailed about the way we want the actual format to be structured you can see here we have activities that I actually misspelled in my agent but it didn't matter um and then finally just a sign off and then just some final additional honorable mentions something like memory and context management um some reasoning some error handling but typically I think that these can be just kind of one or two sentences that can usually go in like the rules or instructions section but it depends on the use case like I said so if it needs to be pretty robust then creating an individual section at the bottom called memory or error handling could be worth it it just depends on like I said the actual use case and the goal of the agent Okay cool so now that we've got through that document let's hop into Ed end and we'll just do some really quick examples of some reactive live prompting okay so I'm going to hit tab I'm going to type in AI agent we're going to grab one and we're going to be communicating with it through this connected chat trigger node now I'm going to add a chat model real quick just so we can get set up up and running we have our 40 mini we're good to go and just a reminder there is zero system prompt in here all it is is is that you are a helpful assistant so what's the first thing to do is we want to add a tool test it out so I'm going to add a um Google Calendar tool I'm just going to obviously select my calendar to pull from I'm going to you know fill in those parameters using the model by clicking that button and I'm just going to say this one's called create event so we have create event and so now we're going to do our test and see if the tool is working properly I'm going to say create an event for tonight at 700 p. m. so send this off we should see the agents ble to understand to use this create event tool because it's using an automatic description but now we see an issue it created the start time for October 12th 2023 and the end time for also October 12th 2023 so this is our first instance of reactive prompting it's calling the tool correctly so we don't really need to prompt in like the actual tool name yet um it's probably best practice just to just to do so but first i'm just going to give an overview and say you are a calendar actually no I'm just going to say you are a helpful assistant because that's all it is right now and we don't know what else we're adding into this guy but now we'll just say tools is create event just so it's aware use this to create an event and then we want to say final notes um here is the current date and time because that's where it messed up is because it didn't know the current date and time even though it was able to call the correct tool so now we'll just send this same thing off again and that should have fixed it we reactively fixed the a err and um we're just making sure that it is working as it should now okay there we go it just hit the tool and it says the event has been created for tonight at 700 p.

m. if I click into my calendar you can see right there we have the event that was just created so cool now that's working what we're going to do now is add another tool so we'll drag this one over here and let's say we want to do a send email tool we're going to send a message we're going to change the name to send email and just so you guys are aware like how it's able to know right here tool description we're setting automatically if we set manually we would just say you know use this tool to send an email but we can just keep it simple leave it as set automatic I'm going to turn on to subject and message as defined by the model and that's going to be it so now we just want to test this thing again before we add any prompts we'll say send an email to Bob example.