An AI Prompt Engineer Shares Her Secrets

so in the short time I have with you all today I want to demonstrate how smart prompting leads to Smart outputs and hopefully give you a few practical tips and techniques that you can take away and use for your own prompt creation process so I work at autogen where we help organizations around the world write more winning bids tenders and proposals by leveraging large language models and linguistic engineering I get a lot of questions around what prompt engineering actually is and what I do in my role so I thought it' be useful to clarify that

can I get a quick show of hands who here has used chat GPT or any other similar platform for their prompting wonderful everyone um perfect so what most of you have been doing in those instances is called prompt crafting prompt crafting is when you interact Real Time with a model and give it a prompt for that individual instance you receive useful and relevant responses but you wouldn't necessarily expect that prompt to work on any other piece of text that anyone else would use so prompt engineering is curating prompts which produce replicable reliable outputs to fulfill

a specific function whilst continuously objectively measuring and improving them it's about setting up Frameworks that scale well in the future with any unknown input all the time there are many prompting techniques out there but these are some of the most popular today I will demonstrate a few of these as we consider their benefits and their drawbacks and we'll do this by focusing on one task which is extracting and classifying things from a data set I'll be using autogen eyes platform today to demonstrate these prompts so the feature that I'll use it offers multiple options um

of outputs for each prompt so you'll see three options on the right hand side up here we have a zero shot prompt it's an instruction with no examples it's what everyone does the first time they uh interact with the larg language model it works well most of the time but it does have a few drawbacks sometimes it can lack a nuanced understanding of the task we're trying to achieve and we'll see that in practice today so in this example I've Ved to classify uh a piece of text and I've that piece of text has a

debatable output so as a human I would expect the product arrived late but the quality is excellent to generally be a positive statement although it has some positive and some negatives in it it's weighted heavier on the positive statement we can see that the model has uh given neutral for all three outputs so for this case zeros shot prompting doesn't encourage that nuanced understanding and I probably wouldn't trust it for even larger sets of data here the model lacks understanding of what I mean by each of those sentiments what does positive mean for me what

does neutral mean for me in the previous example I didn't have enough context so one way of providing it with more context is offering examples of what you want this this is called multi-shot prompting a shot being an example Chain of Thought prompting means asking the model to think step by step and show its reasoning so I've given it pretty much the same prompt but I've given it three examples of what I think a positive negative and neutral statement is you can see as we uh transform the text the outputs are far more Nuance the

first one says positive the second one after explaining its uh uh chain of thought where I've instructed it to think step by step also concludes to be positive in the end it's far closer to what we're looking for one thing I will say is that you should be wary of bias with multi-shop prompting in this example the model might take it to mean that positive statement must always talk about product quality or it always must mention that the site was confusing or that it always has to have one positive and one negative side to it

if you're using multi-shot prompting for larger sets of data you have to make sure your examples cover all bases this can be quite difficult to do as you have to think of every way that something can be interpreted Chain of Thought also helps in this scenario because if I can see the model's thought process I can see where it went wrong and it can help with my model debugging multishot here was really great for improving this task and giving the model some more understanding of my intention but sometimes I want to do something a bit

more complicated and I don't want it to give me just a one-word classification for something let's look at how we can introduce even more complexity with some prompting techniques prompt chaining or multi-step prompting is best for complex reasoning tasks that cannot be instructed in one go it ensures you're working on the best piece of text at each stage and it doesn't leave room for model inconsistency it makes sure that the potentially conflicting instructions don't interfere with one another let's say I want a more complex analysis of some sentiment on a larger body of text as

a human I might break this process down into a few different steps first classifying statements as a whole then extracting themes from the statement and then grouping those themes this type of breakdown also works great with prompting so for the sake of brevity I'm just showing the output of the first prompt that I gave the model the uh The Prompt instructed it to classify a list of customer feedback into sentiments using the same multishot prompt that I showed you guys before except on a larger set of data there's around 25 26 pieces of customer feedback

here this is the first prompt in this chain the second prompt in the chain is a really good zero shot prompt up at the top here you can see we've asked it to identify all the themes in the following list of customer feedback your response should be an exhaustive list of accurate and relevant themes with a number beside them indicating how many times it appeared do not write the same theme out twice feedback colon so I think it's important to note uh some key parts of that prompt so I've asked it to identify all the

themes and then I've repeated myself and said it should be an exhaustive list repetition is always good I've said that it's a list of customer feedback which provides the model with context of what I'm talking about I've asked it to be accurate and relevant so it's clear that I want the themes only to do with customer feedback and then I've said how I want the structure to look you can see with all of the options the structure is exactly how I want it to look the answers are exactly what I'm looking for so I'm happy

with this and we bring it over into the editor so as we move on to the third prompt in the chain you can see how with each step the output is a combination of everything that came before it context which is implicitly woven into the answer gets carried onto the next prompt The Prompt that I've given it here the third prompt in the chain is classify the following themes into positive negative neutral or other categories the response should have the sentiment as a heading followed by the themes which fall under that sentiment under each theme

Write a brief justification of why it falls under that sentiment and then I've given the list of themes as you can see on the right hand side this is far more nuanced accurate and useful than the first uh thing we looked at this is this output is actually a combination of all the techniques we've looked at so far and we could not have gotten this type of information which is this thorough and this accurate through only using one of the techniques once you you've got an output that you're happy with the possibilities really are endless

you can translate it into Jason you can play around with a tone you can turn it into a PowerPoint presentation and much much more so to conclude this short segment Simplicity is always best although a single shot prompt wasn't nuanced or accurate enough for our example today it is most often the best choice the last three examples I showed you of turning it into Jason or translating the tone all of those used a really good successful zero shot prompt a prompt should be direct unambiguous and relevant and each of the techniques we've gone through today

are just different ways of ensuring that the prompt meets those requirements I really hope this offered some insight into what prompt engineering actually is and gave you some tools that you can take away and Implement into your prompt crafting I think we might have time for one quick question if there are any questions in the audience yes we have someone here how do how do you think about it and do you use any tools to refine your prompts and how yeah yeah so that's that's actually a really interesting question um you can prompt models with

how you want them to refine your prompts interestingly so you have to have the techniques in order to instruct the models and how you want your prompt to look anyway sometimes if I'm lacking inspiration on how to begin writing a prompt I will definitely use a model I'll be direct unambiguous relevant clear about what my parameters are clear about what my instructions are and often it gives me really good uh framework the only problem with models are it's sometimes a bit more Nuance in that and you're looking to write prompt for a specific use case

so I know my target audience I know my my customers and I know what they're looking for and I can try put that into prompting but ultimately it's your subjective opinion on whether you think it meets these metrics that you set out so yes absolutely you can use other models not only clae but any of the providers um to give you a first draft of a good prompt and to better your own prompts awesome I think that's all the time we have today but if you have any more questions I'll be around afterwards and please

feel free to add me on LinkedIn and ask me any questions there thank you so much