I built the BEST Voice AI Agent with n8n & ElevenLabs

in this video I'm going to show you how to build this amazing voice AI agent using 11 Labs that will have the ability to access additional tools via this AI agent in nadn that can make independent decision to use what tool based on the user's request that's coming in from this 11 lab voice AI agent depending on the user request our AI agent inside naden will independently make a decision to retrieve information from whatever tool that it has access to including Vector database Google Sheets the ability to check available times for setting up an appointment create appointments has access to real internet live data via our Sur API and we can add much more to this based on the use case we have I'm going to do a quick demo of this first so that way you can see exactly how this thing works and how it can communicate and retrieve information by reaching out to n and this is part of the Deep dive series that I'm doing for voice EI agent using 11 labs for my school Community I'll put the link in the description so that way you can just basically import all of this data and these workflows directly to your nend instance by basically going here clicking on import from file and you can upload this the Json file that's at the bottom of each of these tutorials that are on this school Community Deep dive series for voice AI agents so this particular voice AI agent is basically a support or assistant for my AI Workshop AI agency that I have so basically we can embed this on our website and then it'll be able to have the ability to again set up appointments provide more information so let's go ahead and test this thing out so you see exactly what this thing is capable of so I'm going to go ahead and click on the test workflow so that way our web Hook is listening to the incoming information and then when I head over here and click on call this a agent hi I'm Jarvis how can I help you today hey Jarvis uh I wanted to see if you could provide me some more information about the school Community the AI Workshop School community and how can I join the AI Workshop School Community is a collaborative space where AI enthusiasts and automation professionals come together to share insights solve challenges and learn from each other you can join the community Through the AI Workshop website where you'll find options to become a member and start accessing exclusive resources tutorials and expert connections if you have any more questions or need specific details about joining just let me know uh actually I wanted to see if I can uh book an appointment for uh December 25th at 11:00 a. m. unfortunately there are no available slots for a consultation call on December 25th at 11: a.

m. would you like me to check for an available time on a different date or perhaps a different time on December 25th um yeah let's do uh at 12:00 p. m.

on December 5th you can book an appointment for 12: p. m. on December 25th as that time is available please provide me with your email address and phone number and I will go ahead and book the appointment for you um yeah my email address is jacksmith gmail.

com and phone number is is 510 335 2240 your consultation call has been successfully booked for December 25th 2024 from 12:00 p. m. to 12:30 p.

m. you can join the call using the provided Link in the email that you will receive shortly if you have any further questions or need assistance feel free to ask um yeah also wanted to know about um how AI Workshop can um help my business AI Workshop can help your business by streamlining operations and automating various tasks which allows you to focus on what truly matters our AI agents can handle day-to-day activities such as managing emails thank you Jarvis I appreciate it thank you that's all you're very welcome I'm glad I could assist you if you have any more questions in the future or need help don't hesitate to reach out have a great day all right so that was kind of the full demo uh just wanted to Showcase how this voice AI agent can have the ability to access these multiple tools depending on what the user asking again um there is a little bit of lag but we can play around with different large language models and I can explain how that could be improved and how uh we can add additional prompting to it to be able to uh make that conversation a little bit smoother but this again the whole point of this demo is to just showcase and show you how to connect and give multiple tools to this voice AI agent so as you saw in the demo uh initially I asked a question about the school community and on the prompt I'm going to explain how to direct uh the AI agent to reach out for a particular information and how to add additional tools to be able to uh give this AI agent the understanding that whenever it the user is asking something uh about something particular then which tool to use and then we can send that information to our uh AI agent on nadn and then this AI agent will be able to make independent decision as far as which tool that it has access to it could use to retrieve that information in our particular case when I initially asked it the question about the school Community it reached out to the vector database here because this is all of the information about the school Community is sitting right here and then afterwards I asked it to provide me the availability based on my calendar I asked it hey is the 11:00 a. m.

time slot available and it correctly said no it's not available so therefore it reached out to the correct tool checked the availability and then when it asked it to um book an appointment for 12:00 p. m. which that was an available slot it was able to understand that because it checked the availability and it saw that that uh slot is open and then it utilized the create appointment setting to create that consultation call with the email of the person and again it also said that it will send the uh appointment information via the email which it does I have tested it before I mean obviously this jacksmith .

gmail. com is just a uh email that I just said it but if you wanted to actually do this and I test it out with my own email it sends you a nice email with that confirmation so it's completely automated and then I've also given it this Google sheet tool which has the ability to utilize this from AI functionality to Dynamic dynamically grab the information the contact information and save it to our Google sheet for further processing in the future and again this is just a few tools that I've attached to it so this will just showcase is how you can actually utilize The Voice EI agent prompt system and Tool usage that it has to even attach more tools to this AI agent to make this voice AI agent very very powerful anyway so that's kind of like the overall um uh understanding or just of this how uh the process will work uh so let's go ahead now walk through and uh see exactly what the props looks like how to set up this voice a AI agent inside um our conversational AI voice agent by 11 labs and then what kind of tools uh to attach to this so that way you can call um additional tools based on the user request again this is part of the Deep dive series that I'm doing for my uh School Community called voice AI agent so if you're interested in uh walking through step by step and understanding how to build multiple different AI agents I go through step by step and explain every single aspect of this voice AI agent tool for 11 Labs how to connect um using web hook so if you're interested in that please make sure you join the community our community is great we have a great group of people who are all interested and are all like-minded and they're there to learn from each other collaborate with different projects we have calendar calls where you can jump in and ask any questions if you get stuck in any of these things that I'm building and then obviously all of the all of my AI automations including all of the Json templates are all available in the school Community as soon as you join so please make sure you join because this is a great opportunity for you to take advantage of this great tool Ed in and get a comprehensive understanding on how to learn and build amazing AI agents I'll put the link in the description hopefully I'll see there all right let's get back to this AI agent and walk through step by step and see the different aspect of this so let's go ahead and first I'm going to focus on The Voice AI agent and I'll come back to the NN here so let me go ahead and maximize this so the way to set this up is first of all you have to have an 11 Labs account make sure you have an account once you do that then you will come here to the conversational AI agent uh so this this is a newer product um and right now it's I believe it's on beta but I think it works really well right now I know vapy exists but I think 11 laps is trying to really uh make a statement in this market with this new product which again is incredible and I'm pretty sure it's going to get better and better over time and it most like most likely will dominate the market because it is very powerful uh but anyway so once you create this uh once you come to this agent section right here you can click on create an AI agent or you can come right here here and click on on this plus button and you can create an AI agent from here as well I'm going to click on this AI agent so there's multiple um templates that it already has depending on your use case so let's say if you're creating a support agent you can just click on this the difference between a blank template and a support agent or video game character or math tutor is that it has already prompts in there uh that are predefined to give you kind of a head start but we would start with a blank template so when you click on blank template you can just name this agent so I'm just going to say test one obviously I'm not going to walk through step by step otherwise this video is going to be like 5 hours long I did a video earlier also that I built a um uh support agent for a c Joe's Diner which was like an imaginary restaurant so make sure you check that out as well cuz that's kind of an introductory uh to this and it's a little less complicated because it has less tools access to it so this one is a bit more comprehensive because we're giving it multiple tools inside and end but anyway so once you click on that um uh um AI agent once you create it once you name it so this is where it will come initially this one for example I use that Matt tutor so as you can see U it gives you that first message saying hey hi I'm Matilda what shall we cover today and this is giving you a simple system prompt so let's go ahead and let me explain the rest of these sections and how you can build this comprehensive AI Workshop tool in my case and then for you you can build whatever you want right um so the first thing is most of the time you're going to spend on agent and voice here so on the agent section this is where you can really customize this thing and give it a lot more tools to be able to access additional thing outside of this particular app so the first thing obviously this is the first message this is how this voice agent will greet um the customers so I just said hey I'm jar hi I'm Jarvis how can I help you say you can put whatever you want here and it will exactly read how the first agent whenever a customer interacts with this the language the great thing about 11 lab is it has access to multiple languages so that way if you're building this outside of us or an English speaking country this will uh behave exactly just like it would but for testing purposes I would always suggest starting with English um so after the first so that first message the next thing is the system prompt so the system prompt is very important because this is how you can direct this uh voice agent and give it a Persona give it a personality give it a context of the conversation when to call the tool outside of um this voice AI agent and then also much more as far as how it can behave or how it's supposed to behave with your customers if you're putting this embedding this on a website so for mine I just said hey you're a support agent named Jarvis you're very friendly enthusiastic you work for a company called AI Workshop to help customers with anything they need Z is the owner of AI Workshop agency and AI Workshop School community and I have given this a main goal saying hey your main main goal is to convert the customers that are visiting my website my AI agency work uh uh website which is this right here right the its main job is to convert the customer to set up a consultation call with me if they have a business or if they uh are interested about the school Community I've given it the um kind of instruction to kind of give them the link to the school community so they can join so that's kind of the main Persona and the main goal that I've set via this system PRP with it uh we'll come back to this in a second because this is where I'm giving it access to additional tools and want to call out these tools but let me quickly go to the bottom here uh so for the large language models you have multiple um large language models that you can have access to the fastest one is the Gemini 1. 5 flash as far as the interaction with the customer however the problem with this is that it doesn't have it's not very smart when it comes to utilizing external tools so in my case I'm giving a gp2 40 mini because this kind of has that balance of uh being smart enough to call additional tools function really well and then on top of that it is fast that you can have a decent conversation and initially when you saw that lag like I said we can always play around with this as far as telling the customer whenever it's reaching out to an additional tool that hey uh let me look that out for you look up look up that information for you or I'll be right back you know so we can always do that to make the conversation natural for me I just basically set it um just because again for the demo purposes I want wanted to show exactly what this looks like uh with this plain prompt that we provided in the system prompt but anyway so that's the large language model it can select temperature don't worry about that limit token usage for now you can just Lo use it as it is uh the standard which is Nega -1 and then this one is kind of like right at 0.

5 uh knowledge base I've just uh given it my website's URL there's multiple ways you can add knowledge to this you can add files obviously you can add a URL or you can just add a plain text and if you wanted to know more details how to add files and plain text please watch my pre previous video because when I created that voice AI agent for that Joe's Diner um which is basically a voice agent that has the ability to book reservation for the restaurant I gave that information about the restaurants and the menu itself directly in this knowledge base via uh files and then also the text but in this case uh I'm just using my AI workshop. me which is my AI workshop's um URL right here all right so uh in the bottom so this is where this magic of tools comes in where this will we can give this voice agent the ability to call external tools via web Hook and the way to communicate this information back and forth between this 11 Labs voice agent and our NN is done via our tools here so the way you can add a tool to this voice AI agent is either via web hook or a client we don't have to worry about the client because we're going to be utilizing NN so therefore we're going to stick to the web hook so when you click on web hook the configuration has several uh parameters here the first one is the name you need to name this tool so that way whenever um a this voice agent is having a interaction with the customer we need to be able to prompt it to say Hey whenever this happens use this tool and in my case as I was saying before so after I gave it that Persona I said that if the customer asks anything related to the school community use the school wisdom to retrieve information and this school wisdom right here this is the name of the tool as you can see right here which has um the name in the description and you want to make sure that the name obviously matches the name that you put in the system prompt so the voice AI agent knows when to reach out to this tool um to retrieve additional information in my case like I said when I when I whenever a uh customer asks anything related to the school Community I set to reach out to this school wisdom tool and this thing I've given it a description saying use this tool to retrieve information about the school Community very simple very concise you want to keep it as simple as possible and in order to reach out to NN we're going to use this URL which is going to be our web hook URL uh the method is going to be post so this web hook URL is pretty much let me minimize this a little bit all right so the web hook URL there is this right here so this is a web hook note in order to uh interact with additional tools from the outside and it then has this great tool called the web hook tool and if you just click on ADD node and you search for web hook so this is the trigger right here right so what I'm doing is basically let me double click on this um so let me maximize this a little bit so I have more room all right so here the test URL versus production URL so obviously if this is a uh production application then you will just click on production URL and the difference will be that this little test uh will disappear so if I click on this as you can see right here the test I way but if I click on test URL it goes back to the uh web hook- test so this is basically the URL that I'm copying and post pasting it right here that's that's it right so the method here is going to be HTTP post again same thing and then I'm going to come back and explain uh the rest of this here so that's pretty much the uh configuration when it comes to this specific tool now in the bottom so uh this the tool invoice eii agent has multiple additional things that you can send um as a payload to additional um to third party application in our case and then so you can send headers so if you wanted to send any kind of you know authentication you can send headers in there uh path parameters you don't have to worry about that much query parameters so this is if you are defining some kind of a query in my case I have ident uh this identifier for my query parameter is request and I'm going to explain why that that's very important because we need to be able to get give the AI agent the ability to retrieve that information from this web hook and decide which tool to reach out to based on whatever is coming in but this query parameter again this is just basically saying hey School Community Information request uh because this tool is specifically is for school Community Information you also want to send body parameters you want to describe the body parameters here and I've said in a friendly way confirm if the customers is asking about information related to joining the AI Workshop School community so that's basically the description of the body parameters if you have a tool that uses a web hook you have to have body uh parameters enabled because you need to send that information back to the third party in our case end and you also need to identify what you're sending right and in my case I'm just B basically sending this data saying hey the identify is customer customer is interested in joining this cool Community very very simple right so you can uh add additional properties if you want to send more body parameters you can just click on ADD property and you can send more body parameters here but I'm going to get rid of this for now that's good enough and cancel and then you will just save the changes so the next tool is my get available slots so here let's go back to our prompt here so here I'm saying that if you determine that the customer is leaning towards booking a free 30-minute consultation call you can check the availability using the get available slots tool again I'm calling the tool right there from here so make sure that your naming is correct so uh I said get available slots this has to match right here so I'm giving this tool again this is web hook again um get available slots use this tool to check the availability for the requested date and time for the appointments right very very simple uh same thing as far as the method it's going to be post the URL is going to be the same thing uh no headers query parameters I'm keeping the request ident the identifier as request and I'm going to explain why because we need to be able to um uh identify the request that's coming in what regardless of which tool it's being sent from description same thing I'm saying check uh available times for appointments the body parameters I'm just sending a name and I'm not making this required if you click on required then your voice EI agent will ask this um information before continuing with the process all right so but in our case this is good to go again a lot a lot of this information that I'm sharing obviously there's there's a lot more to it so that's why I said I've done that deep dive topic for a reason my school community so if you get a little confused just make sure you go there it's like a six uh tutorial series that you can watch everything and it will make more sense anyway so book meeting same thing I am just sending additional body parameters right here same but as far as information is the same thing I'm sending the requested date of the appointment uh email of the customer just like you saw uh time and phone number as well okay so those are all the tools and you can see on the top here I've said once you've agreed upon a time to meet you can use the book meeting to book a call once the booking is confirmed let the customer know that they'll receive an email with the details of the schedule call right so you're always always U making sure that the system promp you're providing is pretty crystal clear all right so now let's switch to our end here and see what happens here so as you can see our main web hook obviously is coming in from our our the information is coming in from our voice EI agent from 11 Labs directly to our web hook here let me zoom in a little bit all right there you go so let's jump into this web hook so inside this web hook um as you can see the authentication I've set it as none path you can Define respond this is very important you want to make sure that you you're using this use respond to web hook Nook because we need to be able to send this information back to our um AI voice AI agent right and as you can see this is the information that's coming in so this is the Json view you got all this header coming in uh you have PRS you have the query so that was the query that we defined earlier the request is book a consultation call right the body and here's all that information that we're sending on the body parameter from our voice AI agent as you can see the date of the appointment the email time phone number and name of the person requesting the uh information or requesting the appointment so once that information is coming in now we're uh connecting this thing to our AI agent this AI agent has the ability to grab all of this information from our web hook here so the user request that's the query right the request is book a consultation call the body so this is the information that's coming in uh via the body parameter when the uh user is requesting for an appointment right so that's why I put this or sign uh this two little slashes or these um things which basically means that if the information doesn't exist or if the information wasn't provided then you know it shouldn't throw an error but if it does you will basically absorbing that information so the the system prompt system prompt again this is just basically giving it all of just like a regular EI agent from NN is if you have not familiar with that please make sure you watch my other videos that I explain exactly how to do all these but inside this AI agent I'm just saying you are a customer support agent designed to help you handle various tasks efficiently your primary role so you're giving it a primary role and then I'm also saying you have access to multiple tools like the availability use this tool to check available times for appointments the appointments Vector store uh Vector store tool use this tool to retrieve information about anything related to the school community so this is where we're giving this AI agent inside and it then that instruction when to use this tool same thing I'm using saying use Google sheet use this tool to store customers contact information right because we want to retrieve and store our contact information for our customers in case we need to reach out and do more things but in general it's always good to keep your uh customers contact information for further uh use case in the future Sur API I'm just giving this Tool uh we didn't use that here but you can always give this uh The Sur API tool to be able to retrieve inform information if the customer is asking something related to a uh event that's real time anyway so that's kind of like the overall system um uh prompt right here so once that's done pretty much you are done from there I'm attaching an open a model I'm doing gp4 mini the tool is you know the Sur API here the appointment same thing when the appointment is being created so uh you can add this Google Calendar as a tool which will give you the ability to use this from AI functionality to be able to pull information directly from the request here or directly from the noes that are coming in from previous all right so that's the appointment tool availability same thing we're just uh giving it the um availability calendar or our Google Calendar here to check uh if there's time slots available for our requested time in our case you know it was able to correctly identify initially when I asked it hey is there time available for me to for book a consultation at 11 it said no because it had uh access to this availability or Google Calendar uh tool to retrieve that information and send it back to the user Google Sheets same thing I'm just using this from AI uh function to grabb the name the email and the phone number of the customer that's coming in so that's pretty much it and then obviously our Vector store tool so this is just I inserted a bunch of um information about AI Workshop by um let me go ahead and show you that by simply adding this uh Google drive to download this and I'm sending this information to this Vector Store Pine con Vector store database where I'm inserting that document about uh the frequently asked questions about my AI Workshop agency so all of that is inserted and then we are just basically adding a vector store retriever tool here to access all of that information that's sitting inside our Pine con Vector database so once all of this is done we're uh using this respon to web hook um to send this information back to our voice AI agent so here I am basically saying respond to web hook so this is uh a node that naden has I'm seeing respond with the Json below here all I'm doing is sending the status information back to our voice AI agent and I'm stringifying the information that's coming in from our AI agent output in the previous node this is very important so just you can copy and paste this or you can just use chptr clot to tell you how to uh stringify meaning that to remove some of these special characters so that way you don't need to run into an error this is one way again this doesn't necessarily mean it's the best way to do it uh you can always add additional nodes so here you can add a set node to clean the data that's coming in from your initial web hook you can add another set node here to be able to clean the data that's coming from outside or as an output of this AI agent to send back to your voice AI agent to 11 Labs so there's multiple things you could do and then obviously you can also add additional tools here so let's say if you have a contact database that you want to retrieve information from or if you have a CRM database to be able to uh add the information that's coming in from voicing AG so there's a lot of things you can do here so that's what makes this things very very powerful and like I said in my case I'm just showcasing the whole point of this is to Showcase uh the ability for this voice AI agent from 11 laps to send information back to nadn and and then stores that information or retrieves that information via this AI agent sends information back and you can have this natural conversation with a customer that can do multiple things right so uh on the next tab here the voice obviously I'm using Jarvis this is just for demo purposes obviously you don't want to clone somebody's voice that you don't have the uh rights to um have access to again this just for demo purposes I'm a huge fan of Jarvis and I created that Jarvis voice agent and my other video that was very popular so that's why I just kept it here but you can obviously add um voices that are inside 11 Labs you can clone your own voice or you can do whatever you want so flash here this is their new recommended model for low latency it just means that it can um have really natural conversation and fast conversation however I haven't really tested this to see if this works really well with tools but we can obviously you can always test those things out TTS output format so this is just um this 16,000 Heartz this is good enough but if you want to play around with the other ones you obviously can but it will probably cost more because obviously the more powerful the output format is the more credits you're going to use pronunciation you can even add pronunciation dictionaries on how to pronounce certain uh uh replacement for AI agents or replacement for this agent responses uh again feel free to play that play around with that you can add files there for dictionary uh don't worry about these things you know if you're cloning eVoice this is where you can adjust or parameters there analysis this is very also another very important and Powerful tool you can actually uh collect data from this interaction that this voice agent has with the customer by providing the identifier so let's say if you wanted to provide uh um grab the name and the email of that customer doing this interaction you can certainly do that here you can even have an evaluation criteria where you can customize and evaluate the conversation um that this voice EI just voice AI agents is having with your customers or for whatever use cases and then you can retrieve that later on once I build more powerful AI agents cuz I'm planning to build more with this I will definitely utilize these different tabs that are here Security in advance again this is for kind of more complex uh AI agents that you want to build and obviously when I work with clients and build customize voice agents for them this is where a lot of times I utilize these things based on the uh uh process that they need the widget tab right here so this is where you can embed this widget right here on your website you can customize the look and feel of this you can the appearance here you can have a different background color um you can change the look and feel of this little Avatar right here you can even attach a picture if you want but this one for example you can attach the colors here change the colors you can even add text context here like for example here it says need help start a call you can actually change all of this depending on what your use case is right so there's a lot you can do here but I'm going to go ahead and clear this cuz I don't need to change anything and the only thing you need to do is they provide you this nice little code all you have to do is just copy this paste it on your website or uh uh then you can have this little widget that will pop up and it will be very um powerful as far as how it can interact and give your business the ability to um you know have this natural conversation with this AI agent um and again depending on what your use case is you can really build very customized versions of this for businesses or you even for your own personal personal um website or personal portfolios all right so um one thing I want to point out this is still a bit expensive as you can see in the bottom it says in development calls are built at rate of 500 credits per minute I'm pretty sure these are guys going to lower this because right now this is expensive compared to something like a VAP but I would imagine as this thing becomes better and better as this moves out of the beta phase they will be able to reduce this uh credit um to be cheaper because they want to make sure they want to stay competitive so I'm pretty sure they're going to reduce that amount and they might add more uh tools they might add more abilities for this voice agent because again the future of AI agents definitely voice AI agents because you want to have that ability to naturally conversate with these agents to retrieve information and make it more and more powerful all right well hopefully you found this helpful all these prompts from The Voice AI agent and then also the um agents right here the workflows are all inside the school the classroom section so all you have to do is just go ahead in there in the Deep dive topics just grab uh Click on each um AI agent.