AI Cold Caller with Knowledge Base | Vapi Tutorial

if you have ever tried to give your voice AI caller access to a knowledge base where you can basically feed in custom PDF documents or maybe even website content you definitely know it's going to be technical luckily vapy has just released a feature where you can get your voice assistance both for inbound and outbound access to files and documents that you can just upload to their platform and can use directly within your voice AI assistant and all of that without programming anything yourself within a visual interface at this point in time the feature isn't really documented there's nothing in the docs I checked it all and I literally didn't find anything so all I'm going to show you now is basically what I figured out so far on how you can use that feature and how you can take advantage of that before everyone else so first of all I think the feature is super awesome as it eliminates the need for any external Vector databases that you need to implement into your system through external apis to get your Bot access to this is a solution that is directly integrated into vapi so you can definitely expect the best performance until this point in time I have used custom Vector databases using cudr for example or for our no code solution lovers I have used relevance AI to integrate those custom knowledge bases into my voice AI assistant to give them context but since now everything comes out of the box with inba you will not need to deal with anything like that which is just incredibly powerful now while this video aims to show you what I have figured out so far it is obviously not a complete tutorial as there is a lot of stuff that is just literally not out yet probably or it's not documented at least so in case you would like me to make a deep typ about this specific feature once it is actually completely documented and released definitely let me know Below in the comments would be highly appreciated and with that out of the way let's dive right into it so the first thing I'm going to show you is the actual feature itself obviously you definitely need to have an account on vapi if you would like to see it so all you had all you do is you head over to vapi go to their dashboard and within the dashboard you will now see a new option that is called documents which is located under the overview right here so if we are clicking on documents it looks pretty empty and there's not much to do except of that you can upload documents but you can simply just drag and drop off documents right within here that you can upload and obviously since it's not documented I just Tred to debug and check out what is inside the input field so what kind of formats they support and based on what chbd tells me they basically support markdown PDF plain text and the Microsoft Word types so both Doc and docx so all of those files can basically be uploaded directly inside of the VAP assistant and I have verified that as well so those really work and now to actually show you how you implement that into a static assistant first of all I'm going to go with you through an example in case you have not heard of a static assistant or a transion based assistance which are the both ways that bppi uses to create those powerful AI assistants definitely check out one of my previous videos where I explain and break everything down for you for now let's get that example running by simply going to the assistance Tab and creating a new assistant so in my case I'm just going to use the customer support agent as all I would like to do is literally test the documents so I click on continue I give this wonderful woman again the name of Lisa instead of Eva cuz it's just easier to pronounce on the phone so obviously I can now go through here and change all of the occurrences of the name Ava and change them to Lisa which I'm probably not going to do just for a couple of them so that's enough and now obviously the whole assistant is already predefined so if you have seen my previous videos or you know about vapi you know exactly what to put in here and how to fill it out and the feature we are looking for is basically located right within the model tab right here which is knowledge base so here you can select a document and basically adjust it and add it to your assistant and then click publish so for now let me just quickly publish that as I just adjusted the Master prompt and what I'm going to do now is I head to documents and I upload a document so the document I upload is basically something that I just created using jbt that contains the company details about about a real estate company that is called Bright Horizon realy it is something that I've used in my previous examples a lot and it is just a company I came up with that doesn't really exist and all we're going to do is now use this fictive data from that company to actually feed it into our voice assistant so all you need to do is you need to take the respective file you would like to upload in my case it is a text document and you can click on upload by the way this text document will also be available inside of my resource sub including all of the other information that you see right within this video if you have not heard of my resource up it is a place where I share files and anything templates whatever you see inside of my YouTube videos completely for free with any one of you so that you can literally just copy and use what I'm creating inside my videos inside of your business for your clients for whoever else you would like so that you don't need to build anything by yourself all right with that out of the way let's just click here on upload and upload our company document which just takes a little moment and as you can see here it tells you the size of it you can download it and it also shows you the ID which is going to be very interesting at a later point so after we have uploaded it we can simply head back into the assistant and now when we are clicking on the drop down we will see our file that we just uploaded right here so I can simply select it and I can click on publish so it says Arrow updating assistant which I assume is probably just a visual problem cuz I think if I refresh it this should still be connected and it is so it basically still works so I guess that's just like some visual issue that they have on the on the platform right now but we can ignore that apart from that there's another feature that they just released apparently which I haven't also seen before which is this detect speech emotion and I have read on the on the candy board earlier that this is basically something where when when you when you or the user that this voice assistant speaks with says something it uses sentiment analysis to actually get the emotion or the feeling of the person and append it to your actual prompt what this means is that you can actually design your prompt so that you can actually respond more emotional to the answers of whatever the user asks so in case he's angry it can respond in a certain way if he's happy it can respond in another way and all of that is just basically defined in prompting and I've Just Seen now that they actually have included a part of the prompt here as well that defines already that it can adjust the conversation in a specific emotional way which is great so that's just out of the box I just figured that out myself now which is great so that's not the purpose of the video though so let's just move on with what I actually want to show you so the document I have uploaded now contains the following content which looks like this so it has the name of my real estate agency it has some welcoming text it contains the business details and it also has some opening hours leadership Etc and you can see it says here Jane Smith is the CEO and founder and Chief Operating Officer is John Carter so what we are going to do now is since I have basically uploaded the document this assistant should have access to that information and to prove you that it actually has what we are going to do now is that we are literally just calling that assistant so you can look through here as well there is basically nothing mentioned about Bright Horizon realy in the first place cuz I literally just use the template of that assistant but if I now click on talk with Lisa we will be able to ask them questions and that's exactly what I'm going to do hello this is Lisa how may I assist you today hey Lisa who is the CEO of the company the CEO of Bright Horizon's realy is Jane Smith how many years of experience does he has I'm sorry I need more context to understand who you are referring to could you please specify which person you are asking about sure I'm talking about the CEO and founder how many years of experience does he have Jane Smith the CEO and founder of Bright Horizon realy has over 20 years of real estate experience so as you can see it basically has the context directly within here and you don't need to configure anything else there's basically no Custom Tool or custom function that you need to add to make this work everything will work right out of the box and it's actually pretty fast for the fact that it's actually a vector database in the back end so since I know this question will definitely be coming up I'm going to answer it right here right now and it's going to be how you can actually Implement website content right into this assistant using those documents and for now unfortunately there is no API endpoint documented so I'm not exactly sure if there's one that actually exists but I'm pretty sure there will be or very soon in the future so once that is the case I will probably make a video about it if you guys really want to see that but until then I'm going to show you a manual way on how you can implement this website content in a more structured way into your documents so obviously you need to have a file that you can upload here so what I suggest you to do is you pick a website of your choice where you want to get the information from or where the information where's the information located that you would like to implement so in my case I'm just going to use our agency website I copy the main URL and I head to a tool called to the web which basically allows you to extract website content from my website obviously this is just a free tool and it is manual which is sufficient for that case you can also use any other kind of tools that you can automate with make. com there are tons of web scrapers out there where you can get information from websites in a more visual way without all of the other context around of it so basically the HTML structures Etc so all I do is I paste my website URL within here select that I'm not a robot and I click on convert text and as you can see now it basically shows me the whole text of my website so what you can do is you can basically copy the text you can simply drop it into chat GPD and ask it to write it in a markone language and just structure the information more so that it's better accessible because as you can see right now it still is just text that is written down with a couple of line breaks so basically how it was inside of the website so let's just let's just say I'm going to copy that part of the website right here I can head into chat gbt create a new chat and now say something like like please structure the following content smartly and use markdown language so now I'm just going to paste my content click enter and what it does now it basically creates the markdown which again is just easier later on for chpt to read and as well for Vector databases to figure out the content because in my opinion I just like to have the things structured so as you can see it did it well so all you need to do now is you literally just copy that part you paste it into a text file that you find anywhere on your computer if you haven't created already one you save that content and then you are going to upload that content into the documents right within here and then you can use that information directly inside of the assistant as mentioned this is a manual work around until I know that there's actually an API endpoint available that I can use but for now this is sufficient now lastly to make sure you're not depending on the static assistance inside of vapi I will show you a way on how you can use those uploaded files in inside of a transi and based assistant that you can just use for inbound and for outbound phone calls the easiest way to make it work is using my custom make. com outbound framework that I created for you where you basically can edit your assistance directly inside of vapy in a static way but you can also use custom parameters to kind of simulate a transient based assistant which then runs through my smart automation framework to basically translate that static assistant into a transient based assistant so this whole framework support reports that custom documents right out of the box so you don't need to do anything else except of just work inside of vapy as we have done it already so far I'm going to link up the video up here if you would like to check it out this is where I'm basically configuring this framework so that you get an understanding of how you can simply implement it all of the resources are obviously again available completely for free inside of my resource app so you can simply head over there and get the files and get started by yourself in case there is any reason that you don't want to use my custom framework either because you have your very own platform for it your very own software solution I'm going to show you right here a manual way on how you can Implement them directly into the transient based assistant so that you can leverage that feature from any kind of platform wherever you would like to obviously there's not much of a documentation available so I'm literally just working with whatever I figured out myself here so please bear with me if I'm not able to cover every feature but I will try to get my best out of it and in that case it is probably enough to get you going on both inbound and outbound agent with those custom files using transion based assistant Json all right so the first part is basically figuring out how that structure internally looks and how we can get that information to do that it makes most sense to see what an actual Json based assistant looks like when we actually have set that information and to do that papy offers us a really cool endpoint that is called create assistant which you will find inside of the documentation under API reference and list assistant after you click on list assistant you will see a couple of values here which we can now use to actually make a request to and grab that assistant from within buy and this again in return just returns us the Json based assistant which we which we can then use again to see the values that are actually appended inside of that static assistant so that we can leverage them inside of our transing base assistant it sounds complex it's actually fairly easy the only thing we need for that is a tool that we can use to actually make API requests to and I'm going to use Postman if you have not heard of Postman it is basically a free tool where you can test apis and web hooks I will also link it up in my resource sub so you can simply sign up here and once inside of Postman you will see a view like this where you have a little plus up here which you're going to click which then again adds a new API request or a new web request in the first place so now we obviously would like to get the assistant so I'm going to I'm going to get assistant cu the list assistant actually returns us all of the assistant we don't want that we want to have a specific assistant so the get assistant endpoint is the right one and after you clicked here you will see up here a URL inside of the curly request you're just going to copy that one you head back into Postman you paste it right here you remove this ID parameter at the end which is where we need to define the static I assistant of the ones that we created inside of the vapi dashboard so we head simply to vapi we go to assistant we open our assistant which I already have here and you can see here there is a ID and I can copy that by clicking on that button I head back into Postman paste that URL right at the end and if I send now you will see we get a authentication error because we haven't authenticated ourselves to do that all you do is you click on authorization and you select bar token and you add your bar token right within here after you've done that you can literally click on send and it will retrieve that assistant for you so that you can actually see what is inside of it so this is basically the method I have used to figure out how that structure for that custom knowledge looks like and if you're scrolling down here now you will see within the actual model at the very end we have a new key called knowledge base and this is where we have all of this information available just to show you the other feature that I mentioned earlier if we go into the dashboard and I I enable this detect speech emotion I publish it if I'm going to refresh this whole assistant now we will also see a new key that basically defines this emotion recognition enabled so this one you can also set in case you would like to activate it and you would like to have emotion recognition directly inside of your prompts and the conversation obviously in this video I'm not going to deep into it but I want to just show you the knowledge base and how you actually configure it and what those values mean so as you can see here you have the knowledge base key which basically just defines that this is the information about that knowledge page where you basically can add your custom files you have top K which is a more technical parameter what that basically means is when the stuff runs through the vector database which is basically the logic that happens behind the scenes when you add some custom documents there are a certain number of results or vectors that are closest to each other whenever you ask something to the vector database and top K basically just defines the number of vectors that should be returned whenever you give it some input so it sounds complicated but you can imagine it like a Google search where you basically just type in a word or a longtail keyword whatsoever and it returns you a couple of results and topk would basically just Define the uppermost two results and in case you would change the number for three it would basically just return you the uppermost three results Etc and this information is then again used inside of the prompt for the assistant to access in the moment it is answering so by default I would say you can just leave it at two it is probably like the most common set I mean you can of course also increase it to three or four which again just helps you to cover a bit more of context but the arror rate could also be higher in case it is not properly done so I would just leave it at two if you're not exactly sure what works best otherwise you can OB Bally just try it out just set it to one set it to five whatever works best for you and where you get the best results then you have the file IDs which is nothing else than the IDS of the files that you have uploaded inside of the VY dashboard so if you go to documents right here you have the file ID up here which you can copy and you can literally just paste that one later on as well into the file IDs right here and then the provider is probably just the provider of the vector database so what they use I know canonical is a company that offers AI Solutions as well and I think like more scalable stuff not exactly sure I haven't really worked with them but provider obviously usually just defines the provider of whatever they use to provide you the service in the first place so in order to now use that response as a transient based assistant there are a couple of ways if you have seen my previous videos you probably already know that but I'm just going to copy that part and I paste it into a Json editor called Json editor online.

org it is a free tool you can simply go on the website and you have a an editor where you can literally just create Json visually so all I'm going to do now is I paste this Json here and if you have seen my debugging video you know exactly that you need to remove a couple of values to actually create an assistant that works so I just remove those fields right here and now we also have the knowledge bace right in here with all of our values so if I would like to now create an outbound calling agent or a phone call that calls someone else with that information that we have in here so basically within with a knowledge base all I need to do is I need to format that request depending on what the vapy API requires so to do that I simply head back over to vapi and I go to to the create phone call Api endpoint which is this one so we can now basically just fire a an API endpoint to a custom number and try it out and see if that if that assistant actually has access to the whole files and the knowledge base and to do that again we are going to use our Postman tool to create a temporary API request and we do that by clicking here on the plus and now we heading back into our documentation copying this URL right up here paste it into Postman and I think they require a post request so we're going to change the request type to to post and obviously you need to set up your authorization again where you choose the bar token which is what I'm going to do now so I've just selected that one and what you do now is you basically just take care of the payload which is where the whole logic and Magic happens so you can simply head to body you click on draw and you select Jason from the dropdown so you will have an empty Json right within here so if we're heading now back into the documentation you can see that right within the whole payload here they have a couple of keys that we need to Define which is either an assistant ID or an assistant by the way it's also documented right here so you can read through it in my case we are going to use the assistant cuz we would like to have a transion based assistant and not a static assistant like from the back end so in order to do that what I'm going to do is we simply head over to Json editor I'm going to create a new Json right within here and I call that assistant and I add some double points here and now I'm going to paste this whole assistant Json right within here I click on here on this button here to format it so that it's just a little more visual which is now exactly what we want here so now we have basically the assistant defined and I know for outbound calling there are a couple of other fields we need to Define so if you scroll here to the very end you can see that you need to Define either a phone number ID or a phone number which then basically means that you import this phone number into VY so that they can add their end points into twio if you use twio to actually make those phone calls in my case since I already have a phone number inside of Vy which is right here all I'm going to do is I use the phone number ID field and I create a new key up here with a double point I add my double quotes a comma at the end and now I'm simply going into VY I copy the ID of my phone number and I paste it right within the Json editor right here here so there's one more thing that we need to do for the transient based outbound assistant which is adding the actual customer which is this part so I'm just going to copy this part I head back into our Json editor click enter and paste it here I don't need the extension so I remove this part and I change the name to yanis so this is basically me I can now also make this whole thing Dynamic if I would like to so obviously if you seen my previous videos you know exactly how to make a dynamic and right within here you basically add your phone number so I'm not going to do this so to just not post my phone number but all you need to do then once you have added your number here you copy this whole Json and you head over to postman and you paste it into the body right here so I'm just going to add now my custom phone number and I will hide the field just so you don't see it so give me a second so I've just poed my phone number and all I'm going to do now is I click on send which then should make a phone call to my smartphone where we can see the transier based assistant calling so I'm going to make it on loud so that they can actually hear as well what it says and that it has actually access to the knowledge base that we basically connected to it all right I'm going click Send it sent the request so now on my phone we should see a call coming in which is correct so I accept it and let's talk to her hi there hello how can I assist you today hey who's the owner of your company the owner of our company is Jane Smith cool what are your opening hours on Tuesday on Tuesday our opening hours are from 9 a. m. to 600 p.

m.