The ULTIMATE Local AI Setup: LLMs, Qdrant, n8n (NO CODE!!)

if you wanted to run AI locally and build amazing AI agents all from your machine this tutorial is going to be for you we're going to build this amazing AI agent that's going to utilize all open-source AI tools including the quadrant Vector store that's an open- Source Vector database we're going to use several large language models with the AMA platform and we'll be able to interact with this AI agent using our chat model we'll be able to upload our own documents and immediately interact with it using the vector database this is going to be a great video you're going to learn a lot about how all these tools connect with each other all locally through your machine so you can build really great agents so make sure you stick around till the end because this is going to be a very very useful tutorial all right let's get started all right so first thing you need to do is head over to N8 's uh GitHub it's github. com nn- iio you will scroll down and you'll come here and click on the self-hosted AI starter kit so here they provide you all the guide depending on what machine you're using so if you for example have an Nvidia GPU go ahead and follow this but if you're using Mac then go ahead and follow what going to do down here which I'm going to use because I have a Mac so we're going to basically utilize this right here my goal is always not to use any code at all uh because obviously there's a lot of people who get a little intimidated when they see code so that's why I try to keep it as low as possible therefore we're going to mostly use the Docker desktop app um to be able to download everything we need but for this step we have to just clone uh this GitHub repo so all we have to do is copy that and come to our terminal here and all you have to do is basically paste it and this is going to download everything okay let me zoom in a little bit there you go all right so that got cloned so now the next step is you just need to CD into that folder that got uh created okay so I'm going to press enter all right so now we're inside that folder Let Me Clear again all right great um so before we move to the next step let me quickly uh pull up my desktop Docker app so if you see right here right now I have no images here and no containers so that way you can see that it's empty right now and then as soon as we uh go through and enter this we'll be able to uh see that all of those containers and images is going to show up on our desktop app so this is always a good way um to double check and make sure everything is getting loaded properly all right so now that we're inside this self-hosted AI starter kit the next step is to be able to CD into it we already did that and now we just need to click on Docker uh copy this one go back to our terminal and again let me pull up the desktop app here so you can see it side by side and paste so Docker compos profile CPU app this is going to run um all of the different Docker containers that's required for this AI starter kit in your profile through your CPU so you're just going to press enter and now it's going to run and pull the different containers and images that are inside this including the AMA postgress n8n um and then the quadrant Vector database as well all right perfect so it looks like we got everything as you can see on the right hand side we got all the four images that was inside that um starter kit and on the containers tab you can see right now it says the selfhosted AI starter kit and includes uh the quadrant postgress AMA and an perfect and then same thing on um our terminal here as you can see it downloaded everything that we need and right now it says Local Host discus 78 you can access this from there okay perfect so now we're pretty much done here okay so now uh let's make sure everything is running so if I yep it looks like everything is good to go here everything's running so let's minimize this for now we can copy um this or just go to Local Host 5678 so let me go ahead and do that I'm going to say Local Host 5678 if it's your first time uh this is going to pull up and ask you to sign up for an account um but if you have already logged in through this Local Host before then it's just going to ask you to sign in so I'm just going to go ahead and sign in okay you're going to enter your email and password and you're going to sign in all right so now at my brand workflow here so let's go ahead and change the name here to local AI kit all right so now the first step we need to add a chat trigger so let's go ahead and do that I'm going to search for chat and chat trigger perfect so here's one difference that we're going to have compared to other uh AI that I've built before uh in this one we're going to give our chat the ability to upload files so what you're going to do is come on the options click on ADD field and you're going to click on allow file uploads so what this is going to do this is going to give this chat the ability um to add documents or add files from right here on the right hand side as you can see now we have this little like plus sign with a document there because we will be uploading our own documents into the vector database through the chat model and I'll explain that a little bit more all right so now that's done all right so the next step is going to be adding our Vector database so I'm going to search for vector and as you can see right here there's several uh Vector databases but the quadrant Vector source so this is the one that we're going to utilize so quadr Vector store is a open- source uh Vector store that you can utilize uh so if you go to their website they show you all you can see all the details um they have uh you can click on pricing um and they have as you can see right here quadrant Cloud starting at zero so you can sign up if you want but we're not going to worry about it because we've already installed quadrant through our AI starter kit but it just gives you a good overview of what uh the this quaron Vector store is and why make it makes it different compared to other vectors so it's like the pine cone Vector score that I've used before all right so so your credential is going to be automatically loaded here because of the fact that we're running everything on Docker right so therefore it's going to have everything loaded there for the operation mode we're going to be inserting documents in this so we're going to click on insert um and for our card collection we're going to do by ID actually and we need to go back and actually test this real quickly so let's go ahead and attach this let me double check click there so now let's just go ahead and add a chat okay this is going to give me an error but that's fine I just want to um showcase how to add this chat in there so now as you can see on the left hand side I have the chat message received so what we need to do is we need to add our collection by the chat ID because we need to provide the ability for a user to upload their own document through the chat window so that's why we need to grab that session ID of that chat that's going to be inputed into this quadrant Vector store where before for example in my previous video that I did using the pine cone Vector database I added a uh step before this with Google Drve where we would grab the document from the Google Drve and upload it into the vector store but this like I said is going to be instantaneous meaning that a user can upload a file themselves and it will get added to the vector store and then they can interact with it all right so once you grab that that's good to go so we're going to get out of this all right so now let's add uh the embedding and document loader to this so you're going to click on embedding so you have several options obviously to add the embeddings but uh you can utilize the open AI but that's not going to be local of course but the whole point of this video is to be able to utilize all local resources so I'm going to use the embeddings AMA so again same thing your AMA account will automatically get uploaded here because you're using um the since it's all part of the same same container so it's going to pre-populate everything for you that's the great thing about this when you're using this AI starter kit through your CPU you don't have to worry about adding or connecting you new credentials all right so for the model I've already uploaded embedded models in this so in this account of mine I already have these but I'm going to show you exactly how to add a new embedded model from the AMA for because obviously you cannot use chat model here this has to be an embeddings model right so therefore we need to go ahead and add a new embeddings so let's say I want to add a new embeddings here obviously your uh when you click on this account if it's your first time this is not going to show uh so therefore I'm going to show you exactly how to add new models there so let's go back to our AMA so if you just go to ama. com right if you go to ama.

com and if if you click on models here this this is going to show you all the models that are available but again we need to grab an embedding model so you're going to come to the top and search for embedding models oh I think I spell that wrong all right perfect there you go so now you have several embedding models there's this one that's very popular by snowflake um or you can this is actually one of the most popular ones this noow embedded text and in the bottom as you can see it says the no make embedded text is a large context L text encoder that surpasses open AI text embedding 8 out2 and the embedding three small so again this is a very useful um embedding model that you should utilize because it's not that large and it's very very powerful but let's go ahead because I've already installed this on my uh AMA I'm going to show you something that I don't have that I can install so let me see here what do I have so I got the mxb embedded large and then the noic embedded text so let's go ahead and add something different go back here there you go okay let's see let me act the snoflake actually there you go this is a pretty good one but that's pretty big I don't want that let me add this 20 2 million parameter one okay all right so here's what we need to do we need to copy this all Lama pull snowflake Arctic Ed right so I'm going to copy this you can do this several ways the coding way is to go to your um Docker yml file so if you go inside your n8n starter kit and open it into in a text file you can come to the docker compos yml file and right here as you can see it's only pulling the Llama 3. 1 right so you can actually add another line here that basically says AMA pull snowflake Arctic and then you can rerun this and that way it will actually download that model for you but because like I said I always want to focus on uh using least amount of cod as possible so let me go ahead and get rid of this I'm going to show you the easiest way to do this is through your Docker desktop so I'm going to open my Docker desktop you're going to come to your containers tab if you're not already there you're going to come and click on as you can see right here amaama latest this is running so I'm going to click on the container for AMA you're going to go to exact right here and we will pull that from here so all you have to do is just paste it right here press enter and this is going to pull that embedded model or any other model that you want from um your AMA you'll be able to basically do the same thing grab that code uh make sure you're pulling it and then just basically say AMA pull and whatever model you want to pull you can add it here right so let's say I want to add another model here right let's say I want to add um a normal a regular model so for example let's let me add something very very small all right for example let's see let's say I want to add this Quin 2 by Alibaba so I want to add let's say this super small . 5 billion parameter right so all I have to do is it says AMA run Quin 2B right so just going to actually just copy this and go back to my desktop and say AMA pull and just paste that model press enter and now it's going to pull that large language model that exists inside the AMA uh platform so as you can see it's like the easiest way to do this because like I said you can use and change the um Docker file inside the starter kit but that you know that's going to require a little bit of coding and if you're not familiar with it then you're going to get lost so this is the easiest way to do this so now let's go ahead and double check here so if I get out of this double click on my embedding now as you can see I have access to that snowflake Arctic inat 22 million parameter and then also I added this uh Quinn 2.

5 billion parameter this wasn't there before right and you can double check go back to the video and make sure these two weren't there we we just added it through that Docker all right so now let's go ahead and add our embedding documents I'm going to click on that click on default data loader uh we're going to select the data type as binary load input all data automatically detect we're good to go there we need to add a token splitter so I'm just going to do a token splitter and have the chunk size be 550 okay and if you need more information about this again you can watch my previous video where I explained or the video that I made about um pine cone database where I explain these things in detail okay so for now I'm going to leave it at that uh so let's go ahead now select our I forgot to select our embedding model yeah just make sure uh so I know we downloaded all this but I kind of like this uh nomic embedded text latest this is um the one that we looked at earlier that said that it's more powerful than um open AI embeded models I'm just going to select that and it's a smaller one because I have a you know my memory in my computer is not that great so that works F but again you can select whatever you want okay so now let's go ahead and test this thing out so another great thing about quadrant Vector database is that you can actually check out the dashboard through your local host and the way you find that is if you go to your Docker and your containers quadrant right here it says it's running in 6333 so let's go ahead and do that go to Local Host and we're going to do 6333 SL dashboard so that's what you need make sure you're putting the/ dashboard there I'm going to press enter and there you go so this is your collections right now um this dashboard is empty there's no collections here because we haven't uploaded anything but let's go ahead and do that so now let's test this I'm want to go to chat now and I'm going to upload a file so I have downloaded this BTC white paper the Bitcoin white paper but you can do whatever you want so I'm going to say what is the Genesis block which is the original Block in the Bitcoin white paper so now if I press enter so now what this is going to do let me back out of here now it's going to split this and add this to the uh quadrant Vector database via this embeddings olawa model so now that's what's happening right now once this goes through and completes you will we will refresh the space and you'll see that uh a new collection would have been added so let's just wait for this to process if your computer's slow like mine this might take a little bit uh but if you have a um computer that has a really good memory and or if you're using a GP this going to go a lot faster but there you go all right perfect so now as you can see uh this went through and 13 item came came out of here so now let's go ahead and actually let me quickly first double check on this yep and as you can see right now this Bitcoin white paper that we just uploaded via our chat got converted into vector and to double check this now I'm going to refresh the page and as you can see we have um our Vector that got added here okay all right so that's perfect so let's go back okay so that step is done now let's go ahead and add our AI agent so what what I'm going to do is click on plus sign here go to AI agent so here what we need to do is because uh there are two inputs there's the quadrant Vector store input and the chat input but we need to connect this to our chat input right because we need be able to interact with the user based on the chat but we'll add a vector store retriever in this session to be able to have access to the vector Store to our document that got uploaded in the first step okay so let me go ahead and actually actually let's go ahead and add that first so just agent it's going to be not a tools agent this time it's going to be a conversational agent right so I'm going to click on conversational agent The Prompt you're not going to say take from previous node automatically because the previous node is a qun vector store what we need to do is say Define below and we're going to define the text right this is going to be coming in from our user from the first uh step from the chat input so you're going to click on the input here and grab the when chat message received you're going to go to Json and from here we're going to grab the chat input right because we want to make sure that we're understanding what the user is asking um all right so it says when chat message receive. item. json.

Chat input perfect so now we're good to go here let's go ahead and back out of this so now let's add a chat model so for chat model we're going to add our AMA chat model and here's going to be the difference so earlier in the previous step this step we added our embedding um and as you can see here let me go ahead and click we added this snow make embedd and text but for this one since this is a chat model we need to be able to add our large language model so you can do this Gemma 2B you can do the Llama 3.