How To Auto Update Your Knowledge Bases | FlowiseAI

4.9k views5510 WordsCopy TextShare

Leon van Zyl

In this comprehensive Flowise AI tutorial you will learn how to keep your Document Stores up to date...

Video Transcript:

In this video, I will show you how to automate the process of keeping your document stores up to date. Document stores are a great solution for managing all your data sources in a single container. This way, you can easily add or remove information from the knowledge base, without having to change the AI chat flow itself.

But one thing you guys have been asking me in the comments is how to refresh the document store automatically when a document loader uses dynamic data. For instance, you might be scraping information from a website that changes often, or you might be pulling data from a database, like Airtable. In those situations, you might want the document store to be refreshed as soon as the data is changed or like every hour, once a day, etc.

I really want to give you the best value possible in this video, so I thought it would be fun to build a real world application together. Here, we have an AI powered business directory. This is similar to the Yellow Pages, but instead of searching businesses using keywords, we can simply have a natural conversation with an AI chatbot.

The business listings are actually just stored in an Airtable database, but this could be any type of data source, like you could be scraping data from a website or pulling from a database. Now, the issue is if we add a new listing to this database, for example, let's add Oak End Barrel, which is a fictitious restaurant, we'll just enter some address over here, like so. Let's just enter some sort of phone number, let's enter a description like "a restaurant that sells steaks and sushi".

Now, I've committed the changes to this table, and if we go back to our chatbot and we ask something like "I'm looking for a sushi restaurant in Johannesburg", we simply get a message saying that it can't find that latest entry. And that brings us to the challenge that you guys have been facing with document stores. Whenever the data source changes, you manually have to execute the following steps.

And in this video, we will automate this process, but let me show you how this works. First, you have to go to Actions, click on Preview and Process, then we need to click on Process to load the data and chunk it, and now we get 17 documents back, which means that the latest entry has been pulled in. Now we have to upsell this data into the vector store by going to Upset Config, and then we need to click on Upset, and now we can finally see that that one record has been added and 16 records were skipped.

So if we go back to our chatbot now, and we ask the same question, we should get the correct response back, and we do. It's very possible that you might have a situation where you want to manually refresh the data, but in many instances, like with this business directory, we want the data to be fresh maybe once a day, or every few hours. So let's have a look at automating this process.

You are going to learn a lot, because in this video, I will show you how to set up the chat flow, the document store, and we will use Airtable as our database in this video. We will also use a workflow automation tool called n8n to run this automation on our behalf, and you will also learn how to use the new Flow-wise API endpoints for updating the document store. First, let's create our chat flow.

I'm using a cloud instance of Flow-wise, but you're more than welcome to use a local instance to follow along with this tutorial. On your local machine, simply open up the command prompt or terminal and enter npx flow-wise start. If this is your first time running Flow-wise, then you will be prompted to install Flow-wise, so simply enter y and press enter, and afterwards, you should see a message like this, and after that, you can access Flow-wise by going to localhost 3000, and you should see a dashboard like this.

Now let's create the chat flow by clicking on add new. Let's give it a name like ai-business-directory, and let's save this. Let's add a new node.

Let's go to agents, and let's add the tool agent node. Let's start by adding an AI model to the canvas. So under add nodes, let's go to chat models, and I'll be using the chat open AI model.

Now I've got plenty of videos showing you how to use chat olama to run models locally for free on your own machine, or you could use grokchat which is also free, and you'll also find the video on my channel for using grokchat. So let's add our chat model node. Let's attach this to our agent.

Let's select our credentials. Let's select our model as gpt40-mini, and I'm going to lower the temperature to 0. 2, as I want our assistant to strictly follow our prompt instructions, which we will set up in a minute as well.

Let's also add a memory node. So under nodes let's go to memory, and let's simply add this buffer memory node, and let's attach it to our agent as well. Now finally let's add a tool node.

So under add nodes, scroll all the way down to tools, and add the retriever tool, and let's attach this tool to our agent as well. This retriever tool will be responsible for connecting our knowledge base to our AI agent. For the tool name, let's call it business listings, and for the description let's enter use this tool to find information about businesses.

Now optionally we can also enable this toggle which will return the source documents in the chat window. Now because this process will be customer facing, I don't want the user to see my data sources, so I'll leave this disabled. Then lastly let's add a retriever.

So under vector stores let's add the document store node, and let's attach this to our retriever tool. Great! Now we simply need to select our document store which we haven't created yet, but before we do that I just want to go back to our tool agent and change the system prompt.

I'm going to clear this, but instead I'll paste in the following prompt. Let's expand this. You are a helpful assistant for a business listing service called AI Business Directory.

Only answer questions related to the business. Your role is to assist the user in finding information related to the businesses like their location, contact information, services, and products. Use the provided tools to retrieve this information.

If the answer is not in the context, then simply respond with something like "I couldn't find the information related to that business. " Think carefully about your responses and ensure that the information returned for each listing does indeed belong to that listing. Feel free to pause the video and then copy over this prompt or enter something relevant to your business.

Now that should be everything we need for our chatbot itself. Let's go and create our document store. So from the dashboard let's go to document stores, let's click on add new, let's call this AI business directory, let's add this, then let's click on this document store, which is currently empty.

Now we can start adding data sources to the store. So let's click on add document loader. Within here let's click on Airtable and now we need to provide information about our Airtable database.

So let's go ahead and set up Airtable. Go over to Airtable. com and sign into your account.

Within the dashboard next to workspaces click on create a workspace. Give it a name like AI business directory and then click on create and let's select start from scratch. Let's rename the space to something like AI business directory and let's press enter.

Let's also rename the table by double clicking on this tab and let's call this listings and let's save this as well. Now we can define our column names as well as upload data. To speed things up you'll find a link in the description to a CSV file which you can download for free.

Then click on add or import, then click on CSV file, click on browse files and select the file that you just downloaded. Then click on upload one file. Let's select create a new table and let's click next.

Then lastly let's click on import and now you should see all of these columns with some sample data. Great! Let's actually delete the table that we created earlier by right clicking on it.

Let's delete this table and let's rename this one. So let's call this listings and let's save this. Great!

We now have a table with data that we can use. If we go back to Flow wise we first have to set up our credentials. So let's click on this drop down, let's click on create new, let's give it a name like AI business directory and for the access token let's go back to air table, click on your profile, then click on builder hub, then under personal access tokens click on create new token, call it AI business directory.

Then let's add a few scopes. First let's add data records read, let's also add schema basis read and let's also add a base and let's add the base that we just created which was AI business directory. Then let's generate a token, let's copy the token and paste it into Flow wise and let's click on add.

Let's also select the credential that which is created. Now for the base id we can simply go back to our table and the base id is this first part of the url that starts with app and a whole bunch of characters after that up until the first front slash like so. Then for the table name it's this section that starts with tbl so let's copy that and paste it in here and then the view is this part over here that starts with viw, it's pasted into this field.

Cool I'm also going to remove this limit as I want this loader to read all the entries in the database and we're not going to add a text splitter at all as flow wise will automatically create a document for each entry in the database and we can see that 16 chunks were extracted and that is because we have 16 records in our database. Let's click on process to now store these changes. Let's also set up our absurd config so under embeddings because I'm using openai I'll use the openai embeddings function I'll select my credentials for the model name I'm going to use text embedding three small and I'll leave all the other values on the default values then for the vector store I'm going to use pinecone then under credentials let's click on create new then go over to pinecone.

io and log into your account or sign up so the first thing we want to do is to get our api key so go to api keys then click on create api key I'll call this ai business directory and let's create this then I'm going to copy this key and paste it into flow wise and for the credential name I'll call this ai business directory as well let's add this let's also select the credential now for the pinecone index let's go back to pinecone and let's create that index under database let's click on create index let's simply call this flow wise then for the dimensions I'll pick on set up by model I'm using text embedding three small so this needs to be consistent with whatever model name you're using over here then I'll click on set configuration and finally we can click on create index so I'll simply copy the index name and add it to flow wise I like giving namespaces to my projects so I'm going to call this ai business directory and I'm going to change the top k to 10 documents this simply means that when the user asks the question the retriever tool will retrieve 10 documents back from the vector store in other words the 10 most relevant businesses to the user's query now finally let's also set up the record manager let's click on postgres then under credentials let's click on create new let's give it a name like ai business directory and let's also go ahead and set up a postgres database to simplify the process of creating a postgres database let's go to superbase. com and sign into your account from the dashboard let's create a new project let's give the project a name like ai business directory let's also set a password let's also set the region and let's click on create new project this will take a few minutes to complete once the project has been created go down to project settings then click on database and under connection parameters we can find all the information needed to complete the setup let's start by copying the username and let's add it to flowwise let's also enter the password that we created earlier and let's click on add and let's select that credential then for the host we can copy that from superbase and add it to flowwise let's do the same thing for the database name like so and let's also copy the port and add it to flowwise as well then the only other setting I want to change is the cleanup method let's change it from none to full this simply means that the record manager will be responsible for preventing duplicate entries in the vector store and delete information when we delete entries from the data source let's save this config let's test it out by clicking on absurd and if everything was set up correctly you should see a message sign that 16 records were added to the vector store and if we go to pinecon we can see that 16 records were indeed absurd to the vector database and if we go to superbase and if we go to database you should see this table with 16 rows great so now that we have a document store we can finally go back to our chat flow let's open up our AI business directory flow and from the document store node let's select our newly created document store let's save this and let's test it out let's say I'm looking for a restaurant near the waterfront in cape town and we instantly get a few responses back like the waterfront seafood spot and if we go to our table and we can see that the waterfront seafood spot is indeed a restaurant near the waterfront in cape town now we can expose this chatbot to our users in many different ways by clicking on this api endpoint button we can embed this chatbot into a website or if we have a python or javascript application we can easily integrate this into our custom applications as well but for simplicity's sake i'm simply going to go to share chatbot let's enable make public and let's copy this url and if we open this url in the browser we will be able to access our AI application from outside of flow wise now we can of course customize the look and feel of this application from within flow wise so for the title i'm going to add AI business directory and i'll also link to an avatar by adding a url to an image that is available online and if you scroll down we can change other things as well like the welcome message the background colors font sizes etc i'm simply going to save these changes and if we go back to our application and refresh it we can see the title and avatar image over here oh and we can also hide this power by flow wise branding so again within flow wise all we have to do is go to this powered by text color and change it to white and of course remember to save your changes and if we refresh this the power by text is basically gone now before we move on to the automation side of this let's simply check if our application is working okay that's great let's also try a complex scenario and i really love this behavior so this is something you can't do with the traditional business directory let's say something like i am planning on visiting cape town tomorrow please recommend a museum that i can visit during the day also suggest a restaurant and hotel near the waterfront let's say in this and this is absolutely amazing we get a museum back and our ai assistant recommended a museum along with a restaurant and a hotel and because this is an assistant with memory we can definitely have follow-up conversations with this assistant as well if you're enjoying this video then please hit the like button subscribe to my channel and share this video now let's have a look at automating updates to our document store at the moment when we make changes to our database so if i add oak and barrel which is a restaurant i'll just enter something like one oak street in johannesburg and i'll just put in some fictitious number and in the description i'll just enter find dining restaurant specializing in steaks and sushi with a rating of 4. 5 great now as i showed you in the intro of this video this data will not be available to the chatbot automatically and unfortunately we would and unfortunately we would have to go through this tedious process of manually reprocess the document loader and then manually absurd these chunks into our vector store using this button over here now thankfully flow was thought about this already and what i did was expose a lot of api endpoints that we can consume from outside of flow wise to automate these type of processes let me show you what i mean when you go to flow wise's documentation and under the api reference documentation you can find api endpoints for pretty much every object within flow wise including the document stores here you will find api's for creating document stores listing document stores fetching details of a specific store and what we're interested in is this process loading and chunking operation which is this step over here as well as absurd chunks from document store which is effectively this step over here so to automate this process you can use any workflow automation tool that you want like make.