Seven AI Agents and a Knowledge-Graph: AGENTiGraph

hello Community you ask me hey is it possible to connect multiple agents to a knowledge craft yes of of course we can and I will show you today here a new methodology that stands for adaptive generative engine for task based interaction and graphical representation agent interaction graph and it is a Noel platform to design that is designed to bridge the gap between llms and knowledge gra for multiple agents beautiful this is here a paper October 15 2024 University of Tokyo University beld Duke Medical School un alant Yale University zamen university conal medicine and Hanan University

beautiful eight institution eight famous institution now we will use our old friends or tools that you know Chain of Thought reasoning a react framework will be implemented in our agents and we use in context learning where we will apply you short learning and this is the way we will build our prompts for the llms now here we have seven agents on day beautiful so let's have a look just look at the name and you might understand what we are talking about the user intent interpretation agent the key concept extraction agent the task planning agent the

knowledge graph interaction agent is say ah here it's happening the reasoning agent the response generation agent and the dynamic knowledge integration agent if we want to feed new noes and edges back to our knowledge graph so interesting eight institution seven new agents let's have a look let's say I have a user query I say hey I want to understand the relationship between quantum entanglement and teleportation what foundational concept should I learn first the US intent interpretation agent identifies this now as a particular request and there is a pred def finded class this here is called

a prerequisite prediction and I will show you the different other classes in a minute so we have pre defined structures that we use the key Concepts extraction agent says okay I extract here the objects entanglement teleportation as key Concepts the planning agent has now to do here to determine here the foundational Concepts leading to or in the environment of quantum and Triangle Man teleportation and map out a learning path to answer the query the knowledge gr interaction you get it it generates here some sparkle queries or Cipher queries or whatever to retrieve Concepts from the

knowledge graph and the reasoning agent analyzes to retrieve data to identify the most relevant foundational topics and the response generation agent crafts the response and then the dynamic knowledge integration agent updates the knowledge graph if we discovered some new relation that that are not established in the knowledge graph and here we have it official documentation visualization we have a user input we have a multi-agent framework that interacts with a knowledge gr and an output is generated and this is here a rather simple representation of a particular workflow so we have here the task definition and

then we have agent one to seven and then somehow all those agents interact in a specific way with a knowledge qu and this is it well we have to have a closer look in there and you might say wait a second this sounds familiar no and you say hey look at this video where you showed us here the interaction between an llm and a knowledge grth for the new gift methodology and over there you showed us here syn on graph deep and responsible reasoning of large language models on Knowledge Graph and although this is already

here from March 2024 we had a look at this and we discovered that instead of just simply fetching here facts from a knowledge gr with a simple Sparkle query the LM could here actually sync its way through the knowledge graph and explore different reasoning path through the knowledge graph like a detective gaing the clues and this is what I told you that syn on graph is rather beautiful full for and you remember I showed you this we had just an llm plus a Knowledge Graph and then we looked here at sync on graph this is

an llm tensor product knowledge graph so this means a stronger interconnect with the knowledge graph plus was simple we generate a sparkle query or any other query that you like from a graph database uh you have a question you got a resp response but if there is a link missing if your knowledge graph is incomplete and most cases knowledge graphs are incomplete you have a problem here but not here because as I told you here under example of canar Australia and Anthony here we have now that we built here subgraphs here of the Epsilon environment

here of our particular topics and then we were able to deduct the information from the query and you remember the mechanism that I told you how think on graph works that it couples here the llm with a Knowledge Graph using the beam search algorithm and this was the beauty that we do not just go with a greedy algorithm but beam search allowed us y the llm to explore multiple possible reasoning paths at once in parallel and then choose here depending on the width of our beam to choose the top three or top five causal reasoning

structure now this here is not the case so this is a one a step back because now we just have simply a greedy query so you see I ask myself reading this wait a minute so I understand perfectly if you have an llm and a Knowledge Graph and you have simply your sparkle query but how do the ORS here in this new paper go now to a closer inter relation between the seven agents and the knowledge graph what is the mechanism because it is not as I showed you here on syn on graphs a beam

search it must be something different so what is it now we have to dive have now a deep dive into every agent in the agent graph and I ask here our little strawberry here to provide a simple example after had some exchange we have strawberry on the topic of this paper so let's have a look but yeah let's make it a little bit bigger here first detail we have how many points we have you have seven points seven agents so the first three agents what are they doing now exactly and the question qu is hey

how does photosynthesis relate to cellular respiration and what concept should I understand first you see very clear predefined question what a coincidence so let's have a look so the user intent interpretation agent determines that the user is asking for both the relationship between two concepts photosynthesis and cellular respiration and any prerequisite knowledge and now you might say ah now I know where the Epsilon surrounding of our topic are coming from so we have here the output is relationship judgment and prerequisite prediction this is here our interpretation what the user wants from thei then we have

the key concept extraction agent now it is happening we extract now the entities from the user query and you see it is photosynthesis and cellular respiration and it also extract here the relation between those two entities is relate to exactly as in the user text so now that we identified what it is really all about we already used up two agents okay let's go with this and then comes another interesting part the task planning agent so whatever we have some intelligence that is invested in the planning of the strategy of the whole system now it

gets interesting the task one and two find the relationship between photosyn synthesis and cellular respiration yes this is exactly the question and identify prerequisite concepts for understanding these processes so we are now looking if you want for additional information explaining maybe in the simplest case what is photosynthesis where is your connection to I don't know animals or to plants or to whatever we have what is cellular respiration what is a cell what is respiration how is this combined what is the latest research on the topic so anything that has a prerequisite concept we're looking for

so those are now these two tasks that we are looking for and now Thea comes and says now the knowledge graph interaction agents comes and says hey I have here a knowledge graph for me structured knowledge is available for me so it builds now or it constructs now and executes here a database query a graph database query a cipher query or a sparkle query whatever you have it doesn't matter what system you're using for both tasks one and two your remember my hey wait a minute wait a minute here task one task two okay so

CCI effect various for both task so now we understand where we get the structured Knowledge from and then retrieves relationship and prerequisite Concepts from the knowledge gra now this is a little bit weake because how does it do this now let's just follow along the reasoning agent analyze the result determined that photosynthesis and cell respiration are complimentary processes that identified at the prerequisite in our case he identified by the AI given the knowledge gr that was available included basic chemistry cell biology energy transfer and in end and then we have a respon generation agent and

a dynamic knowledge integration agent great yeah they implemented this year with a graph database on NE 4G but whatever you don't have to use scii you can go with whatever you like but you see now it is interesting that we just have here a graph database query but where does this relationship and the prerequisite Concepts from the knowledge gr come from for the llm to understand and integrate because where we are is more or less we are here no we have an llm plus knowledge gra and we have here Spar quare or C or whatever

you have and if there is a missing link you have no way to find an alternative rout between those two entities so how does it do this that it is it would say equivalent here to our more Interlink representation of an elment or Knowledge Graph like in sync on graph now at this point here I looked here at explicitly the agent and here the user intent interpretation agent and I used and I looked here at the prompt this is the Absolute Concrete prompt that the authors used so you see here this is what defines what

an agent is doing one you are an expert and LP task classifier specializing in Knowledge Graph interactions and analyze the given query in class ified into the following categories so we don't not go with the complete free search space but we say hey either relation judgment prerequisite prediction path searching concept clustering subgraph completion idea hamster or then everything else a freestyle question and then they provide examples beautiful in context learning is happening and then you see great and then please answer in the following Chason format okay this is Point number one now it gets interesting

because now we have the key concept extraction prompt and this is the official real prompt and they say now as you are an advanced concept extractor whatever this means okay your task is to identify and extract key concept entities and relationships from a given user query using here our old friends here the named entity recognition and relation extract action techniques that we know now for dozens of years and you will then map these to the knowledge graft schema using bird derived Vector representations for semantic similarity and I give you examples and beautiful and this is

exactly where I also said wait a minute you will then map these to the knowledge graph schema using birt derived Vector representation for semantic similarities I have three immediate ideas how I do the mapping so which kind of mapping is it in detail what are you talking about here and unfortunately they are rather wake here in their detailed explanation it is just this sentence and you see here a screenshot and it tells us here yep the maps extracted entities to the knowledge gr by semantic similarity with birth vector repres presentation so we have to have

now a two-step process suddenly within the extraction agent where we have new objects so suddenly we don't have seven agents and a knowledge grth but suddenly we have to build here if you want here a bird model kind of a sentence Transformer what I would do and this is only one option how we do the mapping is this one let's say and over Show an example to make it really clear how does global warming influence seais melting and what are the main factors involved so you see we need some predefined knowledge here what are those

factors so we do not just have a relation between global warming and sea ice melting with the object influence but we want to exactly know how and why and what is in the Thematic clustering here of those terms so this is my idea how it is could be done but if this is the exact case or maybe there are three four five other mapping Technologies I don't know but I want to explain this here and this is the simplest case if you want so step one we extract the entities and the relations from the query

easy no global warming C eyes melting beautiful and the relation we have is influence and maybe you go even with main factors involved then if we use bird we have embeddings we have Vector representation in a mathematical Vector space we now generate embeddings for the query entities and the relation that we just found so using now a bird system but hey stop we have to build a bird system no we have to build here a complete Vector space where our semantic similarities in the linguistic text is now encoded in a close bu relationship in a

mathematical Vector space this is the whole sense of using bird so we have now to embed here each and every term so we have the bird embedding of global warming bird of SE melting and you get the idea so we have one set of vectors if you want but how do we get the knowledge graph embeddings now either you buy those here let's say from open ey or you use your own bird that you created for your domain specific knowledge so let's have a look entities in the knowledge graph let's say that in your knowledge

graph somebody built those system and there are a little bit different terms technical terms you have to label climate change and not global warming and you have ice cap melting instead of C ice melting and you get the idea so we now extract here from a Knowledge Graph embeddings and there's a simple methodology that we can do this not only the entities but also the relations and then but only then if we have the embeddings for the query and the embeddings in the knowledge growth then we can compute the similarities so you see the global

warming here here we have the global warming and the climate change that is here in the knowledge graph they have now a similarity in the constructed bird system that we built out of thousand and thousand and thousand of documents they are now really semantically similar and if we go with C ice melting and ice cap melting we also get here the highest correlation between those two terms and C ice melting is of course the term that we used in the query and ice cap melting is the term that we have available in the knowledge graph

so if you say now there's a mapping between those this is the easiest way I can imagine to perform the mapping but I have multiple other ideas how to do this mapping so you see it is not really specific or if you found here the hint how they do it please leave a reply step five mapping to the knowledge gra and here it's happening now so we have the global warming as I told you here from the query we M this now to the node in our knowledge graph and the best Note is climate change

and for the C ice melting our thematically most Closer by note is ice cap melting in the knowledge graph so now we have now allocation of the qu to the embeddings in the knowledge graph my goodness yeah impact on Knowledge Graph query now we can have for example here with Cipher we formulate now query and Cipher to find a relationship between climate change and icec meltings with a cause relationship but you see this is quite some step no because either okay you buy here the embeddings from some company that you like or wherever you are

or from your llms but normally I build my own bird systems because they are highly domain specific only for theoretical physics I have 12 bird systems so you see you have to have those terms in your knowledge gral and you have to have those terms in your embedding structure great this was agent number two then we have the task planning prompt this is here really The Prompt that generates you the agent and we say hey as the task planning agent Your Role is to decompose the identified user intent into a logical sequence of executable task

for the knowledge graph interaction create an optimal plan considering now that we know the task dependencies and the execution order here's an example and you provide in context learning look do this then do this beautiful now agent number four you know the knowledge graph interacts and you're not going to believe it here as a Knowledge Graph interaction your task is to translate high level task into executable graph queries we now have our ccii queries our Sparkle queries utilize F shot learning and react framework to generate and Define queries dynamically here the examples beautiful example beautiful

in context learning few short learning then we have the reasoning agent prompt apply logical interference to the Rock R results leveraging contextual understanding now from the llm and the reasoning capabilities from the llm the RO results come from the knowledge graph so now you bring them together and you use now the intelligence of the knowledge grth for causal reasoning examples beautiful and then response generation prompt yeah I guess it is clear now to us how this works Dynamic knowledge integration prompt and those were seven agents and the prompt that generate those agent and Define here

the name the scope the role the example and the task and the output form the schema how they exchange information between seven EI agents and one Knowledge Graph or maybe you have multiple Knowledge Graph then I would put in another eight agents to do the coordination here of the information flow to multiple knowledge graphs and the a tell us here our agenda graph has been rigorously evaluated and the ERS tell us here the performance Matrix is achieved 95% accuracy and task classification and 90% success rate in task execution okay beautiful and they tell us here

hey we have two mode either you can jet here with the llm or you can have here an exploration mode that you discover here new relations beautiful so you see a new way with seven agents here that work together interact here in a very specific way I showed you here the specific prompts of each and every agent and this is how the system is build up okay you might say now but we know that in certain aspects we could optimize here this particular structure no because if we go back let's go back here to the

general form no here we know for some agents there are already optimization routines available like a beam search mechanism or how we integrate here the knowledge grth in a more refined way yeah but this is as always so some beautiful research work presented to us here by eight beautiful institutions and they show us here their way plus we immediately know and if you're a subscriber of this channel you immediately know where are the points where you can say hey I just learned from the other videos that we can do further optimization and you see this

is the beauty of currently AI research reading all this beautiful publication by all the AI researcher you understand that each and every time task that you might have there are hundreds of people that already tried to optimize this particular task and in the end you can get maybe a real optimization of your complete methodology so there we are this is exactly what I set out to do I hope you'd enjoyed I hope you had a little bit of fun maybe it was informative and it would be great to see you in my next video