After 7 days letting AI agents control my email inbox... 📮

I've LED an AI agent to take over my email inbox for the past seven days it can read all my messag it can create email dropped and even send out emails on behalf of me it took care of more than 60 emails I received for the past week at the beginning this AI agent has no contacts about me and how I normally behave which caused quite a bit trouble but throughout the week we did a few iterations to really improve the performance it read through hundreds of past emails that I sent before to extract facts and knowledge about me like my Twitter Discord and where do I live and also learn voice and tongue as well as logic about how I normally respond messages based on those past examples and it use those knowledge to create digital me where it behave almost identical like how I do it also develop sophisticated Behavior pattern where it will categorize every single new email first and take different actions based on the type of categories from research about perspect if it is something related to partnership to return a summary about who this perspect is and what does the company do as well as check my calendar time zone to coordinate a time that actually works for both of us and even escalate to me when the case is out of his capacity so in this video I will talk about things I learned from having a real autonomous agent embedding a core aspect of my life as well as how can you build your own Super AI assistant so let's get it so the whole reason I start building this AI assistant is because my email inbox start getting out of control I got a lot more email inflow than I can handle and sometimes I do see the email and I know response but I want to making sure the email is poply written so normally I would just use chbt give some contexts about myself the original email and ask to generate a response but this process is quite time consuming and when I only with my phone I can't really do this because copy pasting things between different app is not that straightforward so this got me thinking what if I can have GPT to just create create a email draft for every single email every morning whether I check my email on the computer or on my phone a draft response is already there and I just need to review and click Send and that's exactly what I did so for the first version I didn't even write any code I literally use zap here to build this workflow where I would trigger GPT whenever I receive a new email with this prompt I'm using and once GPT Returns the result it will create Dr reply the reply process is a little bit complicated I need to search for the email with Pat email address and subra first which give me Str ID that I can use to create a draft to reply and pass on the body response I set this up in 5 minutes but the result is amazing every single new email has a dropped created already I can directly click on that review the results make some edits if I need and click Send it is accessible anywhere even when I'm on my phone this is my first learning by bring AI into users existing flow and make it context where is such a magical experience this is very simple it's already a big deal for me but on the other side the problem is very obvious because this this is simply one GPD prompt it has no context whatsoever about who I am it definitely caus a lots of problem 50% of time the response quality is so bad I can't really use it and it will also hallucinate and make a wrong deis for me so to make this AI assistant useful I has to teach you some facts about myself and the domain AIO so it can answer the question like I do this come the second iterations but before we dive in I want to introduce you to a research the H spot just released so I always like observing how people are hacking chat GPT to get a specific task done because it is great way for me to find product and business ideas if I can find a use case that even 1% of people are hacking with chbt then I can productionize it and build a micro sess around it I already see a few people tried and start getting some attractions for their Niche AI product that's why this research that hpot did is really useful they talk to people from Top firms about how they are hacking chant GPT today at work from lead generation to support ticket resolution and there are also use Cas that I would never expect like assisted with scheduling meetings or create a project timeline or road map so it definitely give me a lot of good ideas and inspiration if you also want to find inspiration for potential business ideas I definitely recommend to go have a look you can click the link in the description below to download this report for free now let's back to How I build the second version of this inbox AI agent I want to digitize and create knowledge base about myself so the AI assistant can mimic my behavior and the approach I decide to take is creating knowledge base for my past emails one is whether I can extract the facts and knowledge about myself just from past emails and also create a database so whenever the AI assistant receive a new email it can try to look up what was last time I received a similar type of email and how did I response before I first explore all the past emails that I've have ever sent and then extract clean data of the CSV file that has two column one original message that people sent me and the other is how I respond before then our take the original reply and turn into a extract facts and FAQ about Json and then I will use this two main knowledge source to do a vector search whenever I receive a new email so that this AI assistant can actually have reference to learn and the result is amazing I will show you how so the first thing we're going to do is export all the past send email I went to take out. google. com and just SL Gmail data and in my specific case I only need to send email so I will uncheck everything and only select send and create export once it finished it has a nbox file nbox a special file type to store emails but it's actually not very straightforward to turn a nbox file into a clean data set that we can use as knowledge base for AI we need to do a few steps to convert nbox file into a CSV file that contains a r email stress including both message and reply and then we will use large language model to do the actual parsing to extract clean data leis to do that I will open Visual Studio code where it has Mbox file in and I create python script to turn Mbox into CSV file first so this script will get the email thre as well as turn the Mbox file into a CSV file with this file name I have included this code in the GitHub link so you can copy paste directly once you did that just do python andbox to CSV ppy this should create a new file here called pass email andbox which will look something like this next you just clean up this a little bit by removing all the emails that non really sent from you and keep only the email Strat which has a column name called body so we got the first two step ready the last thing is we will use large langage model to do the data cleaning and create a knowledge set so I'll come back to visual studio code and create a new python file called email cleaning.

py and inside I have this large language model prompt to ask GPT to turn a email Strat into a Json file like this with two column original message and Json reply and I have the second function called per CSV which will basically take a data from every single row of the email strap pass them into a email pair and save the result into a new CSV file called email pirs after that I also created EMV file to save the open a API key and again I have attached a script in the GitHub link so you can copy and paste it and once you did that you just need to run this script python Emil cleaning. py this function going to take a while depends on how big the CSV file is but once finished you should have a clean data set look like this that we can use as a f sh prompt for the AI assistant so we got our first knowledge base and now we want to create a second knowledge base which is extracted knowledge and facts about me what it does is basically if I mentioned something in the past for example my Discord Link in one of my email reply gbt should automatically extract that what is Jason's Discord and answer is a link and we give GPD hundreds of my past emails so you can actually extract loads of facts and knowledge about myself to do that I will go back to visual studio code add a new file called extract faqpy import a list of libraries I will create one function first to load the CSV data and then I'll create a function called extract FAQ which will taking the whole list of past email reply break them down into two small chunks with 3,000 token each chunk and then we'll do a map reduce method so for each Chunk we will go through this prompt you are a smart AI assistant above pass email your goal is to learn and extract common fq about AI Json including both question and answer and return the result in Json format after we extract fq from each individual chunks then we will combine them together with another prompt called combine prompt the following is a set of feq about AI Json and please this two into a final Consolidated array of fq so this were in the and combine everything together and we will use a load summary chain which should provide the map reduce function from L chain we will return the final fqs and in the end last function to save the extract FAQ Json into a CSV file and once we did those things our firstly load all past emails from the email paris. CSV file and then I will extract just the Json reply column convert it into stream format do the extraction and save the file to FAQs once I finish I can run python extract FAQ py and again this process probably will take a while depending on how big the file is but once it finished you should have fa aq.

CSV file as well and now since I have both knowledge base ready I can create a knowledge retrieval system that whenever I receive a new email it will do a vector search in both knowledge base so that AI assistant can rightly response based on this information I initially use lanching and llama index to create this knowledge retrieval system but very quickly I realized one problem to make this system useful I actually need to continuously update the email reply pair as well as a common knowledge and with the normal retrieval system the updating to the knowledge base is quite cubon so instead of maintaining the knowledge base myself I decide to use another platform called Romus AI where they provide a interface to manage and updating knowledge base so come here first they upload email pairs choose only to vectorize original message as you can see all the email pair will be displayed here like Excel but behind the scenes this is a vector database if later I got a new email pairs that I want to add in I can just add add a new row here to type in new examples and same thing for the FAQ doc I can upload extract FAQs and I can continuously update and addit the FAQ knowledge once I finish I can create a knowledge retrieval tool here or given name email drafter V2 import two knowledge data sets that I just created create user input client email to taking new email that I receed from people and then add a large language model step giving it a special prompt you're an inbox manager for Jason and your goal is to help draft email response mimic the past email Emil reply where I will pass on the past example as well as the FAQ knowledge then I say generate new response based on the new email that I receive and I also optimize it a bit further by clicking on the edit button for both knowledge as you can see at default is choosing most relevant data which is Vector search I will set a query to be the client email so that it will start searching what at the last time a similar type of client email I receive and retrieve both message as well as my reply and I will limit the page size to two and we can try to run this so the example email I give is I love your content you should create a Discord would love to join your community and you can see the general response is actually really good it has similar kind of voice and tone I normally have which is quite short and precise and also have the knowledge about what my Discord link is from the past email so I can simply use this by go to use tab and there will be API end point so I don't need to worry about deployment but in my case since I'm using that here to build up this workflow I can just replace open AI step to the r as step where I will select R tool click sign in which will give me this model so I need to get API project ID as well as the region which you can get from API Keys page at the bottom by creating new secret key and I select the tool name to be the new one that we just created email drafter V2 the two knowledge one I will leave empty but the client email will be the email body that I received from Gmail and replace body text in create draft reply step with the response we got from rcii Tool and that's it with this second version I start getting much higher quality draft email if you want to learn more about knowledge based embedding or implement the whole Knowledge tral from scratch using lanching or Lama index you can check out another video I made before after this I start thinking can I make this AI assistant a lot more autonomous can I actually turn this into a AI agent that has sophisticated decision making ability where it can behave like I do which categorize each email into different categories and depend on the category take on different actions for example if it's Consulting request then I need to do some lead qualification to understand what's the use case what kind of budget they have as well as the timeline if any of those information are not collected it should automatically reply back to the clients to ask for more information but if the information all collected then escalate to me so I can take it from there on the other side if a sponsorship emails I wanted to research about this company figure out maybe three potential video ideas I can make and then ask clay to me for review but if it is other type of message like General coding questions then it should do a knowledge retrieval to create a message and DRFT the email so I start building an AI agent to replicate this process I start putting this giant system prompt where I Define those categories and specific actions that it should take however this agent just continuously ignore the system PRP it sometimes categorize email wrong and it also skip steps very frequently so how to have an agent that actually follows my standard procedure is a problem but after a few iterations I seem to find a solution that might potentially work instead of keeping the standard procedure as a giant system promp I keep system promp pretty simple with just one instruction which is use a custom tool to categorize email based on the standard procedure and this tool not only categorize email and also output specific steps that should take and the result is that this agent seem to be much more controllable it will do the categorization par play and follow exactly the plan that was introduced and escalate to me with the summary of the information so let's dive in how I structure it I create two files one is app. py where defined agent the other is custom tools. py this is where we will Define the list of different custom tools that agent going to use including categorize email research reply email and create email draft so firstly in app.

py I will import a list of different libraries as well as a list of cust tools that we Define on the other file and then I will give a simple system prompt giving a row as well as goal to handle all the incoming emails by categorizing them based on guideline and decide on next steps I'll create a list of different tools that we going to create on the other file passing on the system message and create a memory initialized agent and that's it for agent the main part is in this customer tools again I will import a list of different libraries the first function I got to create is categorize email I will create a prompt first to give very specific definition about each category of email and ask it to return the category name I'm going to use open AI directly and get the result which is category and after that I will put in a list of logic so if the category is consulting or job offer I want it to do a further Tre Arch so I have another function called check Consulting emails which is different prompt that will specifically categorize if those Consulting emails already mention the key information that I need which is what's the problem that persp is trying to solve and what's their budget then return Yes otherwise return no so in here you can say if the email is Consulting and it already got all the information that want then it will forward this email to Jason with summary about the problem the perspective try to solve and their budget but if the email didn't really mention this two information then it will return two steps one is generate email response to collect further information based on guideline and then send a generate email response to pply so this will give agent the guard about what they should do next and the same thing I will replicate for collaboration and sponsorship email for this type of email I wanted to do research about prb and Company first and then forward this email to Jason with a research result so I can have a quick view about whether this company is leged if the category is not reply then it should return this email email has already been taken care of nothing needs to be done now but if it's other type of General email then it will just generate response and then create email draft so this tool will basically do the categorization and return a list of next steps and to define the customer tool popularly I will need to Define two class why is categorize email input which will allow me to give description to the agent to explain the input further so the input will be latest reply form the perspect as well as actual tool class categorized email tool where I can Define the name of tool as well as description about when to use this tool I would Define argument schema to be the categorized email input that I defined above as well as point to the function that I Define above to be triggered when this tool is used and that's pretty much it for the First Tool and then I will also create second tool to generate email response where we're going to use API and point of the relevance knowledge retrieval tool that we created before but if you build a rock pip line with lunching or Lama index you can also just replace this part with your own function but if you're using random ci1 like I did you can just simply call the API endpoint I'm going to do the same thing to define the general email response input as well as general email response tool to give description about input and function so this tool will help us generate high quality response after that I have pretty complicated tool called PB research tool this will basically utilize the research agent that I created before so if you want to dive into more details you can check out that video but at high level it has a few different functions it will use a serer tool to do the Google search for certain keywords and also scraping tool to grab content from a specific website URL and as well as summary function in case the website content is too large and in the end we're putting them together as a research agent and then Wrap This research agent as a customer tool here called P research tool and last but not least we're going to repeat a similar process for escalating tool which will send a email to me for escalating or a reply email tool as well as create email draft and for each of those custom tools and basically using that here to to integrate to different systems so each tool here are basically calling the zapier endpoint and that's pretty much it we can just hook everything together and try a few different test emails so if I try python app.