Gen AI Project Using Llama3.1 | End to End Gen AI Project

today we are going to build end to endend llm project called cold email generator using Lama 3. 1 open source llm we will also use chrom ADB Vector store Lang chain and streamlet let us first understand the problem that this tool is solving there are software services companies like TCS infosis I also have this company at lck Technologies and these companies help their clients build software clients are like Nike JP Morgan Kroger there are a lot of businesses throughout the globe who want to build software and these companies have a huge staff of software engineers and they provide their services to these clients now there's a lot of competition in this market and many times the sales teams from these companies what they do is they use variety of marketing techniques to buid and to acquire projects one of the techniques is C email so a person from from software Services Company such as attick will send a cold email to a potential client now how do they know if these clients have a specific requirement one of the techniques that they use is they will go to their potential clients job portal so let's say I'm on Nike's job portal where they have this requirement for a software engineer in AI ml area and you will look into all the skills Etc that they need and then what attick will do or CCS will do is send a cold email saying that hey it looks like you need a software engineer in AIML domain and we have uh these people so instead of hiring full-time maybe you can hire people from us because uh many times these uh companies operate in countries like India Indonesia where the cost is low and also clients like Nike don't have to hire full-time employees you know if they can hire them on contract based so there are several benefits of using this model so at will send this kind of C email to Nike and many times this cold emailing techniques work now a sales representative or bde Business Development executive at at Lake or TCS will have to form a nice email which is relevant to the job post and one of the things that we can use um is llm using llm we can generate a cold email so let's say you build this kind of tool where you give a job posting okay and when you submit it will go to that job post it will figure out what kind of skills are needed and based on that it will write an email which is relevant so let's say in that job post uh they needed a person who has machine learning with python and devops knowledge see these two and the email will exactly uh talk about that profile that skills you will also give this kind kind of portfolio so this will be like at. com python at lick.

com devops you will give that portfolio portfolio means the past projects that you have done in that area in terms of technical architecture we will go to that career page extract text from that page using Lang chain framework then we will use llm Lama 3. 1 to extract uh job role job skills description in a Json format then we give that to chroma DB which is a vector database in that Vector database we have previously stored the skills and the relevant portfolios for at company okay so let's say I have stored the skills for example for python this is my portfolio www. com okay SL python so these kind of links are already stored we will look into it more uh as we uh proceed in this particular video uh you'll know more so it will basically find relevant information for this uh skill set so if you're talking about react it will fetch you uh portfolio links related to react and then it will give this portfolio link as well as the job description okay this Json it will give these two things to llm and that will generate this kind of nice email you all probably know that Lama 3.

1 is a super amazing open-source tool that you can use it for free many people download this model locally on their computer and run their inference many people also use tools such as olama and this olama tool will make certain things easier for you you can say o Lama pull Lama 3. 1 o Lama run Lama 3. 1 that way it makes things easier for you now we are not going to use locally downloaded model or maybe use AMA because when I previously used AMA the inference was very slow to generate a response it was taking around 5 minutes so what we will do is use something called Gro so Gro is a platform which allows you to run Lama 3.

1 in cloud and the inference is very very fast because they use something called lpu language processing unit that makes the inference which means when you ask a question the response will come back up very fast so go to console. gro. com either create a login using email or just log in with Google I have already logged in I am on a playground you can type uh you know messages like first person to walk on the moon was and you can submit see see you see it was so fast and it is using uh llama 38b 8192 model they support other models too such as mixl whisper large you can just look into it they keep on adding new models we are going to use Lama 370 billion for our project and for that you need API key so you will click here API key create an API key okay so let's say my project whatever just give your project name submit and then you have to copy this API key and save it somewhere this is like a pass password it is your own individual API key that you should not be sharing with others I'm going to delete this key by the way so that should be okay once this is generated you will uh create a folder so I have created this C code code email generator tool so I have created M folder and then I uh went to that folder and just ran python um let me just show you so I ran python M notebook or jupyter notebook and here I'm going to create a new notebook I'll probably call it tutorial Gro I want to test it okay so chat grock is a class which Lang chain framework offers you might be aware that Lang chain is a framework which makes building llm application easier it's a open source module so you just do p install Lang chain install it and then uh you also install pip install Lang chain Gro so first you install pip install Lang chain the P install Lang chain Gro to install that particular module and then um I'm just going to copy paste this code here you import chat grock class from this uh module and then uh you can use mixl whatever but I'm going to use llama 3.

1 okay so let me use that and I have put my API once again I'm going to delete this um but I'm just showing you for demonstration purpose and simply you call llm do invoke and you ask the question this is my question you store that in response variable and then response. content see it was super fast it went to Gro Cloud so you need internet connection OB obviously it asks this question to Lama 3. 1 70b model and receive the response let me take a moment to talk about today's sponsor or firu is your pandas program running slow you can make it faster by installing fire du pip install firu and then use the import hook utility where you supply dhm option while running your Python program no code changes required and your program will run super fast fireex is a compiler as a cated data frame Library fully compatible with Panda's API it can utilize multiple cores on your computer and also it has some optimization strategies which will make your program faster it not only helps you do your data analysis in more efficient way it will also help environment by reducing CO2 emissions because your compute performance is optimized check video description below for fire duux documentation okay let's go back to our video now as a Next Step step we need to set up a vector database and we are going to use chroma DB there are other Solutions such as Pine con milver Etc but chroma DB is open source very lightweight so we will use that in case you don't know what exactly is a vector database then go to YouTube search for code Basics Vector database watch this uh simple 6 minute video it will explain you what exactly it is all right so I'm on chrom ADB website I will go go to documentation and here to install chrom ADB simply run pip install chroma DB in your command prompt so let me go to my command prompt clear this and run this I have already installed so it's not going to do anything after installation you will go to getting started guide and you will write simple code to uh create a sample collection so here I will create a new Python 3 notebook call it tutorial chroma DB and here import chroma DB do client so this is my client and then client do create collection so my collection name is whatever my collection so this is a database in database I'm creating a collection it's like a table where I can insert my records so I will call it a collection control enter execute it then to add documents documents is nothing but the records or data records that you're adding so collection.

add and here documents okay now these documents are usually big text blocks in our case since we're testing this thing we will just provide a simple sentence so we have provided some simple sentences and then IDs is equal to id1 id2 I will explain you in few seconds why we are not using traditional database why we are using this uh special Vector database so control enter so documents are added now uh you can do something called collection. getet and these are all the documents you can see this way you can get all the documents you can also get an individual document by ID you can say collection doget ID is equal to this and it will give you only ID one now here is the most interesting part why we don't use traditional like MySQL type of database so here you can make a query in that you can say query text is equal to uh you can say query is about CH B now you might be wondering what exactly is this Cho B is relevant to Delhi okay in New Delhi which is the capital of India people eat chur so I'm not searching a document by saying New York or delh I'm searching a document by providing a food item that is popular in delhi now that's when this Vector database will come in play it will be able bble to do semantic search so it will be able to tie that query to Delhi the other argument it has is number of results okay and this is results so look at that the first document that I got is Delhi and this is the distance so the distance of this query and this Deli document is 1. 5 whereas the this query and New York document is 1.

8 which means New York is 1. 8 km away whereas Delhi is 1. 55 it's not actually kilometer but it's a distance ukan distance so 1.

55 unit away it is more similar if I say query is about Pisa then see New York comes first not only that if I say quer is about Brooklyn Bridge then New York comes first so you can see it it is doing a meaning based semantic search as a human if I ask you okay this is my query quer is about Brooklyn Bridge which document is more near or more related you will obviously say New York because Brooklyn Bridge is in New York you can also say Okay query is about India C Delhi query is about air pollution I know it it hurts but air pollution in Delhi is higher see you can say I have queries about Big Apple now Big Apple is New York but you can see that the words are not exactly matching but it is doing the semantic search now to delete all the document douments you can simply say collection. delete and then you know this way it can delete all the documents you can also add documents with metadata so I'm just copy pasting same code the additional thing I have is metadata which is for New York I have a Wikipedia page for New York okay so we can add that and when you ask the same query it not only returns Delhi in the metadata it will also return the URL or the Wikipedia link of Delhi this is useful many times when you are building any llm Tool uh llm will generate an answer but you want to know which source it is referring to from where it got that answer so having this metadata will be useful we will see more about this as we code more in this project I created one more notebook called email generator and there I have written the code for basic llm now I want to go to the website so let's say Nike has this particular job posting I want to scrap the data from this website first if you look at the Lang chain documentation it has this class called Web B loader what this class will do is it will uh accept the URL of that job posting and it will extract data from that so it's basically web scraping so let me just uh copy this code here and I will copy paste this particular URL and you can then call loader. load.

pop. Page content and that will be your page data and I will print page data now it's doing web scraping so it will take some time but you can see that it went to this website the data will be in HTML format internally it went through that HTML structure and extracted all this key information see apply for software Engineers search job skip navigation I mean you will see some search job skip navigation you you'll see some unnecessary details as well now we will uh write a prompt for our llm for which we will import this prompt template class let me first do this thing in chat GPD so I will just U you know copy paste some small portion and I will say that uh I will give you a job posting from that extract the skills role and description in a Json format okay and you will say job posting and you will just copy paste the whole thing see I just copy Pate the whole thing when you hit enter chat GPT will give you this kind of nice uh Json object so this is the object that we want to retrieve now you are seeing this thing here is the extracted information in Json format this is called Preamble so if you don't want that right I I just want the Json object I don't want here is the extracted information so let me go here edit this no Preamble and hit send no Preamble means don't give me that initial text see it just gave me a Json object so let me uh import this particular class here then create a prompt template from template and we'll store this into a variable called prompt extract because we are using uh this for extraction purpose we will later on use it for email writing purpose as well and I did some iteration sometimes you don't get your prompt right in a very first shot so after some iterations I created this nice looking prompt where I'm saying that scrap text from website this is my page data instruction the scrap text is from CER page whatever um extract it in Json format these are the uh fields that I need and no Preamble now we already have this llm object so what we going to do is use a pipe operation so prompt extract let me I think close this one prompt extract pipe llm and this will form your chain okay so this is the uh Lang chain chain basically you are getting a prompt and you are passing it to llm and here you can do chain extract do invoke where the input is Page data okay and this will be um this page data here and I will store it in a response object print response. content control enter see it extracted the skills the role name experience so if you look at this page right experience experience see 2 plus years experience this is the power of LM it can extract uh the key information among the all the Clutter and these are the skills which are mentioned here see AWS Cloud devops you can just verify those now this content that we have is in a text format so if you do type see it is an st Str I want to convert this text into a Json object for which we can use Json parser so this is the class that we are going to use from Lang chain and uh we will say Json parser and then Json parer do par what do you want to pass rest.

content so then this becomes the Json respon C now it is a dictionary object so if you do type Json Json or dictionary I mean they are same thing see it's a dictionary now once this step is over the next step is to prepare chroma DB so I have this CSV file where I have all the technologies that my company is working on so atck is working on all these companies okay TCS infosis whatever that company is that will have the capability of all the Technologies and for each technology they will have some portfolio portfolio means okay these are the projects that we have done in this particular technology so if you go to at. com which is a real company uh so we have done all these projects for example uh let's say moon tower tickets so if you go there this is short of our portfolio okay here it explains okay who was the client which Tex Tech we used what was the challenges solution and so on so if you go to case studies here you'll see all our clients and projects so this is basically ATX portfolio similarly here I'm using some dummy URL you you can understand this is a learning project but for each of the Technologies these are the portfolio links and I need to insert this this thing into chroma DB then whenever there is a job posting it should extract the skills from the job and it will match one or multiple of these Technologies and then it should retrieve this portfolio URLs which I will use in my email all right so I have U put that CSV file here and I'm going to just import that here so import Panda as PD and I will create a data frame so read CSV my portfolio. CSV I think that's the name and see I have this in my data frame now you will iterate over your data frame and you will insert those records one by one into chroma DB so since we have written this code before I'm going to save you some time imported chroma DB instead of client see in our sample notebook we use client correct when you use client it will create the chroma DB in memory but I will use persistent client so it will create DB on a disk so in your current folder see right now in my current folder there is no folder for chrom ADB when I execute this code it will create that database store it in a folder so that next time when I want to retrieve my data I have it on the disk and here I'm saying that if not collection.

count means if let's say collection doesn't have any count if it is being created for the first time then you iterate through all your data frame rows folks this is very very easy simple python code you just Hing over uh data frame rows and for each of the rows you are adding collection do add you are adding a document metadata is your link okay metadata is this links and text tag is your main document so control enter and that worked okay so if you look at now your folder you see this Vector store uh folder is created so internally it is storing the data here you don't need to look at these files it will be gibberish for you but this shows that when you use a persistent client it will create a folder and store your records inside that folder now I will do simple collection query I will say Okay I I need a person with experience in Python experience in react and when I do that see for the first one experience in Python it is bringing these two which are python related portfolio for the second one it is bringing react portfolio react native Port portfolio if I do react native I think it just still P see react native came first and then react portfolio so for each of these experiences I requested two results and I requested metadata so which was the portfolio link now my job is nothing but Json response and in my job the skills I have is this so what if we pass all these skills so for example if I pass these skills here just for a demo purpose then see um for my Nike position it will give me for strong computer science fundamental is giving this for Java python goang it is see python it doesn't have goang then for devops it is is giving me my deos portfolio so it is working okay all right so my links are stored here now the way I created The Prompt template for extracting the content out of job posting I will create another prompt template for forming an email so once again I'm not going to type the whole thing because it it will waste your time so here my prompt template says that this is my job description whatever we got from Nike website and the the instruction is you are moan a business development executive at attick atck is an AI and software consulting company whatever uh we have expertise whatever right like usual stuff your job is to write a cold email to the client see this is where we are giving the clean instruction to LNM saying that write a cold email to client regarding job mention above and add the portfolio links and whenever you see something in a curly bracket that is something we will give as an argument to our prom template so this link list okay this is not a data structure link list but this is uh this particular links that you will give in a prom template so you're giving two parameters job description and the link list and after that again usual copy paste stuff so you are creating a chain of prompt email and llm and you're invoking Your Chain by passing these two parameters okay job description and Link uh list uh job description is nothing but the job that we got right so job if you do job it is the Json object the dictionary and the link list is these links so when you control enter look at that that's my final output it generated very nice cold IM saying that okay I came across your job description blah blah blah we have expertise in AIML software development and you can refer to our portfolio our devops portfolio is this our machine learning portfolio is this so I will request you to download this code and I hope you are coding along with me and just look at the description also folks very important thing after 6 month don't come and say that this code is not working which code this code because Nike might take this job posting away this is very temporary so what you need to do is let a smart work don't do dumb work okay don't just download my notebook and run and say oh it's not working and some people just post it on YouTube comment oh it's not working you need to provide a job posting which is active which is something you go copy paste in browser you are able to access so make sure you are changing this to some active job posting which is available when you are running this code so far we have built the raw building blocks it's like we are constructing a home and we have built little pieces and now we will connect all the pieces and create a project structure in my folder I will create another folder called app and in this app folder I will create a proper project structure we use um IPython notebook or jupyter notebook for experimentation to create our base to create our code blocks that we can use in the final project we have completed that step now is the time to create a proper project structure so I will open this in my py Cham so let's go here and what is the name of our folder well called email generator right this guy okay this window perfect here I will add a new file I will call it main. py in that I will create another folder called resource remember we had this uh my portfolio. CSV file we put that into resource so any times you have text file Excel file images it's better to put that in a separate folder called resource now we will uh create a basic UI using a streamlit so if you not installed it just go to your terminal and run pip install streamlit streamlet is a framework that allows you to quickly build UI data scientist Engineers use this to build very quick prototypes so import okay I will configure my python interpreter to be this you can use Virtual environment as well uh import streamlet as St then st.

title and I'm saying that URL input is is uh this so st. title Cod email generator then I adding the text input I'm saying enter a URL this is my default value that I'm adding okay then there is a button St do button submit and if submit do button then st. code now you can refer to stream lists document ation it's a very easy straightforward framework and what I do is I take help of chat GPT I give my requirement and usually chat GPT will give you a very um nice base which will uh save you a lot of time and then you can make edits into that base code so hello um hiring manager I'm from at like I'm just putting some dummy code for now and then language is a markdown okay so let's run this code first so here you can say streamlit run then we have app folder correct and then main.

py so when you run this you get this kind of UI when you hit submit hello hiring manager I'm from blah blah blah so my dumy UI with dummy response is ready now I can write rest of my classes so the first class that I want to add is called chains. piy okay and in that uh I will import first of all all the necessary modules and then I will use this load. environment now what is this exactly so you can read the documentation of do environment online so just say python load.

EnV we have to install this okay so pep install python. envance and after you install it in your root folder you can have a file called do EnV so let me create that file so here I can create a file call it do EnV in that you can put your API key it is a good idea to keep the API key separate from your code it is never a good idea to put it inside the code because you can somehow protect this okay so I have put my API key you need to use your API key folks remember so when I check in this code to GitHub I'm going to remove it so once again don't tell me that it is not working it's not working because you did not put your API ke and whenever uh you say load. environment if you want to receive any response you can do it or you can just run it so what this method will do is it will find EnV file and it will set this as your environment variable once your environment variable is set you can use os.

get environment Grog API key to retrieve that particular API key so if you want to taste anything uh you can say if name is equal to main okay you can say print let's say I just want to test this and when you right click and run it see it is printing that particular API key now I'm going to create a class called chain and in Python this is how you write the Constructor so self. llm is nothing but the chat grock object remember we created that in our notebook so I'll go to that notebook copy paste that thing my model is Lama 3. 1 70b is 70 billion parameters and my API key is whatever I got from the environment variable and then I will write these two functions these two functions are something that we already wrote in our jupyter notebook we wrote this uh basic code and now I have converted that into a function so if you look at it it's the same code uh prompt extract then you're using Json parer to pass the response and you are just returning that response back and another uh chain that we had was chain email okay so I had that raw code in my notebook I took that code wrote this function and my class chain is now complete similarly I will create another class called portfolio so portfolio.

pi and here I will read that CSV file and I will do the chroma DB setup so just to save your learning time I'm going to copy paste from my code so here um you are initializing this portfolio class with a file path which is app resourc my portfolio. CSV so see app resource my portfolio. CSV then you are creating a data frame the same way that we did it before and you are creating chroma DB persistent client and creating this particular collection and when you call load portfolio it will go through that data frame here and it will uh add those records into collection and when you do query links it will just query I will also add utils.

py where I'm going to add a function which will just clean some tag so if there are any HTML tags Etc your you are getting from a web page it will clean it up in our case when we were doing that notebook we did not have to do this because it was straightforward but it is good to have this code because see all these HTTP tags all these special characters um multiple spaces leading and trailing wide spaces it will remove and it will kind of massage your text and make it good look and then I went to my main. py file I put everything in a function so this was not in a function so I created a nice function called create streamlit app and when submit button is hit I'm using that same web based loader that we used in that notebook and I'm calling clean text this clean text function to clean any unnecessary garbage then portfolio. load portfolio will create my chroma DB and llm do extract jobs function will um call this function so so it will create my Json objects for the jobs and for every job so see here let's say one one web page has multiple jobs then I'm taking care of that possibility for every job you will then do portfolio.

query links and then call write email function so in my portfolio once again in Chains actually my right email is that same exit code and in the main function you are creating chain and portfolio and then you are just simply calling this particular function now here my app is already running so what I need to do is go here and click on rerun so when you rerun it will kind of reload that code and rerun it and when you hit submit it is now going to that Nike's job posting it is extracting the skills Etc and it has now created this cold email so a business development executive at attick will just simply copy this and send it to a hiring manager in Nike saying that if you don't want to hire a person full-time we are a service provider software Services AI services and we will uh give you the staff at a cheaper cost um and you you can just give your sales speech there this is the tool folks that companies can use for uh real okay this is the real tool that we have built atle is my company like we have been in business for four five years we have more than 65 employees and we use this kind of techniques so this tool is useful in my own company so you can imagine that this project was not some toy project this is a real project that any software company or any service based companies can use for their uh sales purpose all right that's it I'm going to add requirement.