EASIEST Way to Fine-Tune a LLM and Use It With Ollama

you want to fine tune your large language model and run it locally on your machine using Ollama. Well, in today's video, we're going to do exactly that. So let's go.

First, for the fun part, just finding the right data set. The reason why finding the right data set is so important is when you train a small, large language model with a data set that is relevant to the task you're trying to do. It can actually outperform large models.

What I'm going to be doing today is creating a small, fast LLM that will generate SQL data based off of table data. I provide it. One of the biggest data sets to do this with is called synthetic text to SQL, which has over 105,000 records split into columns of prompt SQL content, complexity, and more.

Im running a Nvidia 4090 GPU, so I'm going to be fine tuning this on my machine using ubuntu. If you don't have a GPU, feel free to do this using Google Colab, which allows you to run training code in the cloud. The great news is that this project does not require a lot of complex hardware to get it up and running.

We're going to be using Unsloth which allows you to fine tune a lot of open source models really efficiently, with about 80% less memory usage. And we're going to be using Llama 3. 1 which is an LLM used for commercial and research purposes, especially in English, and has really high performance.

Make sure that you have Anaconda installed on your machine as well as the Cuda libraries. I will be using Cuda 12. 1 and Python 3.

10 for this project. You want to install the dependencies required by Unsloth which you can find in the Readme. But for simplicity, here it is.

This creates a new environment for us and installed PyTorch Cuda libraries as well as the latest unsloth. You'll also want to install Jupyter if it isn't there already, and then run your Jupyter notebook. And now you're done with the setup.

So let's go into the Jupyter notebook and get started. First, we want to make sure that all the installed requirements are actually there. If you're using Google Colab, this command will install the packages.

Next we're going to import the fast language model by Unsloth. Here we're specifying that we want to use the Lama three eight bit model. We also want to set up a max sequence length of 2048 tokens.

This means that the model will only consider up to 2048 tokens, where a token can be a word, subword character, or even punctuation. When processing or generating text, and we'll set load in four bit to true, which essentially means we're using less bits, as opposed to using the typical 16 or 32 bits to represent the information in the model. Doing this is going to help you reduce memory usage and also reduce the load on your machine.

After running this, you're going to get a cute Ascii image, and that means that your model is loaded. After this, we're going to load in the PEFT model, which is basically Lora adapters. If you don't know what these terms mean, that's totally fine.

Basically, the LORA adapters mean that we only have to update 1 to 10% of the parameters in this model. Without them, it means that we would have to retrain the whole model, not just a small portion, which takes a lot of time, energy, and even money. Unsworth provides this here with the recommended settings.

I trust them, so feel free to read each comment. Now this is where things can get a little bit tricky depending on what data set you're using. The each data set comes different from each other, but they're each formatted in the same way such that the large language model can understand it.

Llama three uses alpaca prompts which look like this. Now, if you remember our data set, it is not as easy as just plugging it in and letting it go off to the races. I have to format my response first before plugging in the data.

I'm only interested in the SQL of my database. The prompt I will be asking for, as well as the generated code and explanation. So I'm going to update my code to reflect this.

Now we set up the training module to supervise to fine tuning. Trainer by hugging face is what I used. There are a lot of parameters to use all that can be described in their own video.

So for example, have max steps which tells us how many training steps to perform. Seed is a random number generator. We used to be able to reproduce results and warmup steps gradually increases learning rate over time.

So now that we have everything set up, let's run it. And that's it. Your model has been trained.

Now before we move on, we actually need to convert this into the right file type so that we can run this locally using a llama. Luckily, onslaught has a one liner we can use to do this. After this is done, we only need to do one step to be able to run this with Allama.

First, open up your terminal. I'm using the warp terminal here. Go to the path of where the file is saved.

Then create a file called Model file and open it up in the code editor. This is Ollamas Docker like file configuration where we can create new models with specific parameters. In our model file we're going to put a prompt.

So something like you're an SQL generator that takes a user's query and gives them helpful SQL to use. Finally make sure Olan was running. And then we're just going to run this command.

This command will then read all the items in the model file you just created, and start using llama Dhcp under the hood to make sure that the model runs on your machine. And congrats! You can now use your fine tuned LM locally, all with the OpenAI compatible API and more in your applications.

If you're curious to know more about Alama, we do have a two minute video out about everything you need to know about Alama here. Otherwise, thank you for watching and I'll see you next time.