Intro
In this blog we are going learn how to run already downloaded .gguf
files into ollama. For this we are going to use Windows machine today. For other operating system the process is very much similar, you just have to change the path as needed.
Downloading and installing the necessary files
First you have to install ollama in your machine. You can visit and download it from there and install into your computer.
For this tutorial you need a .gguf
file which you can download from various places. We are going to download it from huggingface.co. We are going to use Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf
model file.
You can search this name at huggingface.co and download it from there.
Running the model into ollama
Make sure your ollama is installed properly by opening the command prompt and run ollama -v
.
Then follow these steps:
- Move the downloaded
Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf
file to this folder C:\Users\YOUR_USER_NAME\.ollama\models of your windows. - Create a new folder called modelfiles at this path C:\Users\YOUR_USER_NAME\.ollama
- Inside the modelfiles folder create a file called
Meta-Llama
( you can use any supported name you want ) and open it in your any text editor. Don't give any file extension to it, like .txt or anything. -
At the file give the path of your downloaded
.gguf
file. In our case the path is ***C:\Users\YOUR_USER_NAME\.ollama\models\Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf*.- So in the
Meta-Llama
file we are going to write:
FROM C:\Users\YOUR_USER_NAME.ollama\models\Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf
- So in the
Now save the file and open the command prompt at modelfiles directory. (You can just press
ctrl+l
, typecmd
and hit enter to open the command prompt )Now type
ollama create meta-llama -f C:\Users\YOUR_USER_NAME\.ollama\modelfiles\Meta-Llama
I am giving meta-llama as the name, you can give yours.
- It should take few moments based on your hardware and create the file.
- To confirm type:
ollama list
. You should see the details of the create model as NAME: meta-llama:latest.- Now to run the model type:
ollama run meta-llama
. And start chatting with the model.
Top comments (0)