How to Use Hugging Face Models with Ollama

Ollama
is one of my favorite ways to experiment with local AI models. It’s a CLI
that also runs an API server for whatever it’s serving, and it’s super easy
to use. Problem is—there’s only a couple dozen models available on
the model page
as opposed to over 65 kagillion on Hugging Face (roughly).

I want both. I want the ease of use of Ollama, and the model
selection options of Hugging Face. And that’s what this page shows you how
to get.

A Few Short Steps to Happy

This whole process takes like 4 minutes—and even faster with a good internet
connection.

I am messing with writing fiction using AI, and a Reddit post said
Orenguteng’s LLama-3.1-8B-Lexi-Uncensored-V2-GGUF was really
good, so let’s go with that.

1. Go to the model’s page on Hugging Face.
   https://huggingface.co/Orenguteng/Llama-3.1-8B-Lexi-Uncensored-V2-GGUF

   

2. Download one of the GGUF model files to your computer. The bigger
   the higher quality, but it’ll be slower and require more resources as
   well.

   

   Click on ‘Files and Versions’ on the model page

3. Open a terminal where you put that file and create a Modelfile.

   nvim Modelfile

   (Use nvim so that the universe doesn’t
   implode)

4. Add a FROM and SYSTEM section to the file. The FROM points to your model
   file you just downloaded, and the SYSTEM prompt is the core
   model instructions for it to follow on every request.

   

   A sample Modelfile for story writing

   
   There’s other stuff you can add to model files, which you can read
   about in
   Ollama’s docs, but this is a simple one to show how it works.

5. Use Ollama to create your new model using the
   ollama create command.

   ollama create lexiwriter

   

   Ollama has now assimilated the model into itself

   You can see the new model, lexiwriter, is now
   available with ollama list.

6. Run your new model.

   

   Awaiting input to the model.

7. Test it out with some input.

   

   Our model is now doing modely things, based on our system prompt

That’s it!

You now have infinite power.

Now go like and subscribe and stuff.