By Damian Siembor · Published 31 October 2026

Fine-tune a model and run the GGUF in Ollama

From adapter to a GGUF you can run: what the files are, q4_k_m vs q8_0, a real Modelfile line by line, ollama create and run, and what to check when the answers look wrong.

A fine-tune is only useful once you can run it. Almost everyone who fine-tunes an open model for local use ends up at the same place: a single GGUF file, a short Modelfile, and two Ollama commands. This guide walks that path using the files finetuna delivers, then covers the parts that go wrong.

What you get back

GGUF is the single-file model format used by llama.cpp, and read by Ollama and LM Studio. It holds the model's weights in quantized form (a quantization is a smaller, slightly less accurate copy of the model) and the tokenizer metadata in one file. The tokenizer cuts text into tokens, which are pieces of a word, about 3/4 of a word in English. A finished job delivers six files, in the panel, with an e-mail that links to the job:

qwen3-8b-<job>-q4_k_m.gguf   about 5 GB, 4-bit, the default
qwen3-8b-<job>-q8_0.gguf     about 9 GB, 8-bit
Modelfile                    points Ollama at the GGUF
adapter.zip                  LoRA adapter, 40 to 200 MB
report.html                  loss before and after, 20 held-out answers
metrics.json                 the same numbers as JSON

The report shows the loss (how far the model's answers are from yours, where lower is better) before and after training. It also shows 20 held-out answers, which are prompts the model never saw in training.

Download links work while you are signed in, and the dataset and files are deleted 14 days after delivery, so download them. Everything else in this guide is about the first three files.

q4_k_m or q8_0

Both are quantizations of the same merged model. q4_k_m is 4-bit and about 5 GB for an 8B model. q8_0 is 8-bit, about 9 GB, and closer to the full-precision model. The Modelfile points at q4_k_m because it is the smallest. Start there. If you can fit q8_0 in memory and the answers on your own prompts differ, switch. Which one is better is something to test on your task, not assume.

Run it: ollama create and ollama run

Download the GGUF and the Modelfile into one folder (nothing to unzip), open a terminal in it, and run:

ollama create mymodel -f Modelfile
ollama run mymodel

ollama create reads the Modelfile and registers the GGUF under the name you give it. ollama run opens a chat. The name is yours. The report suggests one made from the base model and job id.

The Modelfile, line by line

This is the Modelfile from the published JSON extraction report:

# finetuna job c0f889a13ef1, base Qwen/Qwen3-8B
FROM ./qwen3-8b-c0f889a13ef1-q4_k_m.gguf

PARAMETER stop "</s>"
PARAMETER stop "<|im_end|>"
PARAMETER temperature 0.7
PARAMETER num_ctx 2048

• FROM is the GGUF file, relative to the Modelfile. Change it to the q8_0 file to use that one.
• The stop lines are the tokenizer's end-of-sequence token plus any of <|im_end|>, <|eot_id|>, <end_of_turn> and </s> that exist in its vocabulary (our training script writes this file). They are a safety net so generation halts at an end token. They are not copied from the model's chat format, and they are not a template. The Qwen3 file above lists </s> and <|im_end|> because both are in that tokenizer's vocabulary.
• temperature 0.7 is a starting point. For extraction or classification, a lower value gives steadier output. Set it in the Modelfile or per request.
• num_ctx 2048 is the context window, set to the sequence length used in training (1024 for Qwen3 32B). Prompts much longer than that are outside what the model saw during training.

If you filled in the system prompt field on the new-job page, there is one more line. The Yoda job has it:

SYSTEM """You rewrite what the user says the way Yoda speaks. Reply with the rewritten sentence only."""

That system prompt is placed in the Modelfile for Ollama and was not used during training, which is worth knowing if you expected the model to have learned it. If you want it trained in, put it in your dataset rows.

When the answers look wrong

The Modelfile above has no TEMPLATE line, so Ollama chooses the prompt template for the imported model. A template that does not match the format the model was trained on is the usual cause of rambling, repeated text or an answer that never stops. Check what was applied:

ollama show --modelfile mymodel

Compare the template and stop lines with the base model's documented chat format, and with the TEMPLATE section of theOllama Modelfile reference. Then check the basics: the right file in FROM, enough memory for the file you chose, and a prompt shaped like your training rows. The report's side-by-side answers are generated with the model's own chat template, so they are the reference for what the model should do. If Ollama differs from the report, the difference is in how it is being run.

Qwen3: check the thinking setting

The reports show Qwen3's answers with its thinking mode switched off, so you see direct answers with no reasoning text. Ollama may run it differently. Its blog post on thinking (May 2025) says the command line turns thinking on by default. Its current documentation says the default depends on the model. Whether Ollama treats a GGUF you imported yourself as a thinking model depends on the template it assigns, and we have not tested that for every base model. So check your own file. If you see reasoning text or a thinking field, turn it off like this:

# one run in the terminal
ollama run mymodel --think=false

# inside an interactive session
/set nothink

# native API
curl http://localhost:11434/api/chat -d '{
  "model": "mymodel",
  "messages": [{"role": "user", "content": "Your prompt"}],
  "think": false,
  "stream": false
}'

The documentation says think: false asks for no thinking "if the model permits it". It does not cover Ollama's OpenAI-compatible endpoint, so send the field to the native /api/chat or /api/generate. Whichever way you run it, compare a few outputs with the report's answers.

Other runners, and doing it yourself

llama.cpp and LM Studio open the GGUF directly, without the Modelfile:

llama-cli -m qwen3-8b-<job>-q4_k_m.gguf -p "your prompt"

If you train elsewhere, the conversion finetuna runs is two llama.cpp steps on the merged 16-bit model: convert to an f16 GGUF, then quantize. Merge the adapter first, because Ollama and llama.cpp expect a single set of weights here, not a base plus an adapter.

# merged = your base model with the adapter merged in, 16-bit
python convert_hf_to_gguf.py merged/ --outfile model-f16.gguf --outtype f16
llama-quantize model-f16.gguf model-q4_k_m.gguf Q4_K_M

TheLoRA vs QLoRA guide explains why the 4-bit file is a fresh quantization of the merged model.

What a fine-tune changes, on a real example

The Yoda run took 720 examples, 3 epochs (full passes through the file) and 6.8 GPU minutes on Llama 3.1 8B. The base model already speaks Yoda when asked, but wraps answers in quotes and chooses its own word order every time. After tuning, it reproduces the dataset's pattern, with the object fronted and "you must" at the end, and matches the reference word for word on most held-out sentences. That is the profile of a good fine-tune: not a smarter model, a more predictable one.

Limits

• Llama 3.1, Mistral Nemo and Gemma 3 are gated on Hugging Face. Llama 3.1 and Gemma 3 have their own licence terms (Mistral Nemo is Apache 2.0). Llama derivatives must carry "Built with Llama".
• An 8B model in q4_k_m needs roughly its file size in free memory, plus room for the context. Check before you blame the model for slowness.
• finetuna delivers files. It does not host the model, so you run the model yourself.