Fix: Ollama returns garbage output — 4 real causes

Ollama loaded a model, accepted a prompt, and returned:

The the the the the the the the...

Or:

[endless whitespace]

Or a wall of near-random tokens that vaguely resemble English but say nothing. The model is running, the process isn’t crashed, but the output is unusable. Here are the four real causes and how to fix each.

Diagnose first

Get the actual failure mode before guessing:

ollama run <model> --verbose "Say hi in one word."

Watch the output carefully:

• Repeated single token: likely broken sampling (see cause 1)
• Wall of whitespace: stop tokens missing (see cause 2)
• Vaguely-English gibberish: quantization too aggressive (see cause 3)
• Correct start then garbage: context window overflow (see cause 4)

Cause 1 — Broken sampling parameters

The default temperature, top_p, and top_k interact. When one is set to an extreme, the sampler collapses onto one token forever.

Symptom: The the the the... or . . . . . . . .

Common triggers:

• temperature 0 combined with top_k 1 — deterministic mode that keeps picking the highest-probability token, which is often the same one
• temperature 2+ — too random, then the sampler snaps back to a repeat loop
• top_p 0.0 — no candidates at all, sampler falls back to argmax

Fix — reset to sane defaults:

Create a Modelfile:

FROM llama3
PARAMETER temperature 0.7
PARAMETER top_p 0.9
PARAMETER top_k 40
PARAMETER repeat_penalty 1.1
PARAMETER repeat_last_n 64

Then:

ollama create llama3-fixed -f Modelfile
ollama run llama3-fixed

repeat_penalty 1.1 and repeat_last_n 64 specifically prevent single-token loops.

Cause 2 — Missing or wrong stop tokens

If Ollama doesn’t know when to stop generating, it keeps producing tokens until it hits the max limit — often filling with whitespace, filler, or repeats.

Symptom: correct answer, followed by endless newlines or <|end|> printed as literal text.

Diagnose:

ollama show <model> --parameters | grep stop

Sample output — Mistral defines [INST] and [/INST] as stop sequences; a missing entry here often is the entire bug:

If the list is empty or missing the model’s natural end tokens, that’s the problem.

Fix — add stop tokens per model family:

Llama 3 / 3.1:

FROM llama3
PARAMETER stop "<|eot_id|>"
PARAMETER stop "<|end_of_text|>"
PARAMETER stop "<|start_header_id|>"

Qwen 2 / 2.5 / 3:

FROM qwen3
PARAMETER stop "<|im_end|>"
PARAMETER stop "<|endoftext|>"

Mistral / Mixtral:

FROM mistral
PARAMETER stop "</s>"
PARAMETER stop "[INST]"

Rebuild the model and retry. Stop tokens live in the model card — copy from HuggingFace if unsure.

Cause 3 — Quantization too aggressive for the model

Quantization compresses model weights. Q4 works for most 7B+ models. Q2 breaks nearly everything. Q1 always breaks. Some model architectures need at least Q4_K_M or Q5.

Symptom: vaguely-coherent English that never actually answers the question. Grammar-ish but semantic-free.

Diagnose — check what quant is loaded:

ollama list
ollama show <model> --modelfile | grep -i quant

Or check the tag: llama3:8b-q2_K explicitly names the quant.

Fix — bump to a saner quant:

Rule of thumb for coherent output:

Pull a larger quant:

ollama pull llama3:8b-instruct-q5_K_M

If the machine can’t fit the larger quant, use a smaller model at higher quant. Qwen2.5:3b-q5 beats Qwen2.5:14b-q2 for coherence every time.

Cause 4 — Context window overflow

Prompts longer than the model’s context window get silently truncated by Ollama. The truncation often removes the system prompt or start of the user prompt, leaving the model to hallucinate.

Symptom: first response is fine; later responses in a chat degrade into gibberish.

Diagnose:

ollama show <model> --parameters | grep num_ctx

Compare to what you’re actually sending. A 4096-token context can’t hold a 10,000-token conversation.

Fix — right-size context, or use a longer-context model:

Increase context if the machine has RAM:

FROM llama3
PARAMETER num_ctx 8192

Rebuild:

ollama create llama3-8k -f Modelfile

For very long context (32k+, 128k+), switch to a model designed for it: llama3.1:8b and Llama 4 Scout support 128k+, qwen3:7b and qwen2.5:7b support 32k+ (some Qwen variants go to 128k). Older Llama 2 tops out at 4k.

Also: trim the prompt. Summarize old turns before including them, or use a rolling window (keep only the last N exchanges).

The universal Ollama garbage-output flow

Every unusable-output case, run in order:

# 1. Confirm it's not a prompt issue
ollama run <model> "Say hi in one word."

# 2. Check sampling parameters
ollama show <model> --parameters

# 3. Check stop tokens
ollama show <model> --parameters | grep stop

# 4. Check quant + size
ollama list

# 5. Check context vs prompt length
ollama show <model> --parameters | grep num_ctx

Most garbage-output cases resolve at step 2 or 3.

Prevention

Standardize new Ollama installs with:

1. Modelfile per model — never rely on raw defaults for production
2. Sampling defaults: temperature 0.7, top_p 0.9, top_k 40, repeat_penalty 1.1
3. Stop tokens explicitly listed
4. num_ctx matched to actual usage (8k for chat, 32k+ for RAG)
5. Quant floor: Q4_K_M minimum for anything customer-facing
6. Verification prompt in CI: “say hi in one word” — output must be hi or hello, else fail the pipeline

Bottom line

Ollama garbage output isn’t randomness — it’s a specific failure with a specific cause. Wrong sampling, missing stop tokens, over-aggressive quantization, or context overflow. ollama show <model> --parameters reveals most of these in under a minute. Fix the parameters or bump the quant; coherent output returns immediately.