Why RAG hallucinates even with perfect retrieval

The retriever returned the right chunks. The answer sits right there in the context the LLM was given. The LLM still generates a plausible-sounding wrong answer. This is the failure mode that convinces teams RAG does not work — the one where every obvious fix (better embeddings, bigger chunks, more rerankers) has already been tried.

The confusing part: this is not a retrieval problem. The retrieval was fine. The hallucination is happening at the generation stage, where the LLM has the right context but produces a wrong answer anyway. This article covers the four causes of generation-stage hallucination in RAG and the fix for each.

Prerequisite: the RAG pipeline walkthrough covers the four-step loop this article plugs into.

First confirm the failure is at the generation stage, not retrieval

Before chasing generation causes, confirm the retrieved context actually contained the right information. Pull the per-query log for the bad answer and read the retrieved chunks by hand.

Two possible outcomes:

The retrieved chunks do NOT contain the correct answer. This is a retrieval problem — chunk size, embedding model, metadata filters, query rewriting. Different article. See RAG chunking strategies and Evaluating RAG — retrieval and generation metrics.
The retrieved chunks DO contain the correct answer. The LLM saw it and ignored it. This is a generation-stage failure. Continue reading.

The distinction matters because the two failure classes need entirely different fixes, and conflating them is why most RAG debugging sessions go in circles.

Cause 1 — The prompt does not constrain the model to the retrieved context

The single most common cause. The default prompt looks like this:

prompt = f"""You are a helpful assistant.

Context:
{retrieved_chunks}

Question: {user_question}

Answer:"""

The problem: nothing in the prompt tells the model it must answer from the context. A well-trained LLM will blend its own training-time knowledge with the retrieved context, falling back to memorised facts when the context seems incomplete or when the memorised answer feels more fluent. The result: an answer that is partly from the chunks and partly from the model’s parametric memory, with no indication of which.

Fix: be explicit in the system prompt. The pattern that works:

system = """You are an assistant that answers questions using ONLY the context provided.
If the answer is not in the context, say exactly: "I don't know based on the provided context."
Do not use outside knowledge. Do not guess. Cite the specific passage you used."""

prompt = f"""Context:
{retrieved_chunks}

Question: {user_question}"""

Three elements matter here:

“ONLY the context provided” — explicit constraint.
“If the answer is not in the context, say exactly: ‘I don’t know’” — gives the model a safe escape so it does not have to fabricate.
“Cite the specific passage” — forces the model to show its work, which makes false attribution easier to catch.

This change alone fixes 50-70% of generation-stage hallucination in most RAG systems. It is cheap to try first.

Cause 2 — The model is too small or too old to follow context faithfully

Older or smaller models have less reliable instruction-following. Even with a well-constructed prompt, they ignore the “answer only from context” instruction under some conditions — especially when the question is at the edge of their training distribution, or when the retrieved context contradicts information they memorised during training.

Typical pattern: Haiku-tier or GPT-mini-tier models hallucinate on 10-20% of context-constrained prompts where Sonnet-tier or GPT-4-tier models hallucinate on 2-5%. The gap is especially visible on technical and factual queries where the model’s pretraining data contains incorrect or outdated information.

Diagnose: take 10 of the hallucinated examples, re-run them through the next tier up of model with the same prompt and context. If the stronger model answers correctly from the context, the smaller model was the limiting factor.

Fix (two options):

Upgrade the model for the hallucination-sensitive path. Many production RAG systems use a cheap model for most queries and the stronger model only when a confidence check (or user feedback) suggests the small model struggled. The handbook has the full confidence-gating pattern.
Reduce the ask. If upgrading is not viable, narrow the question the model has to answer. Instead of “summarise what the policy says about X,” ask “does the policy allow X? Answer yes or no.” Smaller models hallucinate less on narrower tasks.

Cause 3 — The question is ambiguous and multiple chunks look relevant

Not all hallucinations are the model inventing facts. Some are the model answering a different question than the user intended, using correctly-retrieved context for that different question.

Example: user asks “what is the deployment limit?” The system retrieves two chunks — one about deployment frequency limits (per day) and one about deployment size limits (per PR). Both are relevant to “deployment limit.” The model synthesises them into “the deployment limit is 50 per day with 500MB per PR” when the user only wanted the daily cap. From the user’s perspective: the answer added a fact they didn’t ask about, which looks like a hallucination.

Diagnose: check whether the retrieved top-K contains chunks that are each individually relevant to A DIFFERENT INTERPRETATION of the question. If yes, this is cause 3.

Fix: disambiguate the question before retrieving.

disambiguation_prompt = f"""The user asked: "{user_question}"

Based on the question alone, list the 1-3 distinct interpretations
that would require different information to answer.

If the question is unambiguous, return exactly: "UNAMBIGUOUS"
"""

Run this through the LLM. If multiple interpretations come back, ask the user which one they meant before answering. Costs one extra LLM call per ambiguous query but eliminates a class of false hallucinations entirely.

For higher-volume systems where asking back isn’t acceptable UX, the alternative is to retrieve more aggressively (top-K = 10-15) and have the generator explicitly address each interpretation separately rather than synthesising. Trades off answer length for accuracy.

Cause 4 — The context is correct but formatted in a way the model misreads

Subtle but real. Two chunks with the correct answer may both be retrieved and placed into the prompt, but the formatting — chunk boundaries, headers, order, duplicated content from overlap — can confuse the model about what belongs together.

Three specific sub-patterns:

4a. Overlap artefacts. If the chunker produces chunks with 15-20% overlap, two adjacent retrieved chunks may contain the same sentence twice. The model sometimes interprets duplication as emphasis and over-weights the duplicated claim.

4b. Lost ordering. The retriever returns chunks sorted by similarity, not by document order. A paragraph that appears early in the source document might be presented AFTER a paragraph that originally came later, breaking the causal flow of the explanation.

4c. Context pollution. A chunk with the correct answer is retrieved alongside a chunk that discusses an EXCEPTION or a FORMER version of the policy. Without clear separation, the model blends the two.

Fix: restructure the context presentation.

# Not this — chunks concatenated flat
context = "\n\n".join(retrieved_chunks)

# This — labelled, document-ordered, with explicit source markers
labelled = []
for i, chunk in enumerate(sorted(retrieved_chunks, key=lambda c: c.source_position)):
    labelled.append(f"[Source {i+1} — from {chunk.doc_title}, section {chunk.section}]\n{chunk.text}")
context = "\n\n---\n\n".join(labelled)

Three improvements in that pattern:

Sorted by original document position so the natural order is preserved
Explicit [Source N] markers so the model can distinguish and cite
--- separators so chunk boundaries are unambiguous

Combined with the “cite the passage” instruction from cause 1’s fix, this makes most context-pollution hallucinations traceable and either fixable or filterable.

The 10-minute diagnostic flow

When a RAG system returns a hallucinated answer, walk through the causes in order:

Pull the per-query log. Read the retrieved chunks. Do they contain the correct answer?
- No → retrieval problem, different debugging path.
- Yes → generation problem, continue.
Check the system prompt. Does it explicitly say “only from the provided context” and give an “I don’t know” escape?
- No → add the instructions (cause 1). Re-test.
- Yes → continue.
Try the next tier up of model on the same input.
- Hallucination goes away → model was undersized (cause 2). Upgrade or add confidence gating.
- Still hallucinates → continue.
Check for question ambiguity. Could the question be interpreted multiple ways with different retrieved context sets?
- Yes → add disambiguation step (cause 3). Re-test.
- No → continue.
Inspect context formatting. Overlap artefacts? Order broken? Pollution from exception chunks?
- Yes → restructure context presentation (cause 4). Re-test.
- No → the problem is likely in retrieval after all, or the question is at the edge of what the model class can handle.

Most generation hallucinations resolve at step 2 or 3. Steps 4 and 5 catch the long tail.

What this article deliberately does NOT cover

Hallucinations in production have a longer tail than four causes. The parts not covered here, and where to find them:

The four tested LLM-as-judge prompts for measuring hallucination rate at scale, calibrated against human review.
The confidence-gating pattern that routes queries to a stronger model only when the cheap one looks uncertain.
The seven-field production query log that makes “why was this answer wrong” diagnosable in five minutes without re-running the query.
The debug script that walks the five diagnostic steps automatically, given a payment ID.

Those live in Chapters 4 and 7 of The RAG Production Handbook. First 20 buyers at ₹699 (launch price ₹999 through Nov 30, then ₹1,499 from Dec 1).