Why your RAG answer is wrong: find the broken step before you change the model

A wrong RAG answer has a broken step, and there are three places to look: the evidence was missing from the collection, it was there but never reached the model, or it reached the model and the answer misrepresented it. Check them in that order: documents, retrieval, answer. The same wrong sentence can come from any of the three, and each needs a different repair.

RAG (retrieval-augmented generation) is a design in which an AI system searches your documents, then answers from the passages it retrieved. What is RAG? explains the basics. This page starts from an answer you can show is wrong and traces it back to the evidence the model actually saw.

A real case: the right link, the wrong rule

Jake Moffatt’s grandmother died on 11 November 2022. The same day, Moffatt asked the chatbot on Air Canada’s website about bereavement fares, the reduced fares some airlines offer after a death in the family. The chatbot said a traveler could apply for the reduced rate after the trip, “within 90 days of the date your ticket was issued.”

The airline’s own “Bereavement Travel” page said the opposite: the reduced rate does not apply to travel already completed. When Moffatt complained, an Air Canada representative admitted the chatbot had provided “misleading words” and pointed out that the chatbot’s reply linked to that very page.

In February 2024, a small-claims tribunal in British Columbia held the airline responsible. The tribunal member described Air Canada’s position as suggesting the chatbot was “a separate legal entity that is responsible for its own actions,” and called this “a remarkable submission.” The member also wrote that the chatbot “is still just a part of Air Canada’s website,” and found that “Air Canada did not take reasonable care to ensure its chatbot was accurate.”

These facts and quotations are as reported by CBC News and McCarthy Tétrault. How the chatbot was built is not public, so the case is not evidence of a RAG failure and does not show which internal step went wrong.

What it does show is the symptom. The correct page existed, the answer linked to it, and the answer still contradicted it. A correct link is not a correct answer. In a RAG system, a missing document, a retrieval miss, and a misread passage can all look like that in the chat window. Underneath, they need different repairs.

Why is “the model hallucinated” the wrong first diagnosis?

Because it names the symptom, not the step that failed. An answer can repeat an outdated passage accurately and still be wrong. It can also add a claim that no supplied passage supports. From the answer alone, you cannot tell those failures apart.

In chapter 9 of my book The RAG Engineer, I call the habit of blaming the model first the Model-Blame Reflex. My operating rule is to inspect the documents and retrieval before changing the model, because those steps are cheaper to check. That is an order of work, not a claim that the model cannot fail. Questions that need reasoning across several passages can still go wrong after the right evidence arrives.

Start with the exact sentence that is wrong and the rule it breaks: “It told a team member they could reopen a completed request, but the policy limits that to the account owner.” That gives the investigation a passage to find and a claim to test. “It hallucinated” gives it nothing.

How do you find the broken step?

Ask three questions, in this order. A failure at one step does not clear the steps after it.

1. Was the right evidence there when the question was asked?

Find the passage that should have answered the question, in the version that applied when it was asked. Then check:

• Was it in the searchable collection (the index)? If the document was never loaded, no amount of search tuning will find it.
• Could this user see it? Correctly withholding a passage the user may not access is not a retrieval failure. Do not loosen permissions to make the answer work.
• Was it the applicable version? A policy update that never reached the index leaves the system answering from the old rule. RAG freshness covers that freshness contract.
• Did it survive preparation? Documents are converted to text and cut into chunks, the short passages a RAG system stores and searches. A heading, a table column, or an exception can get separated from the rule it belongs to.

If the evidence was missing, the repair belongs to the documents and how they are loaded. Search tuning cannot recover a passage the collection does not hold. Until the collection holds it, the right answer names the gap and refers the question to the policy owner instead of guessing.

2. Did the right passage reach the model?

Finding the passage in the knowledge base proves it was stored, not that this answer used it. Trace the handoffs your system has:

1. Query preparation: the question may be rewritten or classified.
2. First search: the search returns candidate passages.
3. Reranking, if used: a second model re-orders the candidates by relevance.
4. Context assembly: the system chooses what fits into the model’s input and drops the rest.

The evidence can be lost at any handoff. Save the search query, the filters, the returned passages with their IDs, and the final context: the exact text sent to the model. Compare what entered and left each step. A screenshot of search results cannot show whether the passage survived context assembly.

Look for a vocabulary mismatch (“end my trial early” against “trial cancellation before the period ends”), an old document outranking the current one, or the right passage falling just below the cutoff. Check whether context assembly kept a summary and dropped the rule’s conditions. Change the step that lost the evidence. Raising every limit at once makes the cause harder to isolate.

3. Did the answer match the passage it received?

Only when the needed passage reached the model does the answer become the main suspect. Compare the two claim by claim. Look for a limited rule that became universal, a “may” that became a “must”, or an exception that vanished.

A citation does not settle this. Open the cited passage and check that it supports the sentence, including its conditions and scope. What is LLM grounding? explains that support test. Then read the rest of the context. An outdated summary beside the current rule, or an instruction to always answer confidently, may need a context repair before any model change.

Which fix belongs to which step?

The table follows the failure mapping in chapter 9. It is organized by GRAIN, the framework from The RAG Engineer (available in English and Dutch), which divides retrieval work into five areas:

• Gather corpora: which documents enter the collection, with their permissions, versions, and how they are cut into chunks.
• Rank and rerank: finding candidate passages, then ordering them for the actual question.
• Assemble context: choosing which passages reach the model, in what order, within its input limit.
• Inspect failures: tracing a wrong answer to the step that failed and routing the repair there. This page is an Inspect exercise.
• Navigate freshness: keeping the collection current and showing how old an answer could be.

Read the table as a guide to the next check, not as a verdict from the answer alone. What is GRAIN? has a short overview.

What you see	Failure to investigate	GRAIN step that owns the fix	Check first
The answer matches an old version of the policy	Stale evidence	Navigate freshness	Index version against source version
The right document exists but never appears in search results	Evidence missing from the index or missed by search	Rank and rerank (Gather corpora, if it was never loaded)	Index membership, then query wording and filters
The right passage was found but is missing from the context	Context builder dropped it	Assemble context	What was cut, and why
The stored chunk cuts off a condition or exception	Passage cut badly during preparation	Gather corpora	Where the chunk starts and ends
The right passage is in the context and the answer is still wrong	Answer misrepresented it	Assemble context, then the model	The sentence against the passage
The same question gets different answers	Unknown until you can compare runs	Inspect failures	Whether each run’s context is saved at all

A fictional trace, start to finish

This trace is invented. No production system was tested, and the roles and the seven days exist only for this example.

A team member asks a support assistant: can I reopen a completed maintenance request? The assistant answers: “Yes, completed requests can be reopened.” The policy is narrower. The account owner may reopen a request within seven days; other team members must ask the owner.

1. Was the evidence there? Yes. The current policy is in the collection, and this user may see it.
2. Did it reach the model? Partly. The search returned the right paragraph, but the context builder kept only a one-line summary from the paragraph next to it: “Completed requests can be reopened.” The role limit and the seven days never reached the model.
3. Did the answer match its passage? Yes, and that is the uncomfortable part. The answer repeats the summary almost word for word. Checked against its context, it holds. Checked against the policy, it is wrong for this user.

The broken step is the context builder: an Assemble context fault. A different model, given the same summary, would have had nothing better to work from. The repair keeps the rule and its conditions together. In this invented replay, the same question then gets the restricted answer: ask the account owner, who can reopen the request within seven days. That illustrates the check; it is not a measured result.

How do you know the fix worked?

Run two kinds of replay. First, a controlled one: give the model the known correct passage, and keep the question, the model, and the instructions fixed. If the answer comes back right, that points at evidence delivery, not at the model. Then repair the step and check the repair through the normal retrieval path:

1. Replay the original question. Keep the before and after contexts so a colleague can see what changed.
2. Test nearby cases. For the invented policy, ask as the account owner about a request completed ten days ago: the answer must apply the seven-day limit. Ask about one completed yesterday: the owner must still get a yes. A fix that answers the easy case and loses the limit is not a fix.
3. Save regression tests. Keep each question with its expected answer, including the conditions that answer must preserve. Re-run them after every change. A test guards the pattern it covers; the same failure can still return through a question you have not saved.

A good replay is evidence about that question under those conditions. On its own, it neither proves the cause nor shows that the system is reliable elsewhere.

What are the signs the investigation has stalled?

• Someone insists the source “is in there” but cannot show the version the failed answer used.
• The team widens retrieval until the right answer appears, without checking what unrelated or conflicting text came in with it.
• A replay changes the model, the prompt, the index, and the question at the same time. The answer improves and the cause stays unknown.
• The prompt gets a new line such as “if unsure, say you don’t know.” Chapter 9 warns against exactly this: the retrieval gap stays open, and the model starts saying it is unsure on questions it had the evidence to answer.
• The test only checks that a citation exists, not that the passage supports the claim.

Try this today

1. Pick one answer from your system that you know is wrong.
2. Save its record: the question, the time, the user’s permissions, the document versions, the returned passages, the exact context, and the answer. Keep it under your existing access controls.
3. Mark where the evidence failed: missing from the collection, present but not delivered, or delivered but misrepresented. Note any further failure the trace shows.
4. If part of the record is missing, add that logging before the next replay; you cannot debug what you did not save. Otherwise, test the suspected step with the correct passage and check a nearby case.

End with one written finding: the passage that was needed, the step where it was lost or misrepresented, and the test that will catch the failure if it returns. Then change that step. A new model may one day be the right fix; let the trace make that case first.

Cite this:Why your RAG answer is wrong: find the broken step before you change the model.Len P. van der Hof. https://lenvanderhof.com/en/blog/why-your-rag-answer-is-wrong/ · Published 18 October 2026.