What is RAG? is the definition: look up a named passage, then write. This page is the operating choice. When is that method the right one, which shapes people actually ship, and what “effective” is allowed to mean when nobody will show you the file.
When RAG is the job
Use RAG when three things are true at once.
- The answer has to match a corpus you own: contracts, handbook, incident reports, specs, tickets.
- That corpus changes. Last year’s clause is not this year’s.
- Someone must be able to open the passage if the sentence is later disputed.
Notice-period questions, refund rules, safety steps, “what did we tell this customer in March” are RAG jobs. The product is the folder under the lamp. Generation is the last mile.
When it is the wrong job
Authoritative public lookups. If the answer belongs to a current registry, API, or first-party page, query that source. Indexing a copy into your private corpus adds staleness without adding authority.
A one-off document task. If the relevant file fits in context and the job ends today, open the whole file and keep it with the work. Build retrieval when reuse, corpus size, change, permissions, or repeatable citation justify an ingestion path.
Durable relationship questions. “How are these accounts connected?” “What depends on this service?” “Who owns the claim in chapter 4?” If the relation itself must be stored and traversed, use a knowledge graph, not a similar paragraph. You can build that graph from text. It is a different architecture, not a bigger vector store.
A decision that needs a named reviewer. Retrieval does not replace HITL. A correct clause can still be sent to the wrong person.
Two preconditions people skip
Teams argue about embeddings and leave two operating decisions unwritten. Those decisions constrain every architecture.
Freshness. How long between a document changing and the system answering from the new version? Set a service target, then test it with a known edit: re-upload a handbook, change a spec, ask the dependent question, and record the gap. An index that quietly lags is worse than no index because the wrong answer arrives with a citation attached.
Permissions. Retrieval inherits whatever you indexed, not whatever the asker is allowed to read. If salary letters, board minutes, and the HR handbook went into one collection, the model can retrieve all three unless you stop it. Enforce access during retrieval against the asker’s identity, partition corpora by access boundary, or index only material every user may read. Filtering after the model sees the passage is too late.
Decide what happens when nothing good is found. Test the retrieval floor on known misses, not only successful demos. If no eligible passage clears it, return no answer and name the next route. Refusal is a feature you build on purpose.
Walk one: the policy that had two versions
This failure is ordinary and consequential.
Someone asks the assistant how much notice a customer needs to give before cancelling. The system returns a clean answer with a citation. The citation points at terms-v3.pdf. The signed agreement with this customer is terms-v4.pdf, uploaded in March, and it moved the number.
The components behaved as configured. The product still failed. Both files were in the corpus and matched the question. Nothing in the index said one superseded the other, so ranking picked on similarity, and v3 happened to phrase the clause closer to the query.
The repair is not a better embedding model. It is metadata and a rule.
- Every indexed document carries an effective date, a version, and a status: current, superseded, draft.
- Superseded and draft documents are filtered out of retrieval by default, not merely ranked lower.
- The answer names the version, not just the file. “Clause 12.3 of terms-v4, effective 4 March” is auditable. “According to our terms” is not.
- Somebody owns deletion. A corpus that only grows will keep every mistake retrievable forever.
Run this on your own system this week. Pick one policy you know has been revised. Ask the question. Look at what came back. If the answer cannot tell you which version it read, the architecture argument is premature.
Four shapes, four jobs
People list architectures as if later ones make earlier ones obsolete. They do not. They answer different questions.
Naive retrieve-then-generate. Chunk the corpus. Embed. Fetch nearest neighbours. Stuff them in the prompt. Write. This is the demo. It is enough when the question names a clause and the right chunk is near the query. It fails when two versions of the policy sit next to each other, or when the question is about the whole corpus.
Hybrid search. Keyword or BM25 next to dense vectors, then a reranker. Use it when names, IDs, and rare tokens matter: contract numbers, SKUs, error codes. Embeddings are bad at exact strings. Lexical search is bad at paraphrase. The pair is furniture that earns its keep. It still is not Inspect.
Agentic retrieval. The model may call search as a tool, look again, refuse. Use it when one hop is not the question: “compare last year’s MSA to this year’s and name the clause that moved.” You now have a workflow. You also have more ways to skip Inspect. The loop in GRAIN still binds: Gather, Rank, Assemble, Inspect, Navigate.
Graph-index RAG. Microsoft’s GraphRAG method extracts entities and relations from text, clusters them into communities, and generates community summaries. Local questions still want a passage. Global questions such as “what themes recur across this corpus?” have no single chunk to fetch; that is the job GraphRAG was built for. It adds an extraction and community-summary indexing pipeline. Its public GitHub README, read 2026-08-27, says the project is largely in maintenance mode. Treat it as a published method, not a default install.
Sort by the question, not the slide.
| The question sounds like | What it actually needs |
|---|---|
| ”What does clause 12.3 say?” | Naive retrieval, with version metadata |
| ”What did we quote on order 88-4471?” | Hybrid search. Exact IDs, not paraphrase |
| ”How did this year’s MSA change from last year’s?” | Agentic retrieval. More than one hop |
| ”What are the recurring themes in two years of support tickets?” | A graph-index method, or an analyst |
| ”What breaks if we deprecate this service?” | A knowledge graph. Not retrieval |
None of these is “most effective” as a slogan. Test three things: did retrieval use the current authorised version, can a human open the supporting passage, and does the system refuse when no passage supports the answer? A sophisticated graph that fails those tests is still a fluent guess with extra ceremony.
What to demand on Friday
Take one answer from the week.
- Which passage was gathered?
- Why this version, not the older one?
- What was assembled around it?
- Did anyone inspect a miss, or only the demo that worked?
- Next lookup, or refuse?
If those lines are blank, stop arguing chunk size. You shipped generation with extras. Evaluate the reasoning, not the fluency is the adjacent test when the sentence is tidy and the source is not.
The RAG Engineer owns the longer practice. It is available now. You can already refuse a slide that says RAG and cannot open the folder.