Failure modes and their controls
| Dimension | Likely cause | Control |
|---|---|---|
| Fluent but unsupported answer | Retrieval returned nothing relevant and the model continued anyway. | Require a citation for every claim; refuse when no permitted source is found. |
| Correct source, wrong summary | Context window crowded, or chunk boundaries split the relevant passage. | Structure-aware chunking, re-ranking, and answer-level evaluation. |
| Stale answer | Index not updated after a source change. | Monitored ingestion cadence with freshness visible per source. |
| Restricted content surfaced | Filtering applied after retrieval, or applied in the prompt. | Permission filtering enforced during retrieval, tested with real identities. |
| No answer when one exists | Lexical and semantic retrieval both miss the phrasing. | Hybrid retrieval with query expansion, plus refusal analysis. |
| Cost or latency spike | Retrieval depth and prompt size grown without a budget. | A cost and latency envelope per task, measured per answered question. |
The retrieval pipeline, in order
Every RAG system has the same stages: ingest a source, parse and chunk it, embed and index the chunks, retrieve candidates for a question, optionally re-rank them, assemble a prompt, and return an answer with citations. Each stage can fail independently, which is why a single accuracy number tells you very little about where the problem is.
- Ingest: connector, cadence and source identity
- Parse and chunk: structure-aware splitting with metadata
- Index: embeddings, lexical terms and filterable attributes
- Retrieve and re-rank: candidate set, then ordering
- Answer: grounded generation with citations and refusals
Chunking and indexing are design decisions
Chunk size is a trade-off, not a default: small chunks improve precision and lose context, large chunks carry context and dilute relevance. Structure-aware splitting — by heading, section or record boundary — usually beats fixed character counts on real documents. Metadata attached at ingest (source, owner, version, effective date, permissions) is what makes filtering and citation possible later.
Hybrid retrieval combines vector similarity with keyword search such as BM25, because embeddings miss exact identifiers like product codes and clause numbers, while keyword search misses paraphrases. A re-ranker, often a cross-encoder model, then reorders the top candidates by reading the question and each passage together.
Permissions are enforced before the model sees anything
Access control belongs in the retrieval path: filter results by the requesting identity before any text reaches the model. Instructions in a prompt are not an access-control design. Test with users who hold different permissions, including a role change and a revocation, and verify both the answers and the citations, because a wrong citation can disclose a source the user should never have seen.
Freshness, deletion and revocation
An index is a derived store and inherits the obligations of its sources. When a document is updated, withdrawn or its access changes, the index has to follow within an agreed window — and the previous version has to stop being retrievable, not merely be marked outdated. Test the deletion path deliberately; a re-index that leaves the old chunk searchable is a data-protection incident with extra steps.
Citations and evaluation are two different tests
Check whether the right permitted source was retrievable, then whether the answer is supported by what was retrieved. A correct document can be summarised incorrectly, and a fluent answer can hide a retrieval failure. Record whether the citation resolves to the version actually used, and keep unanswerable questions in the evaluation set instead of filtering them out.
- Retrieval recall measured separately from answer support
- Citations checked against the version used, not the latest version
- Unsupported and ambiguous questions retained in the set
Illustrative example · hypothetical organisation
Worked example: an HR policy assistant
Take a hypothetical HR assistant that answers employee questions from the policy library.
The corpus is limited to current, approved policies, each with an owner and an effective date. Documents are split along their headings, and every chunk carries the policy name, section, effective date and the employee groups it applies to.
A question from a contractor is filtered to policies that apply to contractors before retrieval runs, so employee-only benefits are never retrieved for them. The answer cites the policy section, and when no permitted policy covers the question, the assistant says so and points to the HR contact.
When a policy is replaced, the old version leaves the index the same day, and an evaluation question about the change is added to the test set.
Failure modes worth testing before launch
RAG systems fail in recurring ways, and each has a control that can be designed in advance. The table below lists the patterns to test with the people who own the corpus, not only with the developers who built the pipeline.
Trace the claim, not just the retrieved document.
An answer can cite the right document and still say more than that document supports. In this hypothetical supplier-policy example, the source requires an annual review for suppliers in one controlled-products category. The generated answer says that all suppliers require an annual review. A useful trace shows where the category condition disappeared between the source, model context and final claim.
Scope and assumptions: The question, policy and answer are illustrative. A citation or a model-based grade is not independent proof that a claim is correct. Use the actual applicable source version and a domain-owned rubric, and do not send private source text to a grader without the required data-use decision.
| Dimension | Trace step | What to retain and decide |
|---|---|---|
| Question and scope | Record the supplier category and time context the question actually provides. Write down whether that is enough to choose the applicable rule. | If the category is missing, the expected outcome may be a clarifying question. A reference answer must not quietly assume context the user did not supply. |
| Retrieved candidate | Record the source identity, version and passage location returned by retrieval, together with the access decision. Finding the document only establishes that this stage had a candidate. | Confirm the relevant condition exists in the retrieved material. If it is absent here, investigate corpus coverage, parsing or retrieval before changing the answer prompt. |
| Delivered context | Inspect the passage actually included in the model input after selection, truncation and context assembly. The full retrieved document may contain a condition the final prompt omits. | Check whether the controlled-products qualifier survived. Keep enough structure to interpret the sentence; retrieving a qualifying paragraph does not help if another stage discards it. |
| Claim support | Compare each material claim with its supporting passage. In this example, the universal phrase all suppliers is not supported by a rule for one category. | Mark the scope mismatch explicitly even when the annual frequency and citation are correct. Check the complete meaning as well as individual sentences; a valid fragment can become misleading in combination. |
| Response boundary | Choose the supported answer, clarification or refusal from the evidence available to this reader. Keep the cited source version attached to the supported statement. | Recheck that the final answer retains the category condition and that the citation resolves for the reader. A successful link check and a supported-claim check are different results. |
At narrow widths, scroll the worksheet to read every column. With the table focused, use the arrow keys to scroll.
Worked decision: the condition is present in both the retrieved passage and the final model context, but the answer broadens it to every supplier. Record an answer-support failure, not a retrieval miss. If context assembly removed the condition instead, the same visible error would have a different cause. Keep both fixtures so that a fix at one stage cannot hide a regression at the other.
- OpenAI's retrieval example explicitly distinguishes failures in finding relevant evidence from failures in using it. Preserve that distinction when reviewing a trace rather than assigning every wrong answer to the index.
- Sentence-level and whole-response review can reveal different errors. Automated graders can assist, but their mistakes and disagreements with human judgments need evaluation too; a fluent grading explanation does not settle a disputed policy meaning.
Use this trace as a diagnostic case in the retrieval and answer evaluation guide, keeping source availability, retrieval and answer support as separate observations.
Official references for this example:
Questions and trade-offs
Question
Does RAG stop hallucination?
No. Retrieval supplies evidence; it does not remove the model's tendency to complete an answer. What RAG adds is a way to check: require citations, refuse when no permitted source is retrieved, and evaluate answer support separately from retrieval quality so an unsupported answer is visible.
Question
Can we point RAG at everything we have?
Technically yes, and it is usually a mistake. Indexing everything widens the permission surface, dilutes retrieval and multiplies deletion obligations. Start with a bounded, owned collection where the questions are known, and expand only when the evaluation shows the wider corpus improves answers.
Question
Which vector database should we use?
Start from requirements: filtering by permissions and metadata, hybrid keyword and vector search, update and delete behaviour, and where your data already lives. Many teams begin with the vector search built into a database or search service they already run, such as PostgreSQL with pgvector or a cloud search service, before adding a dedicated product.
