A fluent answer is not evidence that retrieval worked.

Why a single score cannot tell you the system works

A retrieval system has at least two stages that fail independently, and one end-to-end score collapses them. An answer can be correct because the right passage was retrieved, or correct because the model already knew the fact and the retrieval was irrelevant, or wrong despite perfect retrieval. Those three have different fixes, and a single number tells you which is happening in none of them.

The confound worth naming explicitly is model prior knowledge. Ask a system a question whose answer is widely documented publicly, and a strong model will answer correctly whether or not retrieval returned anything useful. Evaluation sets built from generic questions therefore measure the model, not the system — and they look excellent right up until someone asks about an internal policy.

This is why evaluation sets for enterprise retrieval have to be built from questions whose answers exist only in the corpus. It is more work, it produces lower scores, and the lower scores are the informative ones.

Retrieval: did the right evidence come back at all

Retrieval is measured against a known answer set: for a given question, which documents or passages should have been returned. Recall asks what share of those were found within the cut-off; precision asks what share of what came back was relevant. Both are computed before the model sees anything, which is exactly what makes them diagnostic.

Recall at the cut-off is the metric that matters most in practice, because everything downstream is bounded by it. If the passage containing the answer is not in the top k, no amount of prompt engineering recovers it — the model is being asked to answer from evidence it was never given. When answer quality is poor, retrieval recall is the first thing to check, and it is frequently the whole story.

The cut-off itself is a trade-off instead of a setting to maximise. A larger k raises recall and lowers precision, fills the context with weakly relevant passages, increases cost per query, and in long contexts can bury the relevant passage among distractors. Reranking exists to relax this: retrieve broadly, then reorder with a more expensive model and pass a smaller, better-ordered set forward. Measure recall before reranking and precision after, or the two effects cannot be separated.

One retrieval metric is easy to forget and produces the worst kind of failure: permission correctness. A retrieval that returns the right document for a user who is not entitled to it scores perfectly on recall and is a disclosure incident. It belongs in the retrieval evaluation, tested with real identities, and it is part of governing the corpus instead of an afterthought.

• Recall at the cut-off, measured before reranking
• Precision after reranking, so the two stages are separable
• Coverage: what share of the evaluation questions have any supporting evidence in the corpus at all
• Permission correctness: run the set as several identities and compare what each may see

Answer: is the response actually supported by what was retrieved

Groundedness asks whether the answer's claims are supported by the retrieved context, and it is the measure most closely aligned with what enterprises actually fear. Relevance asks whether the answer addresses the question. These come apart routinely: an answer can be entirely relevant and partly invented, or perfectly grounded in a passage that does not address what was asked.

Microsoft's Foundry evaluator set makes the separation concrete — it ships Retrieval, Document Retrieval, Groundedness, Groundedness Pro, Relevance and Response Completeness as distinct RAG evaluators, and recommends combining Retrieval, Groundedness, Relevance and content safety for RAG applications instead of relying on any one. That combination is a reasonable default precisely because each one catches a failure the others miss.

Citation correctness deserves separate treatment from groundedness, because a claim can be supported by the corpus while being attributed to the wrong document. For a reader, the citation is the thing they will check; for an auditor, it is the thing that has to hold. Measure whether each citation resolves to a real, current document that actually contains the supporting text — not merely whether a citation is present.

Most of these are judged by a model acting as grader, which is a practical necessity and a real limitation. Model graders drift when the grading model is updated, they are more generous to fluent text, and they inherit the biases of whatever produced the answer when the two are the same family. Calibrate against human labels on a sample, periodically, and treat a score movement after a grader change as suspect until proven otherwise.

The unanswerable question is the test most systems skip

Every evaluation set should contain questions the corpus genuinely cannot answer, and the system should decline them. This is the single most informative test available, because a system that never refuses is not being careful — it is producing an answer for every input, and some of those answers are invented. Systems that score well on answerable questions frequently fail here completely.

Three distinct cases hide under "cannot answer", and a good system distinguishes them. The evidence does not exist in the corpus. The evidence exists but this user is not entitled to it. And the evidence exists and states the opposite of what the question assumes. The correct responses differ — say the corpus does not cover it, say access is restricted without confirming the content exists, and correct the premise respectively — and conflating them either leaks information or misleads.

The second case is subtle enough to be worth dwelling on. "I cannot find anything about the Henderson acquisition" and "you are not authorised to see material about the Henderson acquisition" are different disclosures, and the second confirms the acquisition exists. Refusal wording is a governance decision, not a copywriting one.

Abstention also trades against helpfulness, and the trade is real. A system tuned to refuse whenever confidence is marginal becomes useless; one tuned never to refuse becomes untrustworthy. Track both rates together — refusals on answerable questions, and answers on unanswerable ones — because either in isolation can be gamed by moving a threshold.

• Include unanswerable questions in every evaluation set, roughly one in five is a workable starting ratio
• Distinguish missing evidence, restricted evidence and false premises in the expected response
• Track incorrect refusals and incorrect answers as a pair, never one alone
• Write refusal wording deliberately: a refusal can itself disclose

Operations: latency, cost and freshness are part of quality

A system that answers well in ninety seconds is not usable for a support agent on a call, and one that costs more per answer than the work it saves is not viable regardless of accuracy. Operational measures belong in the same evaluation as quality measures, because in production they are traded against each other constantly — reranking, larger context and multi-step retrieval all buy accuracy with latency and cost.

Cost per successfully answered query is more honest than cost per query, because it charges the failures to the successes. A system that is cheap per call but answers half the questions usefully is not cheap. Computing it requires the quality evaluation to exist first, which is part of why the two belong together.

Index lag is the operational measure with a quality consequence, and it is invisible in the response. An answer citing a policy withdrawn last week is wrong in a way no groundedness score detects — the claim is faithfully grounded in the retrieved context, and the retrieved context is out of date. Measure the interval from source change to retrievability, per source, and surface the document's own last-modified date in the answer.

What each measure tells you, and what it does not

The most common evaluation mistake is reading a metric as a verdict on the system instead of a signal about one stage of it. Each measure below has a narrow question it answers well and a set of failures it is structurally blind to, and knowing the second column is what stops a dashboard of green scores from being reassuring.

Used together, the pattern of scores is more diagnostic than any single value. High retrieval recall with low groundedness points at the generation step or the prompt. Low recall with high groundedness usually means the model is answering from prior knowledge. High scores everywhere with no unanswerable questions in the set means the set is too easy.

Building an evaluation set that is worth trusting

The evaluation set is the most valuable artefact in the whole programme and the one most often improvised. It needs real questions in the words people actually use, expected answers confirmed by whoever owns the content, the documents that should support each answer, and a deliberate share of questions that cannot be answered. Fifty questions built this way are worth more than five hundred generated ones.

Synthetic question generation has a specific and limiting failure: questions generated from a document tend to be answerable from that document, phrased in its vocabulary, and free of the ambiguity real questions carry. They inflate retrieval scores because the query already contains the target's language. Use them to broaden coverage after a real set exists, never as the foundation.

Collect the real questions before the system is built. Support tickets, internal search logs, the questions a new joiner asks in their first month — these are the distribution the system will actually face, and they are available for free before any model is chosen. Teams that skip this end up evaluating against questions an engineer imagined.

Maintaining the set is part of the job. As the corpus changes, expected answers go stale and previously unanswerable questions become answerable. An evaluation set that has not been reviewed in six months is measuring the system against a corpus that no longer exists, in the same way an unmaintained test suite stops being believed.

Offline gates, online signals, and the gap between them

Offline evaluation runs the set against a version and gates the release. Online measurement watches what real users do. They answer different questions: offline tells you whether a change is safe to ship, online tells you whether the system is useful, and neither substitutes for the other. A release gate without online signal ships systems that pass tests and disappoint users.

Treat a configuration change as a release. Chunk size, embedding model, cut-off, reranker, prompt and model version all alter behaviour across every query, and all of them get changed casually because they look like parameters. Version them, run the suite before promotion, and keep the rollback path — the same discipline a pipeline change gets.

The most useful online signals are the cheap behavioural ones rather than explicit feedback: whether the user opened the cited source, whether they reformulated the question immediately, whether they asked the same thing again later. Thumbs ratings are sparse and biased toward the annoyed. Reformulation rate is a good proxy for an answer that did not land, and it costs nothing to capture.

Reading these numbers honestly

None of these measures proves a retrieval system is good. They provide evidence that specific failures are absent from a specific evaluation set at a specific version, which is considerably less than it sounds and considerably more than most systems have. The number that matters — whether the system makes people faster at real work — is not in this article, because it depends on the work.

Three limits are worth stating. Model-graded scores are estimates with their own error, and comparing them across grader versions is comparing two different instruments. An evaluation set is a sample, so a change that improves it may not generalise. And none of this measures the failure mode where the system is accurate, fast, well-governed and answering a question nobody needed answered.

This article also deliberately stops at retrieval systems that answer questions. Agents that take actions need evaluation of the actions themselves — tool selection, parameter correctness, whether a step should have been taken at all — which is a different problem and a larger one.

Questions and trade-offs

Technical references

• Microsoft Foundry: built-in evaluators reference ↗
• Microsoft: retrieval-augmented generation overview ↗
• Microsoft: document-level access control in Azure AI Search ↗
• OWASP: Top 10 for LLM Applications (2025) ↗