Why a single RAG quality score is misleading
Retrieval-augmented generation (RAG) systems combine a document search process with a language model that uses the retrieved passages to answer a question. Each stage can fail independently. The search can miss the policy that answers a user’s question; it can return the right document below several irrelevant chunks; or it can retrieve exactly the right evidence only for the model to invent an extra detail in its response.
That makes a single percentage of “good RAG answers” difficult to diagnose. A useful evaluation distinguishes retrieval quality (did the search find and rank the appropriate evidence?), answer quality (did the final reply address the question correctly and stay within the evidence?) and operational performance (how reliable, quick and resource-intensive was the process?). Microsoft’s RAG evaluator documentation similarly separates document retrieval from groundedness, response relevance and completeness.
The aim is not to optimize every number simultaneously. It is to know which part of the pipeline made the user’s answer better or worse, and whether that improvement survives cases the developers did not select by hand.
Build a test set with relevant and unanswerable questions
Start with actual tasks users bring to the system. For a modest first benchmark, start with a few dozen carefully reviewed questions covering the important task types. Expand the set as new failures emerge rather than assuming that automatically generated cases represent actual user needs. The sample size is a practical starting point, not a statistical guarantee. Expand it as the corpus and the risk of incorrect answers grow.
Include questions answered directly by one passage, questions requiring more than one document, ambiguous requests, outdated versions of the same policy, terminology that does not match the source wording, and questions the corpus does not answer at all. Without unanswerable examples, a system that confidently improvises may appear successful.
For each case, save the query, the current authoritative source and its version/date, which documents or chunks should count as relevant, the essential facts a correct reply must include, and whether the correct behavior is to abstain or ask a clarifying question. Where possible, have two reviewers independently label a subset and resolve disagreements before using it as a benchmark. A retrieval metric is only as credible as the labels behind its denominator.
Keep a small, separate challenge set for final validation. If every engineering change is tuned against the same public examples, the benchmark becomes a development aid rather than an independent quality check. Data handling must also account for private or licensed documents; store only the minimum test data and access logs necessary for evaluation.
Measure retrieval precision, recall and ranking
Precision@k is the share of the top k retrieved chunks judged relevant to the question. If two of the top five chunks are useful, precision@5 is two out of five retrieved chunks. Recall@k asks what share of all known relevant chunks or documents appeared within those top k results. If the labeled corpus contains four relevant chunks and the search retrieves two of them, recall@5 covers two of the four known relevant chunks. These figures are an illustrative calculation, not measured results from any commercial RAG system.
The retrieval metric definitions distinguish precision@k from recall@k by their different denominators. The choice of what counts as a relevant document matters. A paragraph can contain background information yet fail to answer the actual question; conversely, several overlapping chunks from the same document may inflate the number of apparently relevant results. Define whether the ground truth is labeled at document or chunk level and apply the same rules to all configurations.
The Elasticsearch ranking evaluation reference notes that recall@k does not account for the position of relevant documents. Precision and recall alone do not fully capture rank quality. A search that places the best supporting passage first is more useful than one that buries it behind four distractors, even if both have the same recall@5. Metrics such as normalized discounted cumulative gain (NDCG) take ranking positions and graded relevance into account. Microsoft’s document retrieval evaluators describe using labeled relevance judgments for ranking metrics. Without trusted relevance labels, an LLM judge can estimate query-context relevance, but its score is a proxy, not measured recall against the full corpus.
Report scores by question type and document family. High average recall can conceal a serious failure on policy updates, long tables or terminology used by a small customer segment. Also track retrieval “holes”: cases where a source should exist but labeling or indexing is incomplete.
Assess groundedness, correctness and answer completeness
Retrieval metrics say nothing about what the language model actually wrote. Groundedness or faithfulness asks whether the claims in the response follow from the passages supplied to the model. The Ragas faithfulness metric documents one common approach: identify factual claims in the answer, check which claims can be supported by the retrieved context and divide the supported count by the total claim count. That yields a score from zero to one under that metric’s method.
Groundedness is essential, but it is not the same as real-world factual correctness. The retrieved source can be stale, internally contradictory or simply wrong. A model might accurately restate an obsolete price while remaining perfectly faithful to the passage it received. A separate correctness check should compare the response with a verified, current reference answer or authoritative source.
Completeness adds another question: did the reply omit a critical qualifying condition, exception or step? A short answer may be grounded and relevant while failing to mention the eligibility requirement that changes the user’s decision. Microsoft’s RAG evaluators distinguish groundedness from response completeness; the Ragas context recall documentation similarly defines reference-based coverage of information that retrieval should support.
Automated judges are helpful for triage and repeated comparisons, but their outputs require calibration. Prompt phrasing, judge model changes and order of examples can alter labels. Retain representative human-reviewed cases to evaluate the evaluator itself rather than treating every judge score as an objective measurement.
Verify citations and safe refusal behavior
A clickable citation does not prove a claim. The cited passage must actually support the sentence next to it, and the source must be the version the answer should rely on. Check each consequential claim independently: whether the referenced document is accessible, whether a quotation or number appears in that document, whether a condition was omitted, and whether another authoritative source contradicts it.
Use a small set of answer labels—supported, unsupported, contradicted, missing qualification and not answerable—and capture a brief reason when a reviewer marks a claim as unsafe. This supplies a debugging trail that a single averaged groundedness score cannot provide.
The evaluation set should deliberately contain requests with no supporting passage. Measure whether the system acknowledges missing evidence rather than inventing an answer, and whether it asks for clarification when a user refers to an ambiguous product or policy. Safe refusal is not a failure in cases where answering would require speculation; the scoring rubric should recognize that distinction.
A worked example: grounded but wrong
In a deliberately fictional example, a support assistant retrieves an old refund policy even though a newer policy exists outside its search index. Asked about the current rule, the model repeats the obsolete rule and cites the outdated document as if it were authoritative.
Under a purely context-based check, the claim is supported: the model used exactly the passage it received. Under an independently verified current-answer check, it is incorrect. The root cause is not necessarily a hallucinating model; it may be an out-of-date corpus and a retrieval process that ignored document versions. The appropriate fix is to correct source currency and retrieval precedence, then rerun both retrieval and answer evaluations.
This is a hypothetical illustration, not evidence of any actual company’s refund policy. It shows why a high faithfulness result must never substitute for a correctness and freshness review.
Use a metric scorecard rather than a vanity number
The scorecard below keeps the metric’s required inputs alongside the problem it can diagnose. A team can implement the checks manually at first and automate them as the test set grows.
| Measurement | Inputs needed | What a failure suggests |
|---|---|---|
| Precision@k / Recall@k | Query, retrieved top-k, labeled relevant passages | Search returned distractors or missed known evidence |
| NDCG | Ranked passages and graded relevance labels | The most useful sources appear too low |
| Groundedness / Faithfulness | Response claims and retrieved context | One or more claims are unsupported by context |
| Answer correctness and completeness | Response and verified reference answer | Reply is wrong, stale or missing a necessary condition |
| Citation fidelity | Claim, linked passage, version and quote check | Source does not establish the cited assertion |
| Abstention / clarification | Unanswerable or ambiguous test cases | System guesses rather than acknowledging absent evidence |
| Latency and cost | Request traces, duration, retrieval and model usage | Quality gains may be too slow or expensive to deploy |
Where a dataset has no ground-truth relevance judgments, mark measured recall@k as unavailable because the set of known relevant documents is missing. Report that the evaluation uses relevance estimates and explain who or what produced them. Likewise, a judge’s numeric faithfulness score is not interchangeable with a binary human review of factual accuracy.
Diagnose failures at the right pipeline stage
It helps to record each evaluation case as a chain: query, corpus version, retrieval settings, the identifiers and scores of returned chunks, the prompt context, the model output and the verified reference. That permits failures to be classified rather than attributed vaguely to “the AI.”
| Observed failure | Likely layer to inspect | First controlled test |
|---|---|---|
| Relevant source never appears | Corpus coverage, indexing, query rewriting or embedding search | Confirm source is indexed; run exact-keyword and semantic retrieval separately |
| Correct source exists but appears low | Ranking, chunking or top-k selection | Compare graded relevance across retrieval and reranking variants |
| Correct source retrieved, wrong fact generated | Prompt use, answer synthesis or contradictory context | Inspect exact passages and claim support in the final answer |
| Answer matches an obsolete policy | Document currency and version precedence | Check timestamps, supersession rules and authoritative version |
| Claim is right but citation does not support it | Citation selection and attribution | Audit each sentence-to-passage pairing |
| No evidence, confident fabricated answer | Abstention and confidence policy | Run an intentionally unanswerable case and review refusal behavior |
The same retrieval-then-synthesis distinction applies whether the team builds its workflow in code or uses an application platform such as Dify for RAG and agent applications. A visual workflow does not remove the need for independent test labels or evidence inspection.
Run repeatable regressions before deployment
Keep the evaluation set stable while changing one important component at a time: the embedding model, chunk size, top-k count, ranking method, prompt or generation model. Save version identifiers for the indexed corpus and the exact query set. Without those controls, an apparent retrieval gain could simply reflect a different document snapshot or a changed judge.
Compare the new configuration with a baseline on the same representative categories, not just the overall mean. Report cases that improved, cases that regressed and the severity of any unsupported or contradicted answer. A change that improves average relevance but makes rare high-impact questions less reliable may be unacceptable depending on the product.
Periodically refresh the benchmark for new products, document versions and user language, but retain a frozen regression slice to detect previously solved failures returning. Ask a human reviewer to recheck borderline or high-impact judgments whenever the evaluator model or rubric changes.
Track latency, cost and production drift
Quality must be measured alongside its operational constraints. Record median and 95th-percentile response time, retrieval duration, reranking overhead, model token usage, failure or timeout rates, and cost per satisfactorily answered request when that cost can be measured. Be explicit about hardware, model and corpus settings so figures remain comparable.
In production, monitor source freshness, retrieval mix, abstention frequency and meaningful quality regressions on a small protected sample. Log document identifiers and scores where permitted, and avoid capturing full user prompts or proprietary source text when those details are unnecessary. If the corpus is periodically reindexed, version it so a sudden change in answer behavior can be investigated rather than guessed at.
A practical release checklist
Before calling a RAG system ready, require credible evidence that relevant material is being retrieved, that response claims are supported, that answers meet the required reference facts, that citations resolve to appropriate source versions, and that unanswerable questions do not produce confident guesses. Run the same checks after every material retrieval or model change.
There is no universal retrieval precision or faithfulness threshold suitable for every product. A casual knowledge assistant and a workflow delivering consequential policy advice have different failure costs. Set thresholds from the task’s risk and human-reviewed baseline, and keep the test cases needed to explain every important regression.
