A fluent RAG answer still needs an evidence ledger.

Define the questions your system should answer

A fictional equipment library wants a support assistant. Its policy bundle has three short documents: A says ordinary tools can be borrowed for seven days; B says extensions require staff confirmation; C says damaged tools must be reported before their next loan. None of the documents describes overdue fees. These are invented policies for an inspectable exercise, not advice about an actual service.

Create four questions before changing a model or prompt. Record the necessary evidence and the behavior expected when evidence is absent.

question	required evidence	expected behavior
How long is an ordinary loan?	A	answer seven days
Can I extend a loan automatically?	B	explain staff confirmation
What happens if a tool is damaged?	C	explain reporting requirement
What is the overdue fee?	none	say the provided policy does not specify it

The last row matters. An assistant that always produces an answer can look helpful while inventing policy. Abstention is a useful product behavior when the information is genuinely absent.

Inspect retrieval independently

Assume the first three queries each have one required document. A retriever returns A for the first, A and C for the second, and C for the third. Relevant-document coverage across these answerable questions is 2/3. The second query missed B even though it returned plausible library material.

Do not add the unanswerable question to that denominator as if it had a missing required document. Track answerable and unanswerable cases separately. Specify the retrieval cutoff and whether duplicate chunks from the same document count once. More chunks are not automatically more relevant evidence.

Trace the answer at claim level

Now suppose the first answer says: ordinary loans last seven days and extensions are free for another week. The first claim is supported by A. The second is unsupported by the provided bundle. A citation next to the sentence does not fix this: the cited source must actually support the associated claim.

For this exercise, mark each factual claim supported, contradicted or not established. A claim can be true somewhere else and still be unsupported by the allowed evidence. This distinction matters when the assistant is presenting an organization's own policy rather than answering from general knowledge.

Count appropriate abstention and false confidence

For the fee question, compare two responses: the policy bundle does not specify fees, or the fee is five units per day. The second invents a number. Mark it as an unsupported answer even if a reviewer prefers its confident tone. A helpful response can identify the missing information and suggest contacting staff without claiming to have checked an account.

Also measure unnecessary refusal: if A clearly provides the loan duration, declining to answer is a different failure. Keep both error counts visible. A single overall score can hide a system that answers recklessly or refuses nearly everything.

Compare one change on the same frozen cases

Store question IDs, document versions, retrieved document IDs, generated answers, claim labels and reviewer notes. Run the old and new configuration on the same cases. If retrieval improves while support gets worse, investigate the generator instead of declaring an overall win.

An AI evaluator can help triage large sets, but its judgments need calibration against a manually inspected sample. In this tiny exercise, direct inspection is cheaper and clearer than asking another model to guess the policy. Do not use real private support tickets without permission and appropriate handling.

Set a release decision before looking at the score

For this educational bundle, require all explicit numeric policy claims to be supported and the fee question to abstain. A real release also needs representative questions, security tests, access controls, latency and cost limits, and monitoring after launch. Passing four fixtures is evidence about four fixtures. It is not a claim of general correctness, production safety or a vendor certification.