Conformal RAG Guarantees: A Loss-and-Denominator Audit

Executive Summary

Conformal RAG guarantees certify a specified statistical event under a specified calibration design. They do not, by themselves, certify that a particular emitted answer is correct. Quach et al.'s Conformal Language Modeling targets a candidate set containing at least one acceptable response. (Source: proceedings.iclr.cc) (Source: research.google) Kang et al.'s C-RAG analyzes upper confidence bounds for bounded generation losses. (Source: proceedings.mlr.press) Wang, Fan and Wang's SAFER separates finite-budget sampling from filtering and permits abstention when its sampling target cannot be met. (Source: proceedings.iclr.cc) ^[1] These are different objects, with different losses and populations.

A guarantee review must preserve the probability level as well as the loss. Conformal Risk Control supplies an expectation guarantee under its stated conditions; a confidence statement about population risk after calibration is a different assertion. ^[2] Learn then Test provides a framework for testing candidate configurations, including error conditional on predicting. ^[3] A synthetic construction makes the composition problem decisive: every returned set can contain an admissible answer while a selector always chooses an inadmissible member. Perfect existential set coverage then coexists with zero selected-answer correctness.

Abstention creates another denominator boundary. In a second, explicitly synthetic workload, answering 20 of 1,000 questions and making 10 answer errors produces a 2% answer rate, 1% unconditional failure rate, and 50% error among emitted answers. None of these percentages substitutes for another. Claim-level support needs another denominator: FActScore, for example, measures supported atomic facts rather than the event that a complete answer contains any error. ^[4] Source-relative faithfulness also differs from external truth; RAGTruth labels unsupported or contradictory content relative to retrieved material. ^[5]

The appropriate release artifact is a guarantee ledger, a frozen evaluation specification, and separate outcome counts. The scoped reproduction here uses a prespecified finite family of one-sided binomial tests, with multiplicity correction, rather than asserting a newly validated RAG benchmark. ^{[6] [7]} Its synthetic worksheet sets target error to 10%, calibration failure probability to 5%, and tests five thresholds; the resulting per-test cutoff is 0.01. Even zero observed errors requires 44 accepted calibration cases to pass that particular test. Independent development, calibration and test partitions, with document or entity grouping where needed, protect the interpretation of those counts; grouping alone does not establish independence. ^[8]

Introduction and Background

The useful question is not whether a retrieval-augmented generation (RAG) system is “conformal.” It is whether the theorem controls the quantity a reviewer intends to use. An answer set containing an acceptable response, a small mean generation loss, and a low error fraction among emitted answers are separate targets. Published language-generation work illustrates the first target, while risk-controlling prediction sets make the population-loss perspective explicit. (Source: research.google) (Source: proceedings.iclr.cc) ^[9]

Define an admissibility predicate A(x, y, E, v) that states whether answer y is acceptable for question x against evidence E under policy version v. Define a prediction set C(x), a selector g that can choose one member, and an acceptance indicator B(x) that determines whether a single answer is emitted. A bounded loss L lies in a declared interval, usually normalized to [0, 1]. These are proposed audit definitions, not interchangeable terms borrowed from different papers.

Faithfulness concerns a relation to supplied evidence. External truth concerns whether the asserted proposition is true beyond that evidence. Retrieved text can support a proposition without establishing its external truth. RAGTruth's retrieved-content labeling and FActScore's knowledge-source support criterion demonstrate why an evaluation must name its evidence relation. ^{[5] [4]} A statistical procedure protects the labels and loss supplied to it; uncertainty about the evaluator remains uncertainty about the target.

For Reliable Context's context-engineering readership, the practical concern is whether the evidence and label behind an evaluated answer can be reconstructed when a guarantee is invoked. This report compares statistical research procedures, not systems of record. Its source-backed label protocol follows World Wide Web Consortium (W3C) guidance on recording dataset versions and changes. ^[10]

The literature audit is anchored to 18 October 2026. No new empirical benchmark was run. Paper versions, official implementation descriptions and exact statistical calculations were inspected; every numerical example below is explicitly synthetic. Search-volume estimates and unpublished deployment performance play no role in the conclusions.

Key Changes

From answer confidence to an explicit event

The central event in Conformal Language Modeling is existential: some candidate in C(x) passes the admission function. Its Equation (1), established through Theorem 4.2, has the form P_cal{P_test[exists y in C(X): A(y)=1 | calibration] >= 1−epsilon} >= 1−delta. Here epsilon bounds set miscoverage and delta bounds calibration failure. ^[11]

It is not conditional coverage for every particular question x. Nor does it identify which candidate should be selected. The separately calibrated component procedure in that work requires its own loss and calibration; it cannot be inherited by arbitrary post-processing. (Source: research.google)

From one loss to a loss-specific guarantee

C-RAG's illustrative loss is 1 − max_candidate ROUGE(candidate, reference). ROUGE means Recall-Oriented Understudy for Gisting Evaluation, an overlap-based comparison with human references. ^[12] Its calibration statistic averages question-level losses. Proposition 1 presents a generation-risk confidence bound, and Proposition 2 handles valid configurations with family-wise correction. The relevant audit object is bounded generation-loss confidence analysis, not a guarantee for any chosen sentence. ^[13]

Terminology requires care: C-RAG defines R on an output set and reference as a per-example loss, while its confidence analysis uses calibration means. Proposition 1 prints P[R(T_lambda(x),y) <= alpha_hat_lambda] >= 1−delta; its proof uses an expectation and concentration of the calibration average. ^[14] Retain that notation instead of claiming per-question correctness. Its publisher abstract supports the bounded-risk framing; it does not establish selective faithfulness for a newly added selector. (Source: proceedings.mlr.press) ^[15]

From unlimited sampling to a finite budget

Finite sampling changes what can be covered. Conformal Language Modeling returns its accumulated set when its stopping condition or cap is reached. Calibration can instead return null if no configuration is certified. These are distinct branches; reducing the cap can reduce achievable non-null coverage targets. ^[11]

SAFER makes sampling-budget calibration and subsequent filtering separate stages. Its publisher describes abstention when the sampling target cannot be satisfied within the maximum cap; its project describes inference as forming a final prediction set. Neither description supplies an oracle that recognizes the correctness of every test-time candidate. (Source: proceedings.iclr.cc) ^{[16] [1]}

From expectation to calibration confidence

For monotone, right-continuous, bounded exchangeable losses with a feasible terminal setting, Conformal Risk Control establishes E[L_new(lambda_hat)] <= alpha. Its expectation includes calibration and test randomness. This does not mean that, with confidence 1−delta, each realized calibration run has conditional population risk at most alpha. ^[2]

Conversely, Learn then Test tests risk hypotheses and controls erroneous certifications across candidate configurations. Its selective-classification example conditions error on making a prediction. An adaptation must preserve both the hypothesis and that accepted-case denominator. Calling either procedure “conformal calibration” does not erase the distinction. ^[3]

From exchangeability to stated shift assumptions

Exchangeability means that the joint distribution is invariant to reordering observations; independent and identically distributed (i.i.d.) observations satisfy it. The binomial reproduction later in this report assumes i.i.d. question-level observations, a stronger and explicit requirement. Non-exchangeable conformal results require their own conditions and, in some cases, a coverage penalty. ^[17]

Distribution-shift results are similarly specific. Weighted conformal prediction under covariate shift uses a known likelihood ratio in its exact formulation. (Source: proceedings.neurips.cc) C-RAG's shift theorem assumes a bounded Hellinger distance and additional radius conditions. ^[13] Work beyond exchangeability analyzes specified departures from the usual setup. A chronological stress test cannot replace those assumptions. ^[18]

A guarantee ledger for published methods

Table 1 records the minimum source-specific distinctions. “n” denotes the calibration sample size chosen for a replication, not an invented universal sample size. Risk symbols retain their source-specific meanings. A theorem label is useful only when attached to its version, loss and probability event. ICLR denotes the International Conference on Learning Representations. (Source: proceedings.iclr.cc)

The ledger prevents a common composition error: a quantity can remain mathematically controlled while the deployment decision uses another quantity. C-RAG's official workflow separates raw calibration scores from bound computation; a reviewer still has to establish what those scores mean after adaptation. ^[15]

Probability levels and audit records

Conformal Language Modeling's event is a calibration-conditional statement about population probability. ^[11]

A populated ledger should describe a single frozen policy. A changed policy gets a new row linked to its predecessor, allowing a reviewer to identify changes in the output object, source snapshot and acceptance denominator.

SAFER deserves a probability-level annotation. Its filtering calibration uses questions whose sampled set already contains an admissible answer, with correction (n-prime * empirical_loss + 1)/(n-prime + 1) <= beta. Its filtering argument bounds expected binary loss on that attainable population, while Equation (15) prints a nested calibration-confidence statement for final set miscoverage. This report records those as published forms; it has not independently validated an upgrade from expected filtering loss to high-probability conditional filtering risk. That derivation remains a replication check, not a selective-answer certificate. Algebraically, if beta is below 1/(n-prime+1), even zero observed filtering errors cannot satisfy the displayed correction. Small attainable counts therefore need an explicit no-certified-threshold outcome. ^[19]

For each deployed configuration, extend the ledger with:

• Population: eligible questions, exclusions, source domains and workload period.
• Output object: sampled set, retained set, selected answer or selected claims.
• Admission policy: source-support rule, relevance rule and handling of unavailable evidence.
• Loss: exact function, range, direction and unit of observation.
• Numerator: the events counted as errors or miscoverage.
• Denominator: all questions, attainable questions, emitted answers or emitted claims.
• Conditioning: acceptance, attainability, subgroup and calibration information.
• Randomness: calibration draws, test draws, generation and evaluator variability.
• Parameters: risk tolerance, confidence failure probability and calibration size.
• Selection correction: candidate family, threshold search and multiple endpoints.
• Assumptions: independence, exchangeability, boundedness or a quantified shift condition.
• Composition boundary: selector, rewriting, truncation and other later operations.

These fields are an audit specification. Their purpose is to expose an unproved implication before it becomes a release claim.

Implementation Considerations and Process Changes

Freeze the evaluation contract before calibration

Separate development, calibration and held-out testing. Development chooses scores, candidate caps, selectors, label instructions and threshold candidates. Calibration certifies only that frozen family. Testing estimates the behavior of the resulting frozen policy. Reusing the test set to tune choices compromises its generalization interpretation, as the scikit-learn evaluation guidance explains. ^[20]

Questions sharing source documents, entities or repeated templates may carry dependence. Assign connected document/entity groups to partitions before inspecting results. GroupKFold illustrates non-overlapping group allocation, but grouped splitting is a leakage discipline, not a proof that the remaining units satisfy the binomial model. ^[8] The National Institute of Standards and Technology (NIST)'s AI Technology Evaluation similarly uses sequestered evaluation to mitigate train/test contamination. ^[21]

The proposed source-backed protocol should prespecify:

• Eligibility: which questions enter the workload denominator before any answer is generated.
• Source snapshot: permitted corpus, document versions and retrieval observation dates.
• Question grouping: repeated questions, shared entities and connected source-document families.
• Generation policy: model version, prompt, sampling settings and candidate cap.
• Selector: the exact rule turning candidates into an emitted answer.
• Acceptance: score direction, fixed threshold candidates and empty-output handling.
• Admissibility: whether each required proposition must be supported by an available source span.
• Human adjudication: independent initial labels, disagreement resolution and an unresolved category.
• Evaluator audit: blinded human checks of automated labels and documented disagreement counts.
• Abstention: whether each refusal is a non-answer, set failure or another prespecified outcome.
• Unavailable evidence: an explicit not-found or blocked reason, never an inferred correct label.
• Multiplicity: endpoints, configurations and workload slices sharing a confidence budget.

This is a proposed design, not a claim that adjudication makes the labels infallible. SAFER's official artifact offers several admission criteria, illustrating why the chosen criterion belongs in the ledger. ^[16] Its documented options include semantic similarity, bidirectional entailment, ROUGE-L and a large language model (LLM) evaluator. ^[16] An evaluator's agreement with reference answers cannot silently replace source-backed support.

Make labels reconstructible

A defensible record should contain question ID, candidate or emitted answer, claim boundaries, admissibility decision, source span, source version, observation date, label-policy version and adjudication status. W3C's versioning guidance supports explicit dataset versions; the PROV Ontology (PROV-O) represents derivation between an entity and its evidentiary predecessor. ^{[10] [22]}

Keep source publication time, source observation time and label creation time separate. ^[22] Reconstructibility lets a reviewer distinguish a changed answer from a changed source or label rule. It does not prove the source correct. Report the fraction of cases with unavailable evidence and evaluate that category under the prespecified policy, rather than removing difficult cases after calibration.

Audit adaptations rather than inheriting a name

Begin with the published algorithm and its author-linked artifact, then record the exact paper version and immutable software revision used. Author publication records and official repositories provide provenance for that starting point. ^{[23] [24]} No paper-to-code equivalence or new benchmark was established here.

Changing to source-backed human labels, adding an answer selector, reducing the cap or changing acceptance changes the statistical object. If the revised loss no longer satisfies a theorem's assumptions, report an empirical evaluation or use a procedure whose hypotheses match it. A raw calibration-score file and a familiar algorithm name are insufficient evidence of transfer. ^[15]

A scoped reproduction of selective-risk testing

The selected reproduction is a faithfully specified statistical test, adapted from Learn then Test's selective-error construction. It is not a reproduction of a published RAG performance result. Let each i.i.d. eligible question produce a frozen answer, a score and a binary inadmissibility label Z. For each of M thresholds fixed independently of calibration, let B_j denote acceptance, m_j = sum B_j, and k_j = sum B_j Z. The target is r_j = P(Z=1 | B_j=1), provided acceptance probability is positive. ^[3]

Test the unsafe hypothesis H_j: r_j > alpha using the conservative boundary p-value p_j = P[Binomial(m_j, alpha) <= k_j]. Conditional on the accepted count, the binomial model follows from the i.i.d. sampling and frozen acceptance rule. It does not follow merely from arbitrary dependence or repeated source-linked questions. The documented binomial test offers the corresponding lower-tail alternative. ^[6]

Certify j only if p_j <= delta/M. Set p_j to 1 when m_j is zero and do not certify that candidate. The finite-family union bound controls the probability of falsely certifying any unsafe candidate, even though thresholds use overlapping observations. Thus a later choice among certified candidates preserves this particular population-risk claim. The correction requires the original prespecified M, not the number that happened to look promising. ^[7]

Table 2 is a synthetic calibration worksheet, with 200 eligible calibration observations, M=5, alpha=0.10 and delta=0.05. Counts represent nested acceptance sets. P-values were newly calculated by summing exact binomial probabilities; they are not study measurements.

The zero-error rows show why empirical accuracy is not a certificate. Under this synthetic specification, zero errors passes only when 0.9^m <= 0.01, giving a minimum m of 44. Candidate E could be selected for its larger accepted count among the certified candidates. Its one-sided exact upper bound at per-candidate confidence 99% is approximately 9.6971%, obtained by inverting the same binomial tail. Exact interval inversion is the Clopper-Pearson construction. ^[25]

The reproduction can be implemented without ambiguity:

• Freeze: save the candidate family and risk/confidence parameters before calibration.
• Count: calculate integer accepted counts and integer accepted errors.
• Test: evaluate the binomial lower tail at the target alpha.
• Correct: compare every p-value with the prespecified delta/M.
• Select: choose only among certified candidates using the declared utility rule.
• Fail closed: report no certified operating point if none passes.
• Preserve: retain all candidate counts, p-values and selection decisions.
• Evaluate: apply the chosen frozen policy once to the held-out workload.

The official Learn then Test repository supports reproduction of its own experiments; this worksheet instead specifies the adaptation completely and reports its scope. ^[24] NIST's binary-response testing note reinforces that sample size and acceptance criterion should be designed together. A fixed-sample certificate does not authorize repeatedly collecting calibration data until a favorable p-value appears. ^[26]

Data Analysis and Evidence

Synthetic counterexample: coverage survives, selection fails

Consider 100 synthetic questions. For every question, return the set {a_good, a_bad}, with exactly one member admissible under a fixed, stipulated predicate. Let the downstream selector always choose a_bad. Set coverage is 100/100, set miscoverage is 0/100, and selected-answer error is 100/100. This deterministic construction is not a sampled benchmark; no sampling interval applies.

The logical point matches the scope of the published candidate-set event: “there exists an admissible member” does not imply “the selected member is admissible.” Conformal Language Modeling's publisher description makes the former target explicit. (Source: proceedings.iclr.cc) A selector guarantee would require additional assumptions or direct calibration of selector loss. Changing a score threshold is insufficient unless its test targets that loss.

Synthetic counterexample: refusal changes the denominator

Now stipulate 1,000 eligible questions, 20 emitted answers, and 10 inadmissible emitted answers. Suppose each emitted answer contains five assessed claims and each of the ten inadmissible answers has exactly one unsupported claim. There are 100 emitted claims and 10 unsupported claims. These synthetic counts expose denominator differences.

Table 3 separates the outcomes. The intervals are newly calculated two-sided 95% exact binomial intervals for the stipulated question-level counts, conditional on an i.i.d. sampling interpretation. They illustrate uncertainty calculations, not evidence of measured system performance. ^{[25] [6]}

Refusing most questions makes the unconditional failure rate small while leaving half of emitted answers inadmissible. Counting claims instead of answers changes the unit again. FActScore's atomic-fact support objective and RAGTruth's case/word annotation units justify keeping those units explicit; neither source licenses substituting one for another. ^{[4] [5]}

What a held-out report must show

Study scope is not transfer evidence. C-RAG reports evaluation across four datasets and four retrieval models; those design counts do not measure selective source-faithfulness for the proposed workload. (Source: proceedings.mlr.press) SAFER's documented final object is a filtered prediction set. It does not certify a new deployment's emitted-answer accuracy. ^[1]

A held-out report should publish:

• Set coverage: questions with an admissible retained member, with the declared abstention policy.
• Set size: raw retained sizes and the distribution, including empty outputs.
• Emitted-answer risk: inadmissible emissions divided by all emissions.
• Unsupported-claim risk: unsupported assessed claims divided by all assessed emitted claims.
• Any-error answer rate: answers containing at least one unsupported claim, with denominator named.
• Answer rate: emitted answers divided by all eligible questions.
• Uncertainty: raw counts, interval method and the sampling unit.
• Slice results: fixed source domains, question types and source-change periods.

For set miscoverage, use sets containing no admissible member divided by test questions returning sets under the declared policy. If the theorem instead treats abstention as failure or operates over all eligible questions, reproduce that denominator too. Never drop abstentions retrospectively and retain the original theorem's label. Prespecify interval multiplicity when claiming simultaneous slice performance; ordinary per-slice intervals are not simultaneous guarantees. ^[7]

Implications and Future Directions

A deployable claim must name the controlled loss, conditioning population, probability level and frozen policy. A suitable claim template is: “For the specified question population and frozen acceptance rule, this prespecified test controls the probability of certifying selective inadmissibility risk above alpha at delta, under the stated i.i.d. and label assumptions.” It promises neither correctness of each answer nor truth of the labels.

Marginal versus conditional coverage remains a substantive boundary. Coverage averaged across possible test inputs does not establish coverage at every input or within every subgroup. Distribution-free conditional-inference limitations motivate additional assumptions or explicitly relaxed targets. ^[27] Risk-controlling prediction sets likewise center a declared future loss and holdout calibration, not a universal answer-level assertion. ^[9]

A source-change challenge should order source snapshots chronologically and evaluate the frozen policy on later evidence. TimeSeriesSplit documents the temporal problem of training on future observations and evaluating on past observations. ^[28] Such a challenge provides empirical evidence about that change, not distribution-free protection against arbitrary future drift. Weighted covariate-shift and beyond-exchangeability results require their respective mathematical conditions. (Source: proceedings.neurips.cc) ^[18]

Recalibration review should be triggered by changes to:

• Generator: model revision, prompt, decoding or sampling randomness.
• Retriever: corpus access, ranking, filtering or source selection.
• Source corpus: additions, removals, revisions or permission changes.
• Candidate cap: budget, stopping rule or exhaustion policy.
• Selector: ranking, voting, rewriting or answer truncation.
• Acceptance: score, threshold family or abstention rule.
• Labels: evidence policy, evaluator, adjudication or admissibility standard.
• Population: new source domains, question mix, repeated-query structure or workload period.

Not every change necessarily invalidates every theorem; the point is to reassess whether its assumptions and random variables still describe the system. Dataset histories and source derivations make that comparison reviewable. ^{[10] [22]} If dependence requires a different conformal method, use its actual coverage penalty and conditions rather than attaching the old certificate to a new population. ^[17]

Research should prioritize transparent composition results, evaluator-error accounting and dependent-question designs. Better uncertainty scores can improve efficiency, but the certificate still depends on the calibration target and test. Sample-size planning and sequestered evaluation remain practical controls for making the resulting claim testable. ^{[26] [21]}

Conclusion

Conformal RAG guarantees answer a narrow question precisely when output, admissibility, loss, denominator and probability level are preserved. Candidate-set coverage establishes the existence of an admissible member under the specified procedure. It supplies no general rule for identifying that member. The published language-modeling target is therefore useful, but it must remain distinct from selected-answer correctness. (Source: proceedings.iclr.cc)

Bounded generation-loss analysis can support another decision, provided the loss matches what the organization values. Finite-budget sampling and filtering introduce further events and conditioning populations. C-RAG's upper-bound framing and SAFER's final prediction-set framing should remain visible in any evaluation summary. (Source: proceedings.mlr.press) ^[1]

The synthetic examples show why a release review needs separate counts. Perfect set coverage can coexist with a selector that always errs. A low unconditional failure rate can coexist with high error among emitted answers. A zero-answer policy has zero answer rate and undefined selective risk. Neither an attractive score nor a favorable denominator establishes a stronger claim.

The scoped binomial reproduction offers a concrete alternative: freeze the policy and candidates, count accepted errors, apply the declared correction, and retain an uncertified outcome when evidence is insufficient. Its calculations are synthetic demonstrations, not deployment measurements. Sample size and acceptance criteria must be planned jointly. ^[26]

The final deliverable should be a reconstructible claim, not the adjective “conformal.” It should identify the theorem and version, show the loss and raw counts, state the population and assumptions, and describe later operations that remain uncovered. When that correspondence cannot be established, an explicitly scoped empirical evaluation is the defensible result.