Glossary
Faithfulness
Faithfulness measures whether a generated statement is supported by the information supplied for that generation.
Evaluation
Faithfulness asks whether a generated response stays within the information it was given. A statement may be true in the outside world yet unfaithful to the supplied evidence; conversely, a faithful paraphrase of a false source can still be factually wrong. Evaluations should therefore distinguish source support from external correctness. In retrieval-based systems, the unit of analysis is often an individual claim and the passages actually available when it was generated.
A careful review decomposes a response into checkable assertions, locates the passage claimed as support, and tests whether that passage entails each assertion. Numbers, dates, negations, and entity names deserve explicit checking. Automated judges can help triage, but their outputs should be validated against human judgments on a suitable sample. An unsupported assertion should remain visible even when the rest of a paragraph is well grounded.
Faithfulness matters when an organization wants a reader to verify a generated explanation rather than trust its fluent form. Sentence-level citations and evidence spans make the check possible, while a retained source copy prevents later page changes from obscuring what was available. Grounding is the practice of tying claims to inspectable material; faithfulness is an evaluative property of the relationship. Neither alone determines whether the underlying source deserves reliance.
Example
A summary of a Halden Therapeutics AG (fictional) notice is unfaithful if it adds an expiry date that does not appear in the supplied notice.
Faithfulness versus factuality
Faithfulness is judged against the information supplied for a generation, such as a document to be summarised or a set of retrieved passages. Factuality is judged against the world. The two come apart in both directions. A summary can be faithful to a source that is wrong, and it can be factually correct while adding content the source does not contain, which is unfaithful. The study by Maynez and colleagues on abstractive summarisation made the distinction central and found that a large share of generated summaries contained content not supported by the input. The survey by Ji and colleagues separates intrinsic hallucination, which contradicts the source, from extrinsic hallucination, which cannot be verified from it.
For a system that answers from supplied evidence, faithfulness is the property that can be tested without knowing the truth: does each statement follow from the evidence given? The test is limited when the evidence itself is wrong, which is why it is paired with checks on the sources, as described under source reliability.
Measuring it
Methods fall into three groups. Human judgment is the reference: raters see the source and the statement and decide whether the source supports it. Entailment-based metrics use a trained model to predict whether the source entails the statement; the benchmark TRUE re-evaluated such metrics across many datasets and found that they differ in reliability by task. Model-based judgment prompts a language model to rate support, and the study by Zheng and colleagues on LLM judges describes both their promise and their biases. Toolkits such as Ragas package faithfulness-style scores for retrieval-augmented systems, and FActScore decomposes long text into atomic facts and measures the share supported by a knowledge source.
Whichever method is used, validate it against human judgments on a sample from the same domain, and report the agreement. An automatic score that has not been checked against people may measure something other than faithfulness, such as fluency or lexical overlap. The article on claim-level evaluation of LLM citations and the one on a groundedness calibration protocol describe how to do this.
A worked example
Given a filing from Halden Therapeutics AG (fictional) that reports a net loss for two years and a headcount at year end, a model produces three sentences: the net loss figures; a statement that the loss narrowed; and a statement that the company reduced its headcount. The first is faithful. The second is faithful only if the figures show a narrowing, which a check against the table confirms or refutes. The third is unfaithful because the filing gives one headcount and no comparison; a reduction is an inference the source does not support. The scored result is one faithful sentence, one confirmed after checking, and one unfaithful, and the third is removed. The check required reading both the text and the table, which is why automated scores should be tested on documents with numbers.
Checks a reviewer can run
Ask which faithfulness measure is used, with what evidence it was validated, and by what agreement with people. Read a sample of outputs against their sources. Check that numbers, dates and names are verified mechanically. Test with source passages that contradict the model's prior knowledge, to see whether the output follows the source or the model's memory.
Related terms and boundaries
Faithfulness is the measure of grounding, the opposite of the extrinsic part of hallucination, and the property that sentence-level citations make checkable. It is evaluated against a gold standard of human judgments and reported with inter-annotator agreement.
Common misreadings
- Calling a claim faithful merely because it is independently true.
- Scoring a whole answer while ignoring one unsupported clause.
- Treating a citation marker as evidence entailment.
Appears in
- Auditing an LLM answer against its sources
Split an answer into claims and inspect cited passages for support, inference and missing qualifications.
References
- Ji et al., Survey of Hallucination in Natural Language Generation, ACM Computing Surveys (2023).
- Maynez et al. (2020), On Faithfulness and Factuality in Abstractive Summarization, ACL.
- Ji et al. (2023), Survey of Hallucination in Natural Language Generation, ACM Computing Surveys.
- Honovich et al. (2022), TRUE: Re-evaluating Factual Consistency Evaluation, NAACL.
- Es et al. (2023), Ragas: Automated Evaluation of Retrieval Augmented Generation.
- Min et al. (2023), FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation, EMNLP.
- Zheng et al. (2023), Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena.
Found an error on this page? See corrections for what counts as one, how to report it and the dated log.
Enquiry
Where does "Faithfulness" bear on a decision of yours?
This entry sits under evaluation. If a decision in your organization turns on it, send the entity concerned, the sources you fuse to describe it and who may read which of them. We reply with what a first record would cover.
Or write to [email protected] in your own words.