Why Language Models Hallucinate: Incentives and Abstention

What counts as a hallucination

A language model hallucinates when it produces fluent, plausible content that is false or unsupported. The term has several definitions, and measurement requires fixing one.

The natural language generation survey by Ji et al. defines hallucination as generated content that is "nonsensical or unfaithful to the provided source content" and splits it in two ^[1]:

• Intrinsic hallucination: output that contradicts the source. Example: a summary dating the first Ebola vaccine approval to 2021 when the source says 2019.
• Extrinsic hallucination: output that cannot be verified from the source, meaning the source neither supports nor contradicts it. The survey notes that such content may be correct background knowledge, but most of the literature still treats it with caution.

Huang et al. redraw the taxonomy for large language models (LLMs). Factuality hallucination is a discrepancy between generated content and verifiable real-world facts. Faithfulness hallucination is divergence from the user's instruction, from the provided context, or from the model's own earlier statements (logical inconsistency) ^[2].

The US National Institute of Standards and Technology (NIST) Generative AI Profile lists confabulation as a risk category: "the production of confidently stated but erroneous or false content (known colloquially as 'hallucinations' or 'fabrications') by which users may be misled or deceived" ^[3]. The definition makes confidence part of the harm: a wrong answer flagged as a guess is a different risk from one stated as fact.

This article uses hallucination to mean a plausible, confidently stated claim that is false or unsupported by the evidence the system is supposed to rely on. Kalai, Nachum, Vempala and Zhang use the same notion, "plausible falsehoods" ^[4]. Their account has two stages: pretraining creates errors, and post-training and evaluation fail to remove them.

Pretraining: statistical reasons facts go missing

Pretraining fits a base model to the distribution of text in a corpus by minimising cross-entropy loss. Kalai et al. argue that this objective produces errors even when every training document is true: "Hallucinations need not be mysterious—they originate simply as errors in binary classification" ^[4].

The Is-It-Valid reduction

The argument links generation to an easier supervised task they call Is-It-Valid (IIV). A classifier sees candidate outputs, half drawn from the valid training distribution and half drawn uniformly from plausible errors, and must label each one valid or erroneous. Generating valid text implicitly requires answering this question, so it is at least as hard. The paper's main pretraining result, stated informally, is that the generative error rate is at least about twice the IIV misclassification rate, after subtracting a small term that measures miscalibration ^[4].

Classifiers fail for known reasons: a poor model family, distribution shift, noisy labels, or no learnable pattern at all. In the paper's examples, spelling is learnable and rarely wrong; letter counting is a poor-model problem; obscure people's birthdays have no pattern ^[4].

Arbitrary facts and the singleton rate

The no-pattern case matters most for factual systems. Kalai and Vempala formalised it in 2023 for arbitrary facts, meaning facts whose truth cannot be inferred from rules or from other training data. Their examples are "who ate what, when, where, why" factoids and references to publications ^[5].

The key quantity comes from Alan Turing's missing-mass estimator, published by Good in 1953. The share of a sample seen exactly once estimates the probability that the next draw is unseen; Kalai and Vempala call these monofacts. They show that for a model calibrated in their semantic sense, "the probability of generating a hallucination is close to the fraction of facts that occur exactly once in the training data," and that this holds "even assuming ideal training data without errors" ^[5].

The 2025 paper strengthens this result to include prompts and "I don't know" (IDK) responses. It defines the singleton rate (sr) as the fraction of training examples whose prompt appears exactly once with a non-abstaining answer. Its Theorem 2 states that, with probability at least 99%, any learning algorithm's error rate satisfies err ≥ sr − 2/min|E_c| − (35 + 6 ln N)/√N − δ. Here |E_c| is the number of wrong answers for a prompt, N the number of training examples and δ the miscalibration. For large corpora and answer spaces the bound approaches sr − δ. The paper gives its own illustration: "if 20% of birthday facts appear exactly once in the pretraining data, then one expects base models to hallucinate on at least 20% of birthday facts" ^[4].

The result has two limits. First, it covers only arbitrary facts. Kalai and Vempala say there is "no statistical reason" for hallucination on systematic facts such as arithmetic, or on facts that appear many times. Hallucinated references may stem from other causes, such as model capacity ^[5]. Second, the bound sets a floor, not a forecast. Noisy training data push error rates higher, a cause the 2025 paper calls "garbage in, garbage out" ^[4]. Huang et al. add long-tail, outdated and domain-specific gaps ^[2].

Calibration and the arbitrary-fact result

Calibration means that a predictor's stated probabilities match observed frequencies. The standard illustration is Dawid's forecaster: on days with a predicted 30% chance of rain, it should rain about 30% of the time. Kalai and Vempala apply this to pieces of information rather than tokens, because one fact can be worded in many ways ^[5].

A base model calibrated on arbitrary facts must spread probability over plausible statements it has never seen confirmed, and part of that probability lands on falsehoods. The 2025 paper states the converse: its error bound "implies that language models which do not err must not be calibrated" ^[4]. Calibration is what cross-entropy training naturally produces; the paper cites GPT-4 histograms in which the pretrained model is well calibrated on multiple-choice questions and the reinforcement-learned model is not ^[4]. Kalai and Vempala note that post-training has been used to reduce hallucination "at the cost of reducing calibration as well" ^[5].

This does not make hallucination inevitable in deployed systems. A system built from a question-answer database and a calculator, answering a fixed set of questions and saying IDK to everything else, never hallucinates. The lower bound applies to calibrated base models, and post-training exists partly to move away from it ^[4]. The question is whether post-training rewards that move.

Post-training and evaluation incentives

Post-training (supervised fine-tuning and reinforcement learning from human or AI feedback) should remove confident falsehoods. Huang et al. identify one failure inside fine-tuning: standard supervised fine-tuning "typically force[s] models to complete each response, without allowing them to accurately express uncertainty". As a result, models faced with questions beyond their knowledge "are more likely to fabricate content rather than reject it" ^[2].

Kalai et al. place the larger failure in evaluation. A binary grader gives 1 point for a correct answer and 0 for anything else, including an abstention. Their Observation 1 states that for any beliefs a model might hold about the correct answer under binary grading, the best response is never to abstain. "IDK-type responses are maximally penalized while an overconfident 'best guess' is optimal" ^[4]. Of ten popular benchmarks the paper reviews (including GPQA, MMLU-Pro, SWE-bench and HLE), nine use binary grading with no IDK credit; the tenth, WildBench, uses a rubric under which IDK may score below a fair answer containing a hallucination ^[4].

Model A signals uncertainty and never hallucinates; Model B is identical but always guesses when unsure. Under 0-1 scoring, B wins. The authors call this an "epidemic" of penalising uncertainty that a few dedicated hallucination benchmarks cannot fix, because accuracy-only leaderboards outweigh them ^[4].

Worked example: the expected value of guessing

The arithmetic below is illustrative: it applies the paper's proposed scheme, not benchmark results. Consider a question to which the model's best answer is correct with probability p. Under a scheme with confidence target t, a correct answer scores +1, IDK scores 0, and a wrong answer scores −t/(1−t). Binary grading is the case t = 0. The expected score of answering is p − (1−p)·t/(1−t), and it is positive exactly when p > t ^[4].

Under binary grading every row beats IDK, so a model selected on accuracy learns to guess. Under penalties, guessing pays only above the threshold. Now take a fictional test of 100 questions. Two fictional models both know 60 answers with certainty and, on the other 40, have a best guess that is right 25% of the time. Model A answers the 60 and says IDK to the 40. Model B answers all 100 and gets an expected 10 of the 40 right and 30 wrong.

On an accuracy leaderboard Model B ranks first and also produces 30 confident falsehoods. Under either penalty the ranking flips.

The proposed fix: explicit confidence targets

Kalai et al. propose that mainstream evaluations state the threshold in the prompt. Their example instruction reads: "Answer only if you are > t confident, since mistakes are penalized t/(1 − t) points, while correct answers receive 1 point, and an answer of 'I don't know' receives 0 points." They suggest t = 0.5, 0.75 and 0.9 as natural values ^[4]. With a stated threshold one model can be best at every threshold; with a hidden one, no single model wins except one that is always right. They call the target behaviour behavioral calibration: at each threshold, give the most useful answer the model is at least t confident in, and otherwise abstain. It is auditable by comparing accuracy and error rates across thresholds, without asking for numeric probabilities ^[4].

Why retrieval helps and where it still fails

Retrieval-augmented generation (RAG), introduced by Lewis et al., pairs a model's parametric memory (its weights) with a non-parametric memory, in the original paper a dense vector index of Wikipedia searched by a neural retriever. The motivation is that weights-only models cannot easily revise knowledge or show provenance, and may hallucinate; swapping the index updates knowledge without retraining. In a human evaluation of Jeopardy question generation, raters judged RAG more factual than a BART baseline in 42.7% of pairs and BART more factual in 7.1% ^[6].

Retrieval turns arbitrary facts the model saw once, or never, into evidence placed in front of it at question time, attacking the singleton-rate cause. It does not change the incentive. Kalai et al. note that Observation 1 "holds for arbitrary language models, including those with RAG": when search fails to find a confident answer, binary grading still rewards a guess. Nor does retrieval fix miscalculations or other intrinsic errors ^[4].

Huang et al. catalogue how retrieval-augmented systems still hallucinate ^[2]:

Huang et al. also note that RAG is "easily impacted by irrelevant retrievals" ^[2]. Retrieval moves the reliability problem into the corpus and pipeline, where it can be inspected and corrected, but the system still needs a rule for missing or contradictory evidence. See RAG vs system of record.

Abstention as a product feature

Abstention is a deliberate refusal to give an answer, such as IDK, "not found in the record", or a request for clarification. The incentive argument implies it must be designed, rewarded and recorded; it does not appear by default.

First, the threshold belongs to the application, not the model. Kalai et al. note that the ideal penalty "might reflect likely real-world harms" but depends on the problem, the application and the users. which is why they ask evaluations to state it ^[4].

Second, an abstention should say why: "no source in scope", "sources conflict" and "below confidence target" need different follow-up. The authors concede the correct/incorrect/IDK split is incomplete; hedging, omitting details and asking questions also signal uncertainty ^[4].

Third, reporting only accuracy over answered questions rewards answering less; reporting only coverage rewards guessing. Publish both.

Measuring unsupported claims

Measurement must separate supported answers, unsupported or false answers, and abstentions. For long-form output, binary scoring is too coarse because one paragraph mixes true and false statements. FActScore splits a generation into atomic facts and reports the percentage supported by a reliable knowledge source. The authors report ChatGPT at 58% on people biographies and an automated estimator with under 2% error ^[7]. Enterprise output can be scored the same way: split answers into claims and count claims without evidence.

For question answering, use behavioral calibration. Run the evaluation at several stated confidence targets and report correct, wrong and IDK counts at each one ^[4]. If errors do not fall as the threshold rises, the model ignores the incentive.

For deployed systems, NIST suggested action MS-2.5-003 is to "review and verify sources and citations in GAI system outputs during pre-deployment risk measurement and ongoing monitoring activities". MS-2.5-001 advises against extrapolating performance from narrow, anecdotal assessments ^[3].

Operating rules for factual systems

The research supports a short set of rules for systems whose answers feed decisions. Example: Halden Therapeutics AG, a fictional pharmaceutical company, runs an internal assistant answering questions about batch release, supplier qualification and stability data.

1. Unknown is a permitted answer. The response schema includes unknown as a first-class value next to the answer types, as described in declaring unknowns. Prompts, fine-tuning data and rubrics treat it as legitimate; otherwise the system reproduces the binary-grading incentive.
2. Every unknown is recorded. Store the question, time, sources searched and reason code. At Halden, an unknown for "shelf life of batch 24-117" is logged as a gap in stability data, not shown as a blank screen.
3. Thresholds are stated and owned. Each question class has a written confidence target set by the business owner, not the vendor. Batch-release facts carry a high target; meeting summaries a low one.
4. Factual values come with evidence or not at all. A stated value cites its record or document; an untraceable value is treated as an abstention.
5. Evaluations score three outcomes. Acceptance tests report correct, wrong and unknown counts with an explicit error penalty; a release that gains accuracy by guessing more fails.
6. Unknowns feed curation. Recurring unknowns are a work queue; filling gaps lowers abstentions without lowering the threshold.

These rules accept the 2025 paper's diagnosis: hallucination persists partly because systems are scored as if silence were the worst outcome. For factual work, a recorded "unknown" is often the correct answer.

Reliable Context is a system of record built around source-backed values, in which "unknown" is a valid, recorded state.