Hallucination

Hallucination in natural language generation refers to output that is unsupported by the source material or inconsistent with facts relevant to the task. The term covers more than invented names: a model may attach a real fact to the wrong entity, turn a possibility into a certainty, or state a date absent from its evidence. Research distinguishes errors relative to supplied source content from errors relative to outside knowledge, so an evaluation must specify its reference frame.

Detection requires looking at individual assertions and the material that could support them. A fluent paragraph can contain one unsupported clause among correct ones. Citation checking, evidence-span review, and targeted factual verification help expose that pattern. Uncertainty should be expressed when the evidence is incomplete, and abstention may be preferable to a forced answer. A model's confidence wording is not a calibrated probability unless separately tested.

For a defensible record, a hallucinated detail can be propagated into later decisions if its origin is not visible. Grounding and faithfulness checks reduce that risk, but they cannot repair an unreliable source or a mistaken entity match. The question is not only whether a sentence appears in retrieved text; it is whether the cited evidence supports this specific claim about this specific subject at the relevant time.

Causes include gaps and errors in training data, a training objective that rewards fluent continuation over calibrated uncertainty, ambiguous prompts, and failures to use supplied context. Models can also mimic common falsehoods: the benchmark TruthfulQA was built from questions on which humans often hold false beliefs, to test whether models reproduce them. Detection options include checking each claim against retrieved evidence, comparing multiple samples for consistency, and asking another model to judge support, each with its own error rate. Mitigations include grounding through retrieval, requiring citations, allowing the system to abstain, and verifying generated claims before release.

No method removes the risk entirely, so a defensible process measures it. Sample outputs, label unsupported claims, and report the rate with its uncertainty and its breakdown by type of question. Keep the labelled sample as a regression set so that a change to the model, the prompt or the corpus can be tested against it. The article on what fact-verification benchmarks test helps in choosing public benchmarks and in understanding what they do not cover.

Ask for the measured rate of unsupported claims, and how it was measured, on what sample, by whom. Ask whether numbers, dates and names are checked mechanically against the source. Ask what the system does when it lacks evidence. Read twenty outputs and mark every claim; compare the tally with any reported figure.