Source reliability

Source reliability is a property of a source within a defined task, attribute and period, rather than a permanent verdict on an institution. A registry may be dependable for identifiers yet incomplete for operational status; a report may be accurate when published but stale later. Reliability can be measured against known answers, estimated from agreement patterns or assessed through documented editorial and collection practices.

Practitioners distinguish error from coverage. A source that rarely reports a field is not necessarily inaccurate when it does, while a source that repeats another source does not add an independent observation. Estimates also depend on case mix: performance on common entities may not carry over to obscure ones. Reliability should be revisited after a source changes ownership, collection method, scope or update cadence.

For a defensible record, the relevant question is why this source was trusted for this particular claim at this particular time. Retaining the source statement and the basis for its use makes that judgment reviewable. Source reliability informs truth discovery, but it does not by itself resolve a data conflict or prove that a specific claim is correct.

The first is comparison with known answers: a sample of claims is checked against an authoritative record and the proportion correct is reported with its sampling uncertainty. This is the most direct evidence and the most expensive, so it is usually done on a stratified sample. The second is inference from agreement. Methods such as TruthFinder and the models in the survey by Li and colleagues treat a source as reliable to the extent that its claims coincide with the values that the other sources support. This works when sources err independently, and fails when they copy each other, which is why source dependence has to be modelled alongside reliability.

The third is assessment of process: documented editorial review, a stated correction policy, a registry's legal duty to be accurate, the existence of an audit. Process evidence says little about the rate of error, but it says a good deal about whether an error will be noticed and corrected, which matters for the choice of what to retain. Large-scale work such as Knowledge-Based Trust estimates the accuracy of web sources from the facts extracted from them, and it separates the errors of the extractor from the errors of the page; a reliability figure that does not make that separation will penalise a source for the extractor's mistakes.

Reliability describes the source; source dependence describes the relation between sources; a data conflict describes the disagreement that both help to resolve. A reliable source can be wrong on an individual claim, and an unreliable source can be right, so reliability sets a prior and not a verdict. The stored claim with its provenance is what allows the prior to be revised when a source turns out to have been copying or has changed its practice. Where the question is how much to trust a language model's statement about a source, the relevant concept is faithfulness and not reliability.