Skip to content
Reliable Context
Read the paperRequest access

Methods

Estimating source reliability

Assess a source for a defined property and period using checked outcomes and visible uncertainty.

Problem

A source is not uniformly reliable. A register may be authoritative for an identifier and incomplete for a trading name; an article may report a measured outcome well and an address incidentally. Reliability estimates are useful only when tied to a question, period and verification set. Agreement among sources is weak evidence when they copy one another. The aim is an assessment of observed performance and limitations, not a permanent reputation score that substitutes for reading a particular claim.

Steps

  1. Define the task

    Specify entity class, property, geography and period. Define correct, incorrect, outdated and unassessable assertions. Keep coverage separate from correctness: a source can be accurate when it speaks but omit many entities. A broad label such as "trusted source" hides these dimensions and invites transfer to uses the assessment never tested.

  2. Construct an assessment set

    Sample assertions across the relevant range, including difficult and recently changed cases. Establish reference answers from independent evidence with their own dates and limitations. Record the sampling method so an estimate drawn mostly from easy cases is not mistaken for overall performance. If the reference answer is disputed, retain the dispute rather than forcing a false ground truth.

  3. Measure and inspect dependence

    Count supported, contradicted, stale and unavailable claims separately, with denominators and uncertainty. Inspect whether apparent confirmations derive from the same original publication, data feed or filing. A majority may reflect copying. Where the sample is small or selected for one failure mode, describe the result as local evidence rather than a general estimate.

  4. Use the estimate narrowly

    Apply the estimate only to claims within its tested scope and period, and keep direct claim evidence available. Reassess after source policy changes, revisions or a new population of subjects. An estimate can guide review priority; it does not make an unsupported individual assertion true. Document how it affected a decision so a reader can challenge that use.

Checks

  • Reliability is scoped to a property, population and period.
  • Coverage and correctness have separate measures.
  • Sampling, denominators and disputed answers are visible.
  • Copied sources are not treated as independent votes.

Pitfalls

  • Assigning one permanent score to every claim.
  • Using agreement as truth without checking dependence.
  • Ignoring unavailable cases in the denominator.

A worked example

A fictional team wants to know how far to trust three sources for the headquarters country of pharmaceutical companies: a government register, a commercial directory and a news archive. They draw a sample of companies stratified by size and by whether the company moved in the past five years, and establish reference answers from the companies' own filings.

The register is right for most companies it covers and covers only part of the sample, so its coverage is low and its correctness high. The directory covers nearly everything and is wrong more often for companies that moved, since it updates slowly. The news archive covers few companies and is accurate when it speaks. The record gives each source a coverage figure and a correctness figure for this property and this population, with the denominators and the failure types. It does not publish a single trust score.

Variations and edge cases

Where no reference answers are available, reliability can be estimated from agreement, and the literature offers many methods for this. They rest on assumptions, chiefly that errors are independent. When sources copy one another, agreement overstates reliability. A useful step is to test the assumption: look for sources whose errors coincide, and compare estimates with and without them.

Reliability differs by topic. A source can be dependable for identifiers and unreliable for narrative claims. Estimates should be kept per property or class of property, and a new property should start with no estimate rather than inheriting a general one.

Using the estimate

An estimate has a defined job: to prioritize review, to set the order in which sources are consulted, or to weight evidence in a rule that is itself reviewed. It does not turn a doubtful claim into a fact. Where an estimate affects a decision, the decision record names it, and a reader can set it aside and see what the evidence says without it. Estimates should expire. Sources change their practices, and an estimate made on last year's population is a hypothesis about this year's.

How the method is checked

Hold out part of the reference set and check whether the estimate predicts performance on it. Compare against a naive baseline, such as counting votes. Inspect where the estimate and the baseline disagree, and check those cases by hand. If the estimate improves results only on cases that were easy anyway, it is not adding information. Report intervals, not point values, when the sample is small.

Related terms

See also

Further reading in the articles

References

Found an error on this page? See corrections for what counts as one, how to report it and the dated log.

Enquiry

Test this method on one entity.

Name an entity where "Estimating source reliability" would be checked in practice, with its sources and its readers. We reply with what a first record would cover and what it would need from you.

Open a structured enquiryContact and what happens next

Or write to [email protected] in your own words.