Dawid–Skene model

The Dawid–Skene model addresses cases where several annotators label the same items and no complete gold standard is available. It treats each item's true class as latent and estimates how often each annotator emits each observed class. The original method uses maximum likelihood estimation with an expectation–maximisation procedure. Its output depends on the chosen class set and the assumptions connecting annotators, items and errors.

A practical use is adjudicating repeated categorical reviews, such as whether a source statement supports, contradicts or does not address a claim. The input must keep annotator identities and missing assignments distinct. Sparse overlap and rare classes can make estimates unstable; a small set of expert-adjudicated items can help inspect whether the inferred labels and error patterns are plausible.

For an organization defending a classification, the model is evidence about agreement and annotator behaviour, not a replacement for the underlying annotations. Reviewers should retain individual judgments and inspect high-impact disagreements. It is related to truth discovery because both infer an unobserved answer from fallible reporters, but annotators' labels are not automatically independent sources of factual evidence.

A typical use is a categorical judgment that several reviewers each make, for example whether a passage supports a claim, contradicts it, or does not address it. Each reviewer's label is kept. The model's estimate of the true class is kept beside the labels and never in place of them. The confusion matrices are a diagnostic in their own right: a reviewer who often reports supports when the consensus is does not address is revealing a systematic reading of the guidelines, which can be corrected through training or a clearer definition.

Where overlap is sparse, the estimates are unstable, and a few items adjudicated by an expert, a small gold standard, give a way to check whether the inferred classes and error patterns are plausible. Report inter-annotator agreement as well, using a coefficient suited to the data, as surveyed in Artstein and Poesio (2008); agreement and the model answer different questions, the first about consistency among annotators and the second about the likely true class.

The model sits between simple voting and full truth discovery. Compared with majority voting, it estimates how each annotator errs instead of counting everyone equally. Compared with TruthFinder, it works on a fixed set of categories and a known annotator pool, not on open-ended values from web sources. Its outputs feed calibration when the inferred class probabilities are used downstream, and its estimated labels are a weak substitute for a real gold standard, which should be built from adjudicated items when decisions depend on the result.