Glossary
Inter-annotator agreement
Inter-annotator agreement measures how consistently independent annotators apply the same labels.
Evaluation
Inter-annotator agreement measures consistency among people who label the same items independently. Raw percentage agreement is simple but does not account for agreement expected by chance. Cohen's kappa is a chance-corrected measure for two raters assigning categorical labels under its stated assumptions. No single coefficient captures all annotation settings, and a high value does not prove the labels are substantively correct. A shared misunderstanding can produce strong agreement.
An annotation study should define categories, sampling, instructions, and whether raters saw the same evidence. Report category prevalence and a confusion view alongside the aggregate coefficient; rare categories can make an overall figure misleading. Disagreements should be reviewed to find unclear rules, missing source material, or genuinely contested facts. Adjudication creates a final reference label but should preserve the original independent judgments for analysis.
When an organization claims an evaluation is reliable, agreement shows whether its reference judgments can be applied consistently. It helps distinguish model errors from unstable task definitions. A gold standard may still contain errors after adjudication, so source evidence remains necessary. Agreement is also different from calibration: raters can consistently assign labels while a model's confidence scores fail to match its observed error rate.
Example
Two fictional reviewers independently classify Halden Therapeutics AG (fictional) records as active, inactive, or unresolved, then examine where their labels differ.
Why raw agreement is not enough
If two annotators label items as yes or no and agree on ninety percent of them, the result looks strong. If ninety-five percent of items are no, two annotators who both label everything no would agree on ninety-five percent of items and have shown nothing. Chance-corrected coefficients address this by comparing observed agreement with the agreement expected by chance. Cohen's kappa does this for two annotators and nominal categories; coefficients such as Krippendorff's alpha extend the idea to more annotators, missing labels and different scales of measurement. The survey by Artstein and Poesio compares the coefficients as used in computational linguistics, explains how their assumptions about chance differ, and argues that thresholds for acceptable values depend on the purpose of the annotation. Implementations are available in standard libraries, including scikit-learn.
The general topic is covered in the Wikipedia article on inter-rater reliability. A coefficient summarises consistency, which is a necessary condition for validity and not a sufficient one: annotators may agree consistently on a wrong reading of the guidelines.
Using agreement to improve the task
Low agreement is information about the task. It may mean the guidelines are ambiguous, the categories overlap, the items require expertise the annotators lack, or the question has no determinate answer for some items. Examine the disagreements, not just the coefficient. Group them by item type and by label pair, and read a sample. A cluster of disagreements between supported and not addressed on items with hedged wording points to a missing rule about hedges. Repair the guidelines, retrain and re-measure. The experience reported by Snow and colleagues with non-expert annotators shows that agreement and accuracy against experts can both be estimated and that combining annotators and correcting for bias can recover high-quality labels for some tasks.
Report agreement with its context: the number of items and annotators, the label scheme, the coefficient used and why, and a confidence interval. When labels will train or test a system, agreement sets a ceiling on how well any system can appear to do. When labels will be used to judge generated answers, as in the article on a groundedness calibration protocol, human agreement is the baseline against which an automated judge is compared.
A worked example
Two reviewers each judge 100 passages as to whether they support a stated claim about Halden Therapeutics AG (fictional). They agree on 88 of them, or 88 percent. Support is the common label, so chance agreement is high: if the reviewers label support at rates of 0.8 and 0.75, the expected chance agreement is 0.8 times 0.75 plus 0.2 times 0.25, which is 0.65. Kappa is the observed agreement minus chance, divided by one minus chance: 0.88 minus 0.65 over 0.35, about 0.66. That is a moderate result and not the near-perfect agreement the 88 percent suggests. Reading the twelve disagreements shows that eight concern passages that state an expectation, and the guideline is revised to say how expectations are to be labelled. A second round gives kappa near 0.8.
Checks a reviewer can run
Recompute the coefficient from a table of labels. Ask which coefficient was used and whether its assumptions fit the data. Ask for the distribution of labels, since skewed marginals depress kappa. Read a sample of disagreements. Ask what was changed in response to them. Ask whether agreement was measured on a fresh sample after the change.
Related terms and boundaries
Agreement qualifies a gold standard and caps the meaning of any precision and recall computed against it. The Dawid–Skene model goes beyond agreement by estimating each annotator's error pattern. In null semantics terms, disagreement among annotators is itself a signal that a value is contested.
Common misreadings
- Reporting only percentage agreement for highly imbalanced labels.
- Treating agreement as proof that the shared label is true.
- Hiding pre-adjudication disagreement.
Appears in
- Evaluating extraction against a gold set
Compare extracted claims with reviewed annotations, including spans, scope and abstentions.
References
- Cohen, A Coefficient of Agreement for Nominal Scales, Educational and Psychological Measurement 20(1) (1960).
- Artstein and Poesio (2008), Inter-Coder Agreement for Computational Linguistics, Computational Linguistics 34(4).
- Wikipedia, Cohen's kappa.
- Wikipedia, Inter-rater reliability.
- scikit-learn documentation, Metrics and scoring (including Cohen's kappa).
- Snow et al. (2008), Cheap and Fast, But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks, EMNLP.
Found an error on this page? See corrections for what counts as one, how to report it and the dated log.
Enquiry
Where does "Inter-annotator agreement" bear on a decision of yours?
This entry sits under evaluation. If a decision in your organization turns on it, send the entity concerned, the sources you fuse to describe it and who may read which of them. We reply with what a first record would cover.
Or write to [email protected] in your own words.