Methods
Evaluating extraction against a gold set
Compare extracted claims with reviewed annotations, including spans, scope and abstentions.
Problem
An extraction system can return a plausible value while attaching it to the wrong entity, date or sentence. Exact string match misses these failures and can penalize harmless formatting differences. A gold set gives a reviewed basis for measuring the whole claim: subject, property, value, scope and supporting span. It also exposes disagreement among annotators and cases where the source is ambiguous. Evaluation should report what the test set represents and which error types remain hidden.
Steps
Define the target claim
Specify fields to extract and evidence required for each. Write annotation rules for aliases, units, dates, negation, uncertainty and multiple values. Decide which cases require abstention because the document does not support an answer. Settle these rules before scoring so evaluation does not reward a system for guessing what reference annotations left undefined.
Sample representative material
Select documents across source types, layouts, languages or jurisdictions relevant to intended use. Include ordinary cases, boundary cases, absent values and conflicting passages. Keep document versions fixed for a given evaluation. A convenient set of clean documents can estimate performance on clean documents, but it cannot establish performance on scans or ambiguous filings it never contains.
Review annotations
Have qualified annotators label claims and exact supporting passages independently where practical. Reconcile disagreements using written rules and retain original disagreements as evidence of task ambiguity. A reference label is an adjudicated judgment, not infallible fact. Mark unresolved cases and exclude or report them separately rather than quietly treating one annotation as certain.
Score by error type
Match outputs to reference claims using declared normalization rules, then report precision and recall with denominators. Break out wrong entity, value or date, unsupported span, missed claim and unjustified answer where abstention was correct. Inspect examples of each error. Report performance by source slice as well as overall, because one dominant document class can conceal a local failure.
Checks
- Annotation rules and source versions are fixed.
- Reference spans support labelled claims.
- Precision, recall and denominators are reported by relevant slice.
- Ambiguous labels and abstentions are explicit.
Pitfalls
- Scoring only normalized value strings.
- Treating disputed annotations as certain truth.
- Reporting one aggregate score for unlike documents.
A worked example
A fictional team builds an extractor for supplier certificates. The target claim is: this supplier holds this certificate, issued by this body, valid between these dates, as shown in this passage. Annotation rules say how to treat renewals, expired certificates and certificates for a different site.
Two annotators label a sample drawn from the document types the system will meet, including scans and documents in two languages. They disagree on one document in eight about whether a certificate applies to the supplier or only to a named site, and the disagreement is recorded. A reconciliation meeting turns the pattern into a written rule. The system is then scored on claims, not strings: right certificate on the right supplier with the right dates and a supporting passage counts as correct; a right date on the wrong site does not. The report shows the results by language and scan quality, and separately for documents that contain no certificate, where the right answer is none.
Variations and edge cases
Normalization rules affect scores. If "1 March 2026" and "2026-03-01" must count as equal, say so before scoring. If a missing end date should count as open-ended in one field and as unknown in another, say that too. Partial credit can be informative, but report the strict score as well, since a reader deciding whether to rely on a field cares about the claim being fully right.
Gold sets age. Documents change layout, new suppliers appear, and the distribution moves. Keep the set versioned, add fresh samples periodically, and keep a frozen core for comparison across system versions. Beware of tuning the system against the set until it is no longer a test; hold out part that is only used for final reports.
Agreement and the limits of reference labels
Annotator agreement is part of the result. Statistics for agreement beyond chance, discussed at length in computational linguistics, tell you how well defined the task is. Low agreement means that the label is partly a matter of interpretation, and a system's score against such labels has a ceiling. Report the agreement alongside the scores, and treat cases where trained annotators disagree as a separate slice. Precision and recall are defined over the retrieved and relevant sets and should be reported with their counts, not only as percentages, so that readers can see how small a slice is.
How the method is checked
Ask whether a system with obvious flaws would do badly. Include a trivial baseline, such as always abstaining, and a deliberately sloppy one, such as taking the first matching string, and see that the scores order them sensibly. Inspect a random sample of correct and incorrect outputs by hand. If the manual impression differs from the score, the scoring rules are probably wrong.
Related terms
References
- Artstein and Poesio (2008). Inter-Coder Agreement for Computational Linguistics. Computational Linguistics 34(4).
- Manning, Raghavan and Schutze. Introduction to Information Retrieval: evaluation of unranked retrieval sets.
- Xu et al. (2023). Large Language Models for Generative Information Extraction: A Survey. arXiv:2312.17617.
Found an error on this page? See corrections for what counts as one, how to report it and the dated log.
Enquiry
Test this method on one entity.
Name an entity where "Evaluating extraction against a gold set" would be checked in practice, with its sources and its readers. We reply with what a first record would cover and what it would need from you.
Or write to [email protected] in your own words.