Gold standard

A gold standard is a set of reference judgments against which a system is evaluated. The word 'gold' describes its role in evaluation, not infallibility. Labels may come from experts, adjudicated annotation, authoritative records, or a combination. Their meaning depends on the task definition: a correct entity match, a supported sentence, and a valid classification each require a different judgment rule. The set should represent the cases on which conclusions will be drawn.

Building a useful reference set requires written instructions, source access, clear handling of unknown and disputed cases, and review of disagreements. Stratified samples can expose rare but consequential failure types that a random sample might miss. Separating development and test examples reduces the risk of tuning to the answer key. Versioning is necessary when new evidence changes a label; the earlier benchmark result should still be interpretable.

For a defensible organization, a gold standard makes evaluation repeatable and disagreement inspectable. It does not settle every contested real-world fact by decree. Precision, recall, and faithfulness scores depend on the reference set's scope and quality, while inter-annotator agreement reveals where judgments are unstable. A benchmark should identify its evidence date and policy assumptions before its score is compared with a different system or later run.

A gold standard is a snapshot. Facts change: a label that was correct when assigned, such as the current chief executive or the status of a study, may be wrong now. Sources change, so the evidence an annotator saw may no longer exist. Guidelines drift as the team's understanding improves, and old labels no longer match the current definition. Systems also learn the standard: if the same test set is used repeatedly to choose among systems, the chosen system is tuned to its quirks, and its measured performance overstates real performance. The article on RAG evaluation dataset leakage as sources change examines how evaluation sets become contaminated.

Practical remedies are to date every label and record the evidence it rested on, to keep a held-out set that is touched rarely, to refresh a sample on a schedule, and to test with behavioural checks in addition to aggregate scores, as in CheckList, which proposes testing particular capabilities with targeted cases. Where no gold standard can be built, comparisons can still be made with other designs, as in the article on comparing truth discovery methods without gold labels.

Ask for the guidelines, the number of annotators, the agreement figure and the adjudication process. Ask for the date of each label and the evidence it rested on. Ask how often the set has been used for model selection. Read twenty labels and judge whether you would assign the same label. Ask what the standard is not suitable for, and check that it is not being used for that.