Methods
Measuring declared absence
Score supported unknowns as distinct outcomes, with search scope and reason assessed separately.
Problem
An evaluation that rewards only filled values can punish a correct refusal and reward an invented answer. Yet "unknown" is not one outcome: a bounded search found nothing, access was blocked, or the property did not apply. The reference set must distinguish these states from a missing annotation and from a failure to look. A correct declared absence requires the right status and adequate grounds, including what was searched and when; otherwise abstaining everywhere could score well.
Steps
Define absence labels
Specify supported value, not found within a defined search, blocked by licence, not applicable, and unresolved where evidence conflicts or is inadequate. State the evidence required for each label. Keep a missing annotation separate from a reference judgment that no value should be supplied. The classes need distinct scoring because an access barrier cannot be resolved by more confident extraction.
Build reviewed cases
Include positive and negative cases with search boundaries, source versions and review dates. For "not found", document eligible collection and query; for "blocked", document the access limitation; for "not applicable", document the rule excluding the property. Ask reviewers to verify these grounds rather than inferring a label from an empty cell.
Score status and grounds
Compare predicted status with reviewed status, then separately assess whether supplied scope and evidence justify it. Report a confusion matrix so a wrong value, unnecessary abstention and mistaken kind of unknown remain distinct. Measure coverage alongside correctness: a system that always says "not found" should show low useful coverage and many status errors.
Test change over time
Revisit a sample after new sources or access rights arrive. A previously correct "not found" can become stale without having been wrong at the earlier search date. Score the earlier judgment against evidence available then and the current judgment against evidence available now. This prevents later discoveries from being projected into a historical evaluation.
Checks
- Unknown states are separate from missing annotations.
- Each reference absence has a dated basis.
- False answers and needless abstentions remain distinct.
- Coverage and correctness are both reported.
Pitfalls
- Giving full credit to any blank response.
- Treating blocked access as not found.
- Judging an earlier search against later publications.
A worked example
A fictional evaluation asks a system whether twenty companies hold a particular licence. For each company the reference set states one of four outcomes: licence found with a passage, not found in a named register on a named date, blocked because the register requires a paid account, or not applicable because the company does not do the licensed activity.
The system returns a value or a status for each company. Scoring has two parts. First, did it return the right outcome? Second, for statuses, did it give adequate grounds, such as the register searched and the date? A system that says not found for everything gets the not-found cases right and fails the others, and its useful coverage is low. A system that invents licences gets credit for none of them. The confusion matrix shows which statuses are confused with which, and a system that confuses blocked with not found has a different problem from one that confuses not found with a value.
Variations and edge cases
Time cuts both ways. A not found that was right when the register held only older entries may be wrong now. Score earlier judgements against the evidence available then. Keep the evaluation date and the register version in the reference. Where registers change daily, freeze a copy for the evaluation and cite it.
Calibration is related. A system that says it is sure and is wrong is worse than one that says it is unsure. Research on whether language models know what they know, and on the calibration of neural network confidence, offers methods to check whether stated confidence tracks accuracy. Abstention surveys discuss how to evaluate refusals, including the trade-off between answering too much and refusing too much. Report both.
Building the reference set
Reference absences are harder to build than reference values. A value needs one passage; an absence needs a defined search and someone to run it. Budget for this, sample carefully, and mark absences whose reference search may itself be incomplete. Include cases designed to tempt a guess, such as a company with a similar name that does hold the licence. Use the same status definitions as the production system, so that results transfer.
How the method is checked
Check that the score cannot be gamed by a constant answer. Run the all-unknown and all-value baselines and see that both do poorly. Check that reviewers can agree on the statuses for a sample of cases, and that disagreements lead to clearer definitions. Check that the per-status results are published, not just a total.
See also
- Declaring unknowns
- Evaluating extraction against a gold set
- Blank cell vs declared unknown
- Abstention
- Calibration
- Null semantics
Further reading in the articles
References
- Rajpurkar, Jia and Liang (2018). Know What You Don't Know: Unanswerable Questions for SQuAD. ACL.
- Kadavath et al. (2022). Language Models (Mostly) Know What They Know. arXiv:2207.05221.
- Guo et al. (2017). On Calibration of Modern Neural Networks. arXiv:1706.04599.
- Wen et al. (2024). Know Your Limits: A Survey of Abstention in Large Language Models. arXiv:2407.18418.
Found an error on this page? See corrections for what counts as one, how to report it and the dated log.
Enquiry
Test this method on one entity.
Name an entity where "Measuring declared absence" would be checked in practice, with its sources and its readers. We reply with what a first record would cover and what it would need from you.
Or write to [email protected] in your own words.