Glossary
Precision and recall
Precision measures the share of returned positives that are correct; recall measures the share of actual positives returned.
Evaluation
Precision is the proportion of predicted positives that are true positives, while recall is the proportion of all actual positives that the system finds. Both require a defined positive class and a reference set of judgments. They respond differently to a decision threshold: a stricter threshold can reduce false positives while missing more true cases. Neither metric alone describes the cost of errors, calibration of scores, or performance on unlabelled cases.
For evaluation, define the unit being counted before calculating either ratio. A duplicate source mention, an entity match, and a unique claim are different units. Report the population, sampling method, ambiguity policy, and treatment of abstentions. Where labels are uncertain, adjudication and inter-annotator agreement matter because the apparent metric may reflect disagreement in the reference set rather than behavior of the system under test.
An organization defending values must know whether its process tends to accept unsupported claims or miss supported ones. Precision and recall expose parts of that tradeoff, but they do not say whether an individual value is justified. Error analysis should inspect evidence for consequential false positives and false negatives. A gold standard gives the comparison basis; calibration addresses whether numerical confidence corresponds to observed correctness across cases.
Example
In a review of Halden Therapeutics AG (fictional) records, eight of ten flagged matches are correct, while two correct matches were missed; the denominators define the metrics.
Definitions and the trade-off
Precision is the proportion of returned items that are correct. Recall is the proportion of correct items that were returned. The textbook by Manning, Raghavan and Schutze defines both for retrieval and explains why accuracy is a poor measure when relevant items are rare, since a system that returns nothing is nearly always right about the many non-relevant items. The F-measure combines the two into one number, and the documentation of scikit-learn shows the standard definitions and their averaging variants. The Wikipedia article gives the same definitions with worked examples.
The two trade off against each other. A system that returns more items raises recall and usually lowers precision. The right balance depends on the cost of each kind of error. In screening sources for completeness, a missed item may be costly, which favours recall. In publishing facts to a customer, a false item may be costly, which favours precision. State the choice with its reason, and report both numbers at the operating point, not only a combined score.
Choosing the unit and the denominator
Every precision and recall figure depends on what is being counted. For extraction, the unit may be a mention, an entity, a relation or a claim, and the figures differ. For retrieval, the denominator for recall is the set of relevant items, which is usually unknown for a large corpus; test collections estimate it by pooling the results of several systems, and the estimate is optimistic for novel systems. For generated text, FActScore takes the unit to be the atomic fact and measures the share of facts supported by a knowledge source, a precision-like measure of factual accuracy in long-form generation. A statement of precision without its unit and its labelling procedure cannot be compared with another.
Report uncertainty. A precision of 0.9 from forty labelled items has a wide interval. Report the count and an interval, and stratify by category, because an average can hide a category with poor results. Where the labels come from people, report inter-annotator agreement, since a ceiling on agreement is a ceiling on measurable accuracy. The article on entity resolution evaluation beyond pairwise F1 shows how a single figure can mislead.
A worked example
A system extracts statements about sponsors from 200 documents about Halden Therapeutics AG (fictional) and returns 120 statements. A reviewer finds 96 correct and 24 incorrect. Precision is 96 of 120, which is 0.80. The reviewers separately identify 150 true sponsor statements in the documents, so recall is 96 of 150, which is 0.64. The F1 score, the harmonic mean of the two, is about 0.71. Raising the confidence threshold removes 30 returned statements, 20 of them incorrect and 10 correct. The system now returns 90 statements, 86 of them correct: precision rises to about 0.96, recall falls to 86 of 150, about 0.57, and F1 is about 0.72. Whether the second operating point is better depends on which error costs more, and the F1 scores barely distinguish the two, which is why the pair of numbers should be reported and not the combined figure alone.
Checks a reviewer can run
Ask for the unit counted, the labelling procedure and the sample size behind each figure. Ask for precision and recall separately at the operating point in use. Ask for the breakdown by category. Recompute one figure from the raw counts. Ask how the recall denominator was estimated and whether it could be missing true items the system cannot find.
Related terms and boundaries
Precision and recall are computed against a gold standard whose quality limits them, are reported with inter-annotator agreement, and are complemented by calibration when a system outputs confidences. Abstention is a way to trade recall for precision by declining the uncertain cases.
Common misreadings
- Reporting either metric without its evaluated population.
- Counting duplicate mentions as independent cases.
- Treating a high aggregate score as proof for one disputed value.
Appears in
- Deterministic vs probabilistic entity resolution
Deterministic rules make explicit match decisions; probabilistic methods estimate match likelihood from observed features.
- LLM extraction vs rule-based extraction
Language models interpret varied prose; explicit rules suit stable, mechanically defined fields and formats.
- Evaluating extraction against a gold set
Compare extracted claims with reviewed annotations, including spans, scope and abstentions.
- Measuring declared absence
Score supported unknowns as distinct outcomes, with search scope and reason assessed separately.
References
- Manning, Raghavan, and Schütze, Introduction to Information Retrieval (2008), chapter 8.
- Manning, Raghavan and Schutze, Introduction to Information Retrieval, Evaluation of unranked retrieval sets.
- scikit-learn documentation, Metrics and scoring: precision, recall and F-measures.
- Wikipedia, Precision and recall.
- Min et al. (2023), FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation, EMNLP.
Found an error on this page? See corrections for what counts as one, how to report it and the dated log.
Enquiry
Where does "Precision and recall" bear on a decision of yours?
This entry sits under evaluation. If a decision in your organization turns on it, send the entity concerned, the sources you fuse to describe it and who may read which of them. We reply with what a first record would cover.
Or write to [email protected] in your own words.