Glossary
Blocking
Blocking limits record-linkage comparisons to candidate pairs that share selected features.
Identity and entity resolution
Blocking is a candidate-generation technique for record linkage. Instead of comparing every record with every other record, it groups or retrieves records using selected keys or similarities. The aim is to reduce comparisons while keeping the true matches in the candidate set. Blocking does not decide identity; it determines which pairs can be considered by later comparison and review.
A practical blocking rule might use normalised name fragments, geography or identifiers, often with several complementary passes. An exact name key can be fast yet miss transliterations or spelling errors. Analysts therefore measure candidate recall on known matches, including difficult cases, and inspect where records fail to enter the same block. A downstream matcher cannot recover a pair that was never generated.
For defensible identity decisions, candidate-generation limits must be documented and tested. Otherwise a missed match can be mistaken for evidence that the entities differ. Blocking relates to record linkage as search relates to adjudication: it controls the comparisons available, while the linkage decision weighs the evidence within each candidate pair.
Example
A Halden Therapeutics AG (fictional) record uses a translated name; a geography-only block finds a candidate that an exact-name block misses.
Why blocking is needed
Comparing every record with every other costs time proportional to the square of the number of records. A file of one million records has about five hundred billion pairs, which is impractical, and most of those pairs are obvious non-matches. Blocking selects a much smaller set of candidate pairs, those worth scoring, by grouping records that share some feature and comparing only within groups. The surveys by Christophides and colleagues and by Binette and Steorts both treat blocking as a central scalability technique of entity resolution and describe the main families.
The simplest family uses a blocking key built from one or more fields, such as the first letters of a surname together with a year of birth, so that only records sharing the key are compared. Sorted-neighbourhood methods sort records on a key and compare those within a sliding window, which tolerates small differences in the key. Canopy and token-based methods group records that share a token or exceed a cheap similarity threshold. Learned and embedding-based methods retrieve nearest neighbours in a vector space, related to vector search. The right choice depends on the data, and several methods are often combined.
Measuring a blocking scheme
Two measures are standard. The reduction ratio is the proportion of all possible pairs that blocking removes. Pairs completeness is the proportion of true matches that survive blocking, and it is the one that matters for correctness, because a true pair that blocking removes can never be recovered by later steps. A scheme with a high reduction ratio and a low pairs completeness is fast and wrong. Measure completeness on a sample where the true links are known, and break it down by the kind of record, since blocking on a name field fails systematically for records with transliterated or abbreviated names.
The usual remedy for lost matches is multi-pass blocking: several independent keys, each producing candidate pairs, with the union scored. A pair missed by a name-based key may be caught by an identifier-based or address-based key. The cost is a larger candidate set, which is why the passes should be chosen to be complementary and not overlapping. The effect of blocking on final quality should be reported alongside matching quality, as suggested in the article on evaluating entity resolution beyond pairwise F1.
A worked example
A file holds 200,000 company records. Blocking on the first four characters of a normalised name yields groups; the largest, for names beginning with a very common token, contains 3,000 records, giving over four million pairs in that block alone. The design therefore splits oversized blocks by adding the country, and adds a second pass on the registry number where present and a third on the web domain. A review of 500 known matches shows that 96 percent are caught by at least one pass, with the missed ones mostly records whose names were translated. A fourth pass using a token set of the name catches most of those, at a modest increase in pairs. The numbers are illustrative; the process, tuning the passes against measured completeness, is the point.
Related terms and boundaries
Blocking precedes pairwise scoring in record linkage, where the Fellegi–Sunter model typically does the scoring, and is the first stage of entity resolution at scale. It is not itself a matching decision, and records placed in the same block are not thereby asserted to match. Because it limits which comparisons are possible, it is also one place where entity disambiguation can fail silently, when the right referent is never retrieved as a candidate.
Checks a reviewer can run
Ask for the blocking keys, the reduction ratio and the pairs completeness, with the sample on which completeness was measured. Look at the largest blocks. Break completeness down by record type. Ask whether a second pass exists for records the first would miss, and what it costs.
Common misreadings
- Treating absence from a block as proof of nonidentity.
- Evaluating only easy, same-spelling matches.
- Relying on one brittle key.
Appears in
- Vector search vs entity lookup
Vector search finds semantically similar material; entity lookup tests which specific subject a record denotes.
References
- Christen (2012), Data Matching: Concepts and Techniques for Record Linkage, Entity Resolution, and Duplicate Detection, Springer.
- Christophides et al. (2020), End-to-End Entity Resolution for Big Data: A Survey, ACM Computing Surveys.
- Binette and Steorts (2022), (Almost) All of Entity Resolution, Science Advances 8(12).
- Splink documentation, The Fellegi-Sunter model.
Found an error on this page? See corrections for what counts as one, how to report it and the dated log.
Enquiry
Where does "Blocking" bear on a decision of yours?
This entry sits under identity and entity resolution. If a decision in your organization turns on it, send the entity concerned, the sources you fuse to describe it and who may read which of them. We reply with what a first record would cover.
Or write to [email protected] in your own words.