Skip to content
Reliable Context
Read the paperRequest access

Evidence

What we claim, how it was measured, and what would falsify it.

This site makes exactly two factual claims about itself. Both are below, stated verbatim, each with the method that produced it, an explicit note on what it does not claim, and the single observation that would make it false. Everything else on this site is a description of properties you can check on a record, not an assertion about traction, scale, accuracy or customers.

Inspect the demonstrationRead the position paper
A precision dial gauge photographed straight on, its face entirely blank with no numbers or graduations, carrying a single thin cobalt blue needle.

The face is blank on purpose. An instrument with a scale printed on it tells you a number before you have agreed what is being measured; the two claims below carry their scale in words, so that somebody who wanted to dispute them would know exactly what to dispute.

The two claims

Both of them, with their counting rules and their failure conditions.

A claim without a method is a slogan, and a claim nothing could falsify is not a claim about the world. Each one below carries four blocks: the sentence, how it is counted, what it does not say, and what a sceptic should go and look for. Auditing practice draws the same line: evidence is information that supports an assertion and information that contradicts it [1].

Claim 01

In production use in two sectors.

method

production
Running inside a customer's own infrastructure, against their live private and public sources, under their own policies for sources, model routing, reader clearance and retention.
instance
One instance counts when values it produced are being acted on by people who do not work here. A deployment that only we read does not count.
sector
Counted by the customer's industry, not by the number of teams or sites inside it. Two customers in one industry would be one sector, not two.
excluded
A pilot is not production. Neither is an evaluation against a frozen corpus, a proof of concept, a trial on historical data, or anything running on our own machines.
naming
The two sectors are not named, because the customers are not named and in both cases the sector would identify them.

what this does not claim

  • It does not claim a number of customers, seats, users or contract value.
  • It does not claim that either deployment covers the whole of that customer's work.
  • It does not claim anything about volume, throughput or coverage of a corpus.

what would falsify this

Establish that the two deployments sit in one industry, or that at either of them no value produced by the record is relied on by anybody outside this company. Either observation makes the sentence false as written. We will put that question to both customers on behalf of a prospective customer or a reviewer under an agreement, and we will report the answer here whichever way it comes back.

Claim 02

More than six thousand identity decisions made by rule, none since reversed.

method

identity decision
A recorded decision about whether two records describe one real thing. Each is stored with its basis, its evidence and its date, and can be read back individually.
by rule
Decided by a rule stated in advance and applied uniformly, rather than settled case by case by whoever was looking. The rule that decided each one is recorded with it.
reversed
A later decision contradicts an earlier one about the same pair. Because nothing is overwritten, a reversal is a new entry that supersedes the old, and is therefore countable rather than inferred.
count
Cumulative across deployments, from the first decision recorded to the date this page was last revised. Decisions taken during evaluation runs are included; decisions we discarded along with a whole test database are not.

what this does not claim

  • It does not claim six thousand is a large number. In this domain it is a working quantity, not a milestone.
  • It does not claim that none will be reversed. The reversal count is a property of a finite history, and we expect it to leave zero.
  • It does not claim an accuracy, a precision or a recall figure, on our data or on any public benchmark.
  • It does not claim that any of these decisions was independently audited. None has been.

what would falsify this

Find one pair. A single decision that a later decision contradicts makes the second half of the sentence false; a single pair the record holds as one real thing that a reader can show is two, or holds apart that can be shown to be one, makes the rule that produced it wrong even where no reversal has been filed. Every decision is readable back with the evidence it rested on, so this is a finite search over a stated set rather than an impression, and it is the search we would run first if we were trying to break the claim.

Neither claim is a performance measurement, and neither is a proxy for one. If you are trying to work out whether the record would hold up against your own material, the honest answer is that these two sentences will not tell you: bring one entity you already argue about and read what it carries.

How we publish

Failures are published with results.

The problems underneath this product are open research problems [2], and a result that only gets published when it is flattering is not a measurement. Fixing the protocol before the run is the practice called preregistration [3]. Four rules, none of them conditional on the outcome.

Research program
  1. 1

    The protocol is published before the result

    What is being measured, on what data, with what counting rule and what would count as a failure, dated and public before the run. A protocol written after the numbers are in is a description of the numbers.

  2. 2

    The result is published whichever way it goes

    A run that was executed is reported. There is no branch in which an inconvenient measurement quietly does not appear, and no second run substituted for a first one without both being shown.

  3. 3

    A measurement that contradicts us is published with the correction

    If a result narrows or withdraws something stated earlier, the earlier statement is corrected on the page that carries it and the correction is dated in the changelog below. The superseded wording is not edited out of the record of what we said.

  4. 4

    A negative result is a result

    A measurement that shows an approach does not work, or that a distinction we make does not change what a reader decides, is published with the same protocol and the same prominence as one that supports us. It is also the more useful of the two to anybody else working on the problem.

Protocol publisheddated, before the runRunmeasured onceeither directionResult publishedthe measurement supported what we had already saidResult published, with the correctionit did not; the earlier statement is narrowed or withdrawn hereno third brancha run that happened is not unpublished
The protocol is dated before the run; the result leaves in either direction. The dashed path is the one that does not exist.

We do not report a result on a benchmark we designed ourselves without saying that we designed it, and we do not compare against a system we configured on its behalf. Where a protocol was agreed with a research group, the group is named on the protocol and on the result, and the counting rule is theirs. Record linkage has reported match decisions together with explicit error rates since 1969 [4]; a measured error rate is the kind of claim this page will make once a registered protocol has produced one, and until then it claims none.

Changelog

What changed about the record’s behaviour.

Changes a reader of a record would notice, corrections to things we said on this site, and evaluations that made us change something. Newest first. Nothing here is edited after the fact; a later entry supersedes an earlier one.

Position paper
  1. vocabulary

    A not-found now names what was searched, not only the date

    A value declared not found previously showed the question it answered and the date the search was run. A reader could see that something had been looked for, but not where, and so could not tell a thin search from an exhaustive one.

    The declaration now names the sources that were consulted alongside the date. A reader who thinks the boundary of the search was drawn in the wrong place can now dispute the boundary rather than the conclusion, which is a more useful argument to be having.

    No value changed its status as a result of this. The same values are declared not found as before.

  2. surface

    A blocked value now names the policy that blocked it

    A value withheld under a source licence or a reader clearance previously displayed the source it came from and the word blocked. It now names the policy that withheld it and the clearance that would lift it.

    Readers were treating blocked as a synonym for unavailable. It is not: one is a restriction on this reader, the other is a statement about the world, and the record has to keep them apart on the surface as well as underneath.

    Nothing changed about which values are withheld from whom.

  3. correction

    An earlier phrasing on this site overstated something

    This site previously described identity decisions as verified. That word reads as an external audit, and no external audit has been performed.

    The claim now reads that the decisions were made by rule and that none has since been reversed, which is what the record supports and nothing more. The changelog entry stays here rather than being edited away.

  4. surface

    A competing value now shows the grounds for not carrying it

    Where a credible source disagreed with the value on the record, the reader previously saw that a conflict existed and which source held the other position. The competing sentence itself, its date and the stated grounds for not carrying it were one step further away.

    All four now sit with the value. A reader who disagrees with the grounds can say so without first asking us what they were, and the disagreement is about a written reason rather than about a marker.

    Which value is carried did not change. What changed is how much of the argument the reader is shown at the point of reading.

  5. evaluation

    Declared unknowns: not applicable was being used where not found was correct

    An internal evaluation of declared unknowns examined every negative on client-visible surfaces against the question it answered. It found a class of cases in which not applicable had been emitted because the property was unusual for that kind of record, when the property did apply and no source had been searched.

    The two are not interchangeable. Not applicable closes the question; not found leaves it open and owes a list of what was searched. Read as a negative, the first is considerably stronger than the evidence allowed.

    The affected cases were re-declared as not found with their search records, and the distinction between the two is now stated where each is defined rather than left to the reader to infer.

  6. correction

    The phrase every value was overbroad on one page

    This site said that every value exports with its chain of custody. A value withheld under a reader's clearance does not: it exports as a labelled absence naming the policy, with no excerpt and no locator, because the excerpt is the thing being withheld.

    The sentence now says so wherever it appears. The distinction matters to anybody preparing an export for a reviewer whose clearance differs from their own, which is the ordinary case rather than the unusual one.

What we have not measured

Five numbers we do not have, and why they do not exist yet.

These are the figures a sceptical reader would ask for next. We do not hold them. Each line says why, and none of the reasons is that the number would be unflattering.

The four guarantees
Accuracy on a public benchmark
No public benchmark contains the private half of the evidence a fused record is built from. A score on one would measure performance on the public half and would then be quoted as though it measured both. We have not built a substitute benchmark either, and if we did we would have designed our own examination.
Recall of declared unknowns
To score a declared unknown you need an independently established fact that the record failed to find. Where such a fact exists and we hold it, the record would not have declared the unknown in the first place. We do not have a set of cases where the ground truth is an absence somebody else established.
Time to build a record
It is governed almost entirely by how much of the evidence sits inside the customer's walls and in what condition it is kept. The observations we hold come from deployments we cannot describe without identifying them, and an average over two of anything is not a number.
Cost per value
The unit is unstable. A value settled by one retained document and a value contested across five readings occupy the same row and do not cost the same. We have not defined a unit we would be willing to be held to, and publishing an average over an undefined unit would be worse than publishing nothing.
An independent audit
Nobody outside this company has examined the identity decisions, the retained sources or the export path. We would like that to change and we have not commissioned it. Until somebody has, every statement on this page rests on our own reading of our own record.

A missing number is not a quiet claim that the number would be good. If any of these is measured, the protocol goes up before the run and the result goes into the changelog above, in whichever direction it lands.

References

References

The standards and papers this page's method leans on: the definition of audit evidence, open problems in entity resolution, preregistration and the classical statement of record linkage.

  1. [1]

    AS 1105: Audit Evidence

    Public Company Accounting Oversight Board

    PCAOB auditing standard. https://pcaobus.org/oversight/standards/auditing-standards/details/AS1105

  2. [2]

    Entity Resolution: Theory, Practice & Open Challenges

    Lise Getoor and Ashwin Machanavajjhala

    Proceedings of the VLDB Endowment 5(12), 2012. https://www.vldb.org/pvldb/vol5/p2018_lisegetoor_vldb2012.pdf

  3. [3]

    The Preregistration Revolution

    Brian A. Nosek, Charles R. Ebersole, Alexander C. DeHaven and David T. Mellor

    Proceedings of the National Academy of Sciences 115(11), 2018. https://doi.org/10.1073/pnas.1708274114

  4. [4]

    A Theory for Record Linkage

    Ivan P. Fellegi and Alan B. Sunter

    Journal of the American Statistical Association 64(328), 1969, pages 1183-1210. https://doi.org/10.1080/01621459.1969.10501049

Further reading

Articles on this page’s subject.

The research library

The rest of what we know is in the open.

Three pages carry the rest of it. None of them restates the two claims above; each one is a different thing you can hold against us.

/researchResearch programThe problems we think are unsolved, the evaluations we intend to run, and the literature we are reading against./paperPosition paperThe argument the product rests on, with its citations, and a BibTeX entry if you intend to disagree with it in print./guaranteesThe four guaranteesWhat a reader is owed on every value, stated as properties you can check rather than as promises about the system.