Skip to content
Reliable Context
Read the paperRequest access

Research

We publish measurements, including failures.

The four properties this product guarantees rest on problems that are not solved in the literature and are not solved here. We would rather be corrected in public than be quietly wrong in a customer’s file, so the protocol goes out before the result and the result goes out whichever way it falls.

Read the position paper[email protected]
Fields
Provenance, truth discovery and knowledge fusion, record linkage, attribution and evaluation of generated text
Position paper
Version 0.1, 21 September 2026, draft for comment
Open problems
Eight, each stated with what a result would have to look like
Data access
Evaluation sets with their evidence, under a written agreement
Three clear glass prisms and a single lens arranged on a pale grey surface, each splitting the same light differently.
The same light, four instruments, four readings. The interesting question is never which reading is prettiest; it is what each instrument was pointed at, when, and what it would take to point it there again.

Open problems

Eight problems we have not solved.

These are stated as problems because they are problems. The product takes a position on each of them; none of those positions is a proof, and several are deliberately conservative because the measurement that would license something better does not exist yet. Each one carries the result that would change our mind.

The argument in full
  1. 01

    Deciding that two records describe one real thing, from evidence alone

    A threshold on string or vector similarity is a tuned guess wearing the clothes of a rule. We do not know how to state, in general, what evidence is sufficient for sameness across kinds of entity, nor how to audit a decision made that way against one a person would defend. Closest prior work: [7, 8].

    moved byA statement of sufficient evidence that a second team can apply to the same dossiers and reach the same decisions, with the disagreements legible rather than averaged away.

  2. 02

    Measuring whether a declared unknown was correctly declared

    When a system says not found, the ground truth is also an absence, and an absence is not observable. Annotating it means establishing that nothing exists to be read, which is not a labelling task. There is no accepted protocol and no public set whose negatives were themselves checked. Closest prior work: [11].

    moved byAn annotation procedure for absences that two independent groups can run to a measurable agreement, on a corpus where the negatives were verified rather than assumed.

  3. 03

    Detecting the moment a retained source stops supporting its value

    A page is edited, a filing is superseded, a study is withdrawn. The value stands, the excerpt still matches the retained copy, and nothing fails. Which changes to a source invalidate a reading and which are cosmetic is a judgement nobody has reduced to a check that runs.

    moved byA labelled set of real source revisions, substantive against cosmetic, and a detector that beats re-reading everything on cost at equal recall of the substantive ones.

  4. 04

    Comparing disagreeing sources without a standing ladder of authority

    Ranking registry above filing above press release encodes one analyst's priors as a permanent rule, and is wrong the first time the registry is stale. Deciding per case, from the evidence in front of you, without collapsing into per-case arbitrariness, is unsolved in both directions. Closest prior work: [6, 4].

    moved byA per-case adjudication that beats a fixed hierarchy on a held-out set of real disagreements, with reasons that stay stable when the same cases are re-run.

  5. 05

    Evaluating a fused record end to end, when half the evidence is private

    No public benchmark contains the internal half of the evidence, and the organizations holding it cannot release it. Evaluations therefore run either on public fragments, which are unrepresentative of the task, or inside a perimeter, where the result cannot be reproduced by anybody else.

    moved byAn evaluation design in which the private half never leaves the perimeter and a third party can still reproduce the protocol, the metric and the grading decisions.

  6. 06

    Deciding when corporate entities should be related rather than merged

    Two records sharing a name, a director and a registered agent may be one company or two in different jurisdictions. Merging sums an exposure across two counterparties; separating hides the connection. Where the line between an equivalence and a typed relationship falls is not settled for any real registry.

    moved byA corporate-family set with adjudicated decisions in which relate and merge are labelled separately, and the cost of each error is stated rather than assumed symmetric.

  7. 07

    Knowing when an excerpt has been cut away from its meaning

    A quotation can be verbatim, correctly located and correctly dated, and still assert something its document does not, because the sentence before it scoped the claim or a later paragraph withdrew it. Verifying that an excerpt is faithful to its context, not merely present in it, has no reliable test. Closest prior work: [10, 12].

    moved byA test that catches excerpts whose surrounding document scopes or withdraws the claim, at a precision a reviewer would accept without re-reading the document themselves.

  8. 08

    Deciding when a value is old enough to need reading again

    Values age at rates that depend on the property, the source and the entity. Re-reading everything is unaffordable; re-reading on a fixed interval is arbitrary. What signal should trigger re-verification, and what an expiry means for a value already acted on, are both open.

    moved byA signal that predicts which values have actually changed since they were read, measured against a full re-read, and beating the fixed interval it would replace on cost.

a public benchmark can hold this halfno public benchmark holds this halfPublic halfCommercial registry filingpublic, citable, revised without noticeRegulator decisionpublic, dated, sometimes withdrawnPeer-reviewed publicationpublic, stable, rarely decisive alonePrivate halfinside the customer perimeterCounterparty contractauthoritative, not releasableInternal assessmentnames people, cannot be publishedLicensed data feedredistribution forbidden by licenceOne fused valuethe thing somebody signs and acts onAn evaluation that is reproducible covers the wrong half.An evaluation that covers the right half cannot be reproduced by anybody else.
Problem 05, drawn. The evidence behind a fused value divides into a half a benchmark can hold and a half no benchmark will ever hold, and the second half is usually the one the decision turns on. Every evaluation design we have tried buys reproducibility by giving up representativeness, or the reverse.

Reading the literature

What we read, and what we took from it.

Twelve entries across four bodies of work, each with the judgement we drew from it. This is not a survey; it is the shortest list that explains why the product is shaped the way it is, and it is where a disagreement with us should start.

Full reference list

Provenance models

What it means for a value to carry where it came from, and the vocabulary the web settled on for saying it.

  • 2013W3C Rec.

    [1] PROV-DM: The PROV Data Model

    Luc Moreau and Paolo Missier, editors. W3C Recommendation, 30 April 2013

    we tookWe took its insistence that a value is explained by the activity that produced it, not only by the document it names; it is why our records describe a reading, not a citation.

  • 2013W3C Rec.

    [2] PROV-O: The PROV Ontology

    Timothy Lebo, Satya Sahoo and Deborah McGuinness, editors. W3C Recommendation, 30 April 2013

    we tookProof that provenance is interchangeable between systems built apart, and a reminder that agreeing on a serialisation is not the same as agreeing on what counts as a value.

  • 2001ICDT

    [3] Why and Where: A Characterization of Data Provenance

    Peter Buneman, Sanjeev Khanna and Wang-Chiew Tan

    we tookThe sharpest distinction in the literature, and one most products blur: the source that explains a value and the source that contains it are two different claims.

Truth discovery and knowledge fusion

Estimating what is true when sources disagree, and what happens when source reliability is inferred rather than declared.

  • 2008IEEE TKDE 20(6)

    [4] Truth Discovery with Multiple Conflicting Information Providers on the Web

    Xiaoxin Yin, Jiawei Han and Philip S. Yu

    we tookShowed reliability can be inferred from agreement between sources; convinced us that an inferred reliability is an estimate and never a defence you can hand to a regulator.

  • 2014KDD

    [5] Knowledge Vault: A Web-Scale Approach to Probabilistic Knowledge Fusion

    Xin Luna Dong, Evgeniy Gabrilovich, Geremy Heitz, Wilko Horn, Ni Lao, Kevin Murphy, Thomas Strohmann, Shaohua Sun and Wei Zhang

    we tookThe reference point for fusion at scale, and the reason we treat a confidence score as a triage signal on our side of the line rather than something a reader should ever see alone.

  • 2016SIGKDD Expl. 17(2)

    [6] A Survey on Truth Discovery

    Yaliang Li, Jing Gao, Chuishi Meng, Qi Li, Lu Su, Bo Zhao, Wei Fan and Jiawei Han

    we tookThe map of the field, and the clearest demonstration that nearly every method assumes conflicting claims about one already-identified entity, which is the step our problem has not reached.

Record linkage and entity resolution

Fifty-five years of deciding whether two records describe one thing, and what the field still calls open.

Attribution and evaluation of generated text

Whether a generated statement is supported by the source it names, and how that has been measured so far.

  • 2021 / 2023Comput. Ling. 49(4)

    [10] Measuring Attribution in Natural Language Generation Models

    Hannah Rashkin, Vitaly Nikolaev, Matthew Lamm, Lora Aroyo, Michael Collins, Dipanjan Das, Slav Petrov, Gaurav Singh Tomar, Iulia Turc and David Reitter

    we tookGave attribution a definition that can be argued with rather than asserted, and it is the definition every claim we make about citation has to be measured against.

  • 2018NAACL-HLT

    [11] FEVER: a Large-scale Dataset for Fact Extraction and VERification

    James Thorne, Andreas Vlachos, Christos Christodoulopoulos and Arpit Mittal

    we tookIts not-enough-information class is the nearest public analogue of a declared negative, and it shows exactly why an absence relative to one corpus is not a statement about the world.

  • 2023Findings of EMNLP

    [12] Evaluating Verifiability in Generative Search Engines

    Nelson F. Liu, Tianyi Zhang and Percy Liang

    we tookMeasured deployed systems and found sentences their own citations do not support; it is the empirical reason we bind an excerpt at the moment a value is written, not afterwards.

Each title links to the published record or the most stable address we could confirm. The position paper carries the longer list, twenty-five works including the semiring formulation of provenance, the records-management standard and the retrieval-augmented generation paper this site argues with. Every entry has been checked against the published record; if one of them is wrong, tell us and we will correct it here with the date of the correction.

Work with us

Three ways in, and one list of refusals.

For groups working on provenance, fusion, record linkage or attribution. All of it is open to academic and industrial researchers on the same terms, and the terms are written down because they are the part people are right to ask about first.

[email protected]

01 / data access

Data access under agreement

Evaluation sets released with their evidence attached: the value, the excerpt it was read from, the locator, the observation date and the retained document, plus the declared negatives and the record of what was searched to declare them. The negatives are the part that does not exist elsewhere, and the part we most want somebody to attack.

Access is under a written agreement covering redistribution and attribution. It is not a licence to our software and it carries no obligation to show us a draft; we ask only that the version of the set be cited, because the sets are versioned for the same reason the paper is.

Sets are built from material we are free to release, or from fictional entities constructed for the purpose and labelled as such. No customer data leaves a customer perimeter.

02 / co-authorship

Co-authorship on evaluations

We register the protocol before the run: the question, the entities, the sources, the annotation procedure, the metric and what counts as a failure. Fixing the failure condition in advance is the whole discipline, because an evaluation whose success criterion is chosen after the numbers arrive is a description of the numbers.

The result is published whichever way it goes, and a negative result is a result. Where an evaluation has to run inside a customer perimeter, the protocol, the grading rules and the aggregate result are published even though the underlying evidence cannot be, and we say so plainly rather than implying the evidence was seen by anyone outside.

Problems 02, 05 and 07 are where an outside annotation protocol would move us furthest, and where we most expect to be shown wrong.

03 / citation

How to cite

The position paper is versioned, and its version number changes when the argument changes, not when the prose does. Cite the version you read; if we change our position, the old version stays readable with the date it was superseded, which is the same contract the product offers a value.

How to cite

Title
A kernel of truth: records that carry their evidence in organizations that fuse private and public information
Author
Reliable Context Research (corporate author)
Version
0.1, draft for comment
Date
21 September 2026
@techreport{reliablecontext2026kernel,
  author       = {{Reliable Context Research}},
  title        = {A kernel of truth: records that carry their evidence
                  in organizations that fuse private and
                  public information},
  institution  = {Reliable Context},
  year         = {2026},
  month        = sep,
  note         = {Version 0.1, 21 September 2026},
  url          = {https://reliablecontext.com/paper}
}

Changelog

  • 14 October 2026Version 0.1. Added this citation block, the permanent URL and BibTeX. No change to the argument; the version is unchanged.
  • 21 September 2026Version 0.1. First public version, a draft for comment.

To cite a particular claim rather than the paper as a whole, cite the section: the six sections keep their numbers and their subjects across versions even when the prose under them changes, and a section that is withdrawn stays in place with the reason.

For a specific measurement rather than the position, write to us and we will point you at the run, its protocol and the date it was executed.

04 / what we will not do

Three things we refuse

Customer data
We do not publish it, in whole, in part, or in aggregate, and we do not use it to build a public evaluation set. Released sets are built from material we are free to release or from fictional entities constructed for the purpose and labelled as fictional.
Veto rights
We do not co-author a paper whose result we are allowed to veto. If the protocol we agreed produces a result we dislike, it is published with our name on it. A collaborator who needs approval rights over the conclusion should not work with us on that question.
Borrowed numbers
We do not cite a benchmark result we have not run ourselves, and we do not quote a speed-up or an accuracy figure obtained on one entity as though it described a population. A measurement we cannot reproduce is not evidence we are entitled to use.

These are refusals rather than preferences. If one of them makes a collaboration impossible, it was going to produce a result neither of us could defend.

Further reading

Articles on this page’s subject.

The research library

position paper

A kernel of truth: records that carry their evidence

Six sections on fusion and provenance, identity, declared negatives, replayability, the limits of retrieval-centred context, and what a system of record owes its reader. Version 0.1, with twenty-five references.

/paper

evidence

How the two claims on this site are supported

The method behind the production claim and the identity-decision count, what each of them does and does not assert, and the changelog of corrections to this site.

/evidence

work with us

Engagements, and what a first one looks like

The commercial side of the same argument: what we take on, what we need from you, and what you hold at the end of it. Research collaborations run separately, on the terms above.

/work-with-us