Glossary
Data lineage
Data lineage traces data from source inputs through transformations to downstream outputs.
Provenance and citation
Data lineage is the path by which data moves and changes from origin to use. It may describe dataset-level dependencies or the derivation of a particular output value. The level matters: knowing that a dashboard uses a source table does not show which row or transformation produced one disputed number. Lineage can include joins, filters, conversions and manual interventions.
In practice, a lineage record maps input versions to output versions and explains transformations in enough detail to reproduce or audit a result. A field renamed from one system to another is a simple link; a calculated value needs its inputs and formula. Where human review affects an output, that intervention belongs in the path rather than being hidden as an unexplained overwrite.
For an organization that must defend a value, lineage shows which upstream changes could affect it and why two reports differ. Provenance is the broader concept of origins and activities; data lineage emphasizes the path through data processing. An audit trail records actions over time, while lineage shows derivational dependence among data objects and values.
Example
A Halden Therapeutics AG (fictional) dashboard count traces to a dated source extract and a documented filter that excludes withdrawn trials.
Levels of lineage and how it is captured
Lineage can be recorded at several levels. Dataset-level lineage says which datasets feed which. Column-level lineage says which input columns a given output column depends on. Row-level and value-level lineage say which input rows or values produced a given output, and is the form that supports individual disputes. The research literature on database provenance, including the paper by Buneman, Khanna and Tan on why- and where-provenance cited in this entry's references, distinguishes the inputs that justify an output from the locations a value was copied from. Practical tools capture lineage by instrumenting jobs and queries; the open standard OpenLineage defines a common event format for jobs, runs and datasets so that tools from different vendors can exchange lineage.
The W3C PROV data model offers a more general vocabulary for derivation, and a design may use OpenLineage events for operational capture and PROV for exchange and long-term storage. The important point is that lineage is captured as the pipeline runs. Lineage reconstructed afterwards from code and a guess about what ran is a hypothesis.
Using lineage for impact and for defence
Lineage answers two directions of question. Upstream: where did this number come from, and what could make it wrong? Downstream: if this input was wrong, which outputs, reports and decisions used it? The downstream question is the one that matters during a correction, because it identifies what to recompute and whom to notify. For a correction to be complete, lineage must reach the consumers, including exports, extracts and documents, and not only the tables in one warehouse.
Lineage is also a basis for point-in-time reproducibility, when it is combined with versions of the code and the data. A lineage record that says which job produced a value but not which version of the job and which snapshot of the inputs cannot reproduce it. Record the versions, and keep the inputs or a way to restore them. The article on minimum provenance requirements sets out a floor for what to keep.
A worked example
A report shows the average enrolment duration of studies run by Halden Therapeutics AG (fictional). Its lineage records three upstream datasets: a registry extract, a sponsor list and a mapping table that links sponsor names to organisation identifiers. A data-quality check finds that the mapping table had a wrong entry for one sponsor from February to April. Downstream lineage identifies the one report table and two exported charts that used the table in that period. The team recomputes those three outputs, records a correction note with the dates and reasons, and notifies the owners of the charts. Without lineage, the same finding would have required a search through scripts to guess which outputs were affected, and some would have been missed.
Related terms and boundaries
Lineage is the pipeline-facing part of provenance. It underpins an audit trail of changes to data and the point-in-time reproducibility of past outputs, and it is needed to count evidence correctly in the face of source dependence. Column-level lineage alone does not say whether a value is correct; it says only where the value came from.
Common misreadings
- Mistaking table-level lineage for value-level proof.
- Omitting manual corrections.
- Failing to identify input versions.
Appears in
- Data catalog vs system of record
A catalog tells users what data exists and how to find it; a system of record governs accepted values within a defined scope.
- Data warehouse vs system of record
A warehouse organizes data for analysis; a system of record governs the accepted state used for a defined decision.
- Golden record vs evidence-carrying record
A golden record presents preferred values; an evidence-carrying record keeps the dated source sentence with each value.
- Detecting source drift
Compare current source content with the retained cited version and reassess changed support.
- Central Index Key (CIK)
The SEC's identifier for a filer in its EDGAR submission system.
- CUSIP
A nine-character identifier used for securities in North American markets.
- International Securities Identification Number (ISIN)
A twelve-character code for identifying a financial instrument under ISO 6166.
- Market Identifier Code (MIC)
An ISO code for a trading venue or market segment.
- National Drug Code (NDC)
A U.S. identifier for a listed drug product and package configuration.
- Research Organization Registry (ROR) ID
An open identifier for research organizations.
- W3C PROV
A family of specifications for representing provenance of entities, activities, and agents.
Found an error on this page? See corrections for what counts as one, how to report it and the dated log.
Enquiry
Where does "Data lineage" bear on a decision of yours?
This entry sits under provenance and citation. If a decision in your organization turns on it, send the entity concerned, the sources you fuse to describe it and who may read which of them. We reply with what a first record would cover.
Or write to [email protected] in your own words.