Data Lineage and Governance: Tracing a Dashboard Number Back to Its Source
How data lineage tracks a number through every transformation from its raw source to a dashboard, and why this tracing matters for debugging, regulatory compliance and organisational trust in data.
The question lineage exists to answer
An executive looks at a revenue dashboard and asks why this quarter's number looks 8% higher than the finance team's own reconciled figures. Answering that question well requires being able to trace the dashboard's number backward through every transformation that produced it: which raw source tables fed it, which joins and filters and aggregations were applied at each stage, which specific pipeline run generated the version currently displayed, and at what point in that chain the discrepancy with finance's figures was introduced. Data lineage is the discipline and tooling for answering exactly this question — recording, ideally automatically, the full dependency graph from raw ingestion through every intermediate transformation to every downstream consumer, so that any number anywhere in the system can be traced both backward to its origins and forward to everything that depends on it.
Without lineage, this kind of debugging degenerates into an archaeological exercise: someone has to manually read through SQL scripts, dbt models, Spark jobs and dashboard definitions, cross-referencing table names and hoping the person who wrote the pipeline three years ago left comments explaining an unusual join condition. In any organisation past a certain size, with dozens or hundreds of pipelines feeding a shared warehouse, this manual approach simply does not scale, and the cost shows up as debugging sessions that take days instead of minutes, and as a broader organisational problem: analysts and executives who have been burned once by an unexplained discrepancy start distrusting the numbers generally, which is a much harder problem to fix than the original bug, because restoring trust in a data platform after it has been lost takes far longer than the incident that broke it.
Table-level versus column-level lineage
Lineage tooling operates at two distinct levels of granularity, and the difference matters enormously for how useful the resulting graph actually is. Table-level lineage records that table B was built from table A, typically inferred by parsing the SQL or job definitions that materialise each table and extracting which source tables each query reads from. This is comparatively easy to build (most modern data catalogue tools like DataHub, Amundsen or OpenLineage can extract it automatically by parsing SQL query logs or orchestration metadata) and answers coarse questions well: "what feeds the revenue_summary table" or "what would break if I dropped this staging table."
Column-level lineage tracks the finer-grained claim that a specific output column was derived from a specific set of input columns through a specific transformation — that revenue_summary.total_revenue equals the sum of orders.amount after a currency-conversion join against fx_rates.rate, filtered to orders.status = 'completed'. This is substantially harder to extract automatically, because it requires actually parsing and semantically understanding the transformation logic (a SQL SELECT with a dozen joins and CASE statements, or an arbitrary Python transformation in a Spark job) rather than just noting which tables were referenced, and gets genuinely difficult once transformations pass through non-SQL code, external APIs, or opaque business logic embedded in application code rather than declarative pipeline definitions. The payoff for the extra effort is large, though: column-level lineage is what actually lets someone answer "which specific upstream field caused this specific downstream number to be wrong," rather than just narrowing the search to "somewhere in these eleven tables," which is often still too broad to be useful in a genuine incident under time pressure.
Automated capture versus manual documentation
Lineage systems get their information one of two ways, and the reliability difference between them is the single biggest factor in whether a lineage system stays trustworthy over time. Manually documented lineage — a wiki page or spreadsheet where engineers write down what feeds what — degrades the moment the underlying pipelines change and nobody remembers (or bothers) to update the documentation, which happens routinely enough that manually maintained lineage documentation in most organisations is reliably out of date within months of being written, defeating the entire purpose of having it.
Automated lineage capture instead derives the dependency graph directly from the systems that actually run the pipelines: parsing SQL query logs from the warehouse, hooking into orchestration tools like Airflow or dbt to record job dependencies as pipelines execute, and using standardised open protocols like OpenLineage to emit lineage events directly from the compute engines (Spark, Flink, dbt) as jobs run, so the lineage graph is a byproduct of production execution rather than a separately maintained artefact that can drift out of sync. This automated approach is strictly more reliable because it cannot become stale in the way manual documentation does — if the pipeline changes, the next run automatically emits updated lineage — though it still requires genuine engineering investment to instrument every stage of a heterogeneous pipeline (a Python script doing an ad hoc transformation outside the standard orchestration framework is invisible to automated lineage capture unless someone deliberately instruments it), which is why most large organisations run a mix: strong automated coverage for warehouse-native transformations (SQL, dbt) and a smaller set of manually documented edge cases for legacy or bespoke pipelines that fall outside the standard tooling.
Why regulators and auditors care about lineage specifically
Beyond internal debugging, data lineage has become a specific, named requirement in several major regulatory frameworks, precisely because "we produced this number" is not a sufficient answer for a regulator; "we can prove exactly how this number was produced, from which source, and that the process is repeatable and correctly authorised" is what compliance actually requires. The Basel Committee's BCBS 239 principles for risk data aggregation, which apply to systemically important banks, explicitly require institutions to be able to trace risk figures back to their source data and demonstrate the accuracy of the aggregation process, a requirement that is functionally impossible to satisfy without genuine, auditable lineage rather than institutional memory. GDPR's right to erasure creates a related but distinct lineage need: when a data subject requests deletion, an organisation needs to know every downstream table, model feature and derived dataset that incorporated their personal data, not just the original record, which is precisely a forward-lineage query (tracing from a source record to everything downstream that depends on it) rather than the backward-lineage query used for debugging a dashboard number.
This dual direction is worth being explicit about because the two use cases genuinely need different traversals of the same graph: debugging and root-cause analysis are backward queries (given this output, what fed it), while compliance obligations like GDPR erasure, and impact analysis before a schema change ("if I drop this column, what breaks downstream"), are forward queries (given this source, what depends on it). A lineage system that only supports one direction well is only solving half the problem, which is why mature lineage tooling is built around a genuinely bidirectional dependency graph rather than a one-way documentation trail, and why lineage has moved from a nice-to-have engineering convenience to a load-bearing piece of regulatory and governance infrastructure in data-intensive, regulated industries.
Frequently Asked Questions
What is the difference between table-level and column-level data lineage?
Table-level lineage shows which tables feed which other tables. Column-level lineage shows which specific output columns were derived from which specific input columns and by what transformation, giving a much more precise answer when debugging exactly why one field is wrong.
Can data lineage be fully automated?
Largely, for standard warehouse-native pipelines built in SQL, dbt or orchestrated Spark jobs, using tools that parse query logs or emit lineage events during execution. Bespoke transformations written outside these standard frameworks, such as ad hoc scripts, typically need manual instrumentation to appear in the lineage graph.
Why do regulators specifically require data lineage rather than just accurate reports?
Regulatory frameworks like BCBS 239 for banking risk data require institutions to demonstrate, not just assert, that a reported figure was produced through a traceable, repeatable process from verified source data, which requires an auditable record of every transformation the figure passed through.
How does lineage help with GDPR right-to-erasure requests?
It provides a forward trace from a specific data subject's original record to every downstream table, feature or derived dataset that incorporated it, which is necessary to actually fulfil a deletion request across a data platform where personal data has been copied and transformed many times.