What data lineage means at each layer of an AI pipeline, where catalog lineage stops short of the transformation logic auditors ask about, and how to answer a lineage audit for a model output.
By
Billy Allocca
Table of Contents
The Data Lineage Imperative: Why Every AI Model Your Enterprise Runs Needs a Provenance Trail
Data lineage for AI models is the recorded chain from source system, through every transformation and feature, to the exact training set a model learned from and the rows it retrieves at inference. Without column-level and transformation-level lineage in the data layer, you cannot say which data produced an output, whose data was in scope, or when training data ended.
The regulatory deadline is dated. Regulation (EU) 2024/1689, the EU AI Act, requires in Article 10(2)(b) that high-risk training data be governed with documented "data collection processes and the origin of data, and in the case of personal data, the original purpose of the data collection" [1]. Article 12 requires that such systems "technically allow for the automatic recording of events (logs) over the lifetime of the system" [2], and Article 13 requires providers to give deployers "specifications for the input data ... or any other relevant information in terms of the training, validation and testing data sets used" [3]. Those obligations reach Annex III systems on 2 December 2027 under the current timeline [4] [5].
The gap is measured too. The 2026 State of Data Integrity and AI Readiness study by Precisely and Drexel LeBow, covering more than 500 senior data leaders, found 71 percent of organizations with an established governance program report high trust in their data, against 50 percent without one [6]. This guide extends the post on one governance model for data and AI: lineage is the record a single governance boundary produces.
What Does Data Lineage Mean for AI Models at Each Layer?
Lineage is a different record at each stage of an AI pipeline, and a provenance trail is complete only when the four stages are joined. Data lineage is the recorded path of a data element from origin through every system, transformation, and consumer that touched it. Training data provenance is the subset documenting the composition, sources, time window, and licensing of the data a model learned from. OpenLineage, a Linux Foundation AI & Data graduated project since July 2023 [7], models lineage as Runs of Jobs that consume and produce Datasets, with Facets for schema, code, and quality [8].
Layer | What lineage records here | Audit question it answers | Evidence |
|---|---|---|---|
Raw source | System of record, table or topic, extraction time, owner, legal basis for personal data | Where did the data originate and for what purpose was it collected (Article 10(2)(b))? | Catalog entry, ingestion run ID, source snapshot |
Transformation | Each job that read inputs and wrote outputs, the SQL or code executed, column mappings | Which columns were joined, filtered, aggregated, or masked? | OpenLineage run events with SqlJobFacet and column lineage facet [8] [9] |
Feature engineering | Feature definitions, entity join keys, time-to-live, materialization windows, rows written | Which features fed the model, derived from which source columns? | Feature store registry and materialization events [10] |
Training set composition | Snapshot IDs of every input table, row counts, filters, split ratios, cutoff date | What exactly did the model learn from, and when does its knowledge end? | Pinned table snapshots plus the training run record [11] |
The common failure is a broken join between layers: the catalog knows the source table, the MLOps tool knows the training run, and no record ties them to one model version.
Why Do Data Catalogs Capture Table-Level Lineage but Not Column-Level or Transformation-Level?
Catalog lineage stops where SQL parsing stops, and AI pipelines spend most of their life outside parseable SQL. Column-level lineage records which input columns produced each output column; transformation-level lineage also records the operation (join, filter, aggregation, masking) and the code that performed it.
Vendors have moved past pure table-level lineage, so the comparison is about coverage boundaries. DataHub documents column-level lineage for BigQuery, Snowflake, Redshift, and dbt, with a SQL parser it rates at 97 to 99 percent accuracy [12]. Unity Catalog captures column lineage for SQL and DataFrame workloads but only table-level lineage for UDFs, and none for path-based reads [13]. Collibra's technical lineage includes "all source code and data transformation details" [14]. Marquez has served column lineage since version 0.27.0 in October 2022, with Spark the first producer to emit it [15]; in the OpenLineage matrix, Spark emits column lineage while dbt, Feast, and Great Expectations emit dataset-level facets [16].
Granularity | What it records | Where it breaks in AI pipelines | Tools that capture it (per vendor docs) |
|---|---|---|---|
Table-level | Dataset A was read to produce dataset B | Cannot show whether a PII column was read or dropped | DataHub, Unity Catalog, Collibra, Marquez, all OpenLineage producers [12] [13] [14] [15] |
Column-level | Output column derives from specific input columns | UDFs, path-based reads, pandas or PyTorch code, renamed objects [13] | DataHub (warehouse and dbt sources), Unity Catalog (SQL and DataFrame), Collibra technical lineage, Marquez via Spark [12] [13] [14] [15] |
Transformation-level | The operation and code per column, typed DIRECT or INDIRECT with subtypes such as JOIN, FILTER, AGGREGATION, MASKING | Needs a producer that emits code facets; most notebook training code emits nothing | OpenLineage column lineage facet and SqlJobFacet; Collibra source-code capture [9] [14] |
The last row is the AI gap. A feature computed in a notebook, written to a feature store, and read into a training job crosses three tools, and if any one fails to emit an event the trail ends there.
How Do You Track Training Data Lineage From Source to Feature Store to Training Set?
You track training data lineage by pinning every input to an immutable snapshot and emitting a lineage event at each hop, so the training set can be reconstructed from records. A feature store is a system that defines, computes, and serves model features consistently between training and inference. Feast emits OpenLineage events for registry changes (feature views, entities, data sources, saved datasets) and for each materialization, with facets carrying TTL, join keys, time windows, and rows written, with no pipeline code changes [10]. Apache Iceberg adds the fixed point in time: time travel by snapshot ID or timestamp, plus a snapshots metadata table listing every committed version [11]. The OpenLineage column lineage facet records, per output field, its input fields and transformation type, including whether masking was applied [9].
Hop | Emitting system | Record captured | Pinned by |
|---|---|---|---|
Source system to raw table | Ingestion job (Spark, Flink, CDC) | Run ID, source, row count, schema | Iceberg snapshot ID of the landing table |
Raw table to cleaned table | SQL or Spark transformation | Column lineage facet with JOIN, FILTER, MASKING subtypes | Snapshot ID plus job version |
Cleaned table to features | Feast feature view materialization | Feature definitions, entities, TTL, window, rows written [10] | Materialization run ID |
Features to training set | Training data assembly job | Input snapshot IDs, split ratios, cutoff timestamp | Saved dataset in the feature registry |
Training set to model version | Training run (MLflow, Kubeflow, or similar) | Model artifact hash, hyperparameters, training set reference | Model registry entry |
Data mesh architectures raise the difficulty because domain teams publish data products independently. Federated computational governance, the fourth data mesh principle, calls for "automated execution of decisions by the platform" [17], and lineage emission is the clearest case.
What Does a Lineage Audit Response Look Like for an AI Output?
An audit response is a set of joined records that walks backward from one output to the sources that shaped it, and the cost of lacking one is public. The Dutch Data Protection Authority fined the Tax Administration 2.75 million euros in December 2021 for using applicants' dual nationality as an indicator in an automated risk system for childcare benefits; the data should have been deleted in January 2014 and about 1.4 million records remained in 2018 [18]. In December 2023 the Stanford Internet Observatory found 3,226 suspected instances of child sexual abuse material in LAION-5B, a five-billion-link dataset used to train Stable Diffusion, and LAION withdrew it [19]. In both cases outsiders discovered what was in the data after the models shipped.
Step | Auditor's question | Record consulted | Example answer |
|---|---|---|---|
1 | Which model version produced the output? | Inference log with model ID and timestamp | credit_risk_v14, served 2026-08-14 09:12 UTC |
2 | Which features or rows were in the request? | Inference log with dataset and row identifiers | 12 features from service risk_v3; 2 rows from loans table, snapshot 8842 |
3 | Which training set did that version learn from? | Model registry entry pointing to a saved dataset | Saved dataset train_2026q2 |
4 | Which source tables and snapshots composed it? | Training run event with input snapshot IDs | 6 Iceberg tables, 41.2 million rows, snapshots listed |
5 | Whose data was in scope, under what purpose? | Column lineage to source, joined with catalog purpose tags | Customers with accounts opened 2019 to 2026; nationality excluded by MASKING at hop 2 |
6 | When did the model's knowledge end? | Cutoff timestamp on the saved dataset | 2026-06-30 23:59 UTC |
7 | Who read the training data, under which policy? | Policy engine audit log for the training job's identity | Service account train-risk, Ranger policy risk-training-pii-masked |
Steps 5 and 7 are the ones a catalog alone cannot answer, because they depend on what the transformation did and which policy governed the read.
Why Not Just Record Lineage in the MLOps Tool?
MLOps platforms record the training run well and the data poorly, because they see the data only when it enters the run. MLOps is the set of practices and tools for versioning, training, deploying, and monitoring models. A model registry stores a hash of the training file and the hyperparameters, and that is worth having. It cannot tell you the file came from six tables, that one column was hashed and two dropped, or that a source table received a correction three days after the cutoff.
Provenance is lost upstream of any training framework. Longpre and colleagues audited more than 1,800 text datasets across 44 collections and found license omission above 70 percent on hosting sites and error rates above 50 percent where a license was stated [20]. Inside an enterprise, purpose of collection, legal basis, and column sensitivity belong to the source data and reach a training tool only if the data layer carries them forward. Inference-time AI, including retrieval-augmented generation and agents querying databases, has no training run at all; the output depends on which rows a query returned under which policy, and only the layer that executed the query can log that. The companion guides on policy-as-code for AI data access and data contracts for AI agents cover the enforcement and schema halves of that problem.
NexusOne Anchors AI Lineage in the Governance Layer
NexusOne produces lineage as a byproduct of governance, because every read by a human, a training job, or an agent passes through one identity model and one policy engine. Identity is federated from Active Directory, Okta, LDAP, or SAML into Keycloak, and a policy defined once in Apache Ranger is enforced across Trino, Apache Spark, Apache Kyuubi, and S3 object storage per object. Tagging a dataset in DataHub and mapping a role to that tag generates the Ranger policies at bucket, schema, and table level, following the tag-based model Ranger documents for separating classification from access [21]. Step 7 of the audit table, which policy governed the training read, becomes a log lookup.
The AI & Data Control Plane extends the record to inference. Each agent and MCP request is classified at the boundary, PII is redacted before any egress to a frontier model, per-role token budgets apply, and every request is logged with the datasets it touched. Federated query across every source keeps one lineage graph across platforms, and because the estate runs on open formats such as Apache Iceberg, snapshot IDs pin every training input on-prem, in the cloud, hybrid, or air-gapped. The post on enterprise AI governance as one control plane explains the architecture and the EU AI Act data layer guide maps it to articles. To walk through your own audit path with an architect, book an expert consultation.
Key Takeaways
Lineage for AI is four joined records: source origin and purpose, transformation logic, feature derivation, and training set composition with a pinned cutoff.
EU AI Act Articles 10, 12, and 13 require origin documentation, automatic logging, and dataset specifications for high-risk systems from 2 December 2027 for Annex III.
Catalogs capture column-level lineage for warehouse SQL and dbt, and coverage ends at UDFs, path-based reads, and notebook code, where training pipelines live.
OpenLineage column lineage facets, feature store events, and Iceberg snapshot IDs together make a training set reconstructible.
An audit response is a stepwise join from inference log to model version to saved dataset to source snapshots, with masking and policy records answering whose data was in scope.
Lineage has to be captured where the read is enforced, in the governance layer, because MLOps tools see data only after provenance is gone.
FAQ
What Is Data Lineage for AI Models in the Enterprise?
Data lineage for AI models is the recorded chain from each source system, through every transformation and feature computation, to the training set a model version learned from and the rows it retrieves at inference. It requires pinned snapshots and a cutoff date so the training set can be reconstructed. OpenLineage represents it as runs, jobs, datasets, and facets, including a column lineage facet per output field [8] [9].
How Does Data Provenance Support AI Model Governance?
Provenance is the evidence that governance policies were applied to the data a model learned from. It shows that a sensitive column was masked before training, that personal data was collected for a compatible purpose as Article 10(2)(b) requires [1], and that a given source snapshot was in or out of scope. Without it, a policy is intent with no record of execution.
How Do You Track Training Data Lineage in an Enterprise?
Emit a lineage event at each hop (ingestion, transformation, feature materialization, training set assembly, training run) and pin every input to an immutable table snapshot. Feast emits OpenLineage events for feature views and materializations [10], Spark emits column lineage [16], and Apache Iceberg snapshot IDs give each input a version that can be queried later [11]. Store the joined record in the governance layer that authorized the reads, so lineage and access log share one identity.
What Does Data Governance for AI Outputs Look Like for a CDO?
It means being able to answer, for any single output, which model version produced it, which rows or features fed it, which training set that model learned from, whose data was in scope, and when training data ended. The seven-step audit table above is the working form of that answer. Article 12's logging requirement and Article 13's dataset specifications are the regulatory drivers [2] [3].
References
Article 10: Data and Data Governance, EU Artificial Intelligence Act (artificialintelligenceact.eu), https://artificialintelligenceact.eu/article/10/
Article 12: Record-Keeping, EU Artificial Intelligence Act (artificialintelligenceact.eu), https://artificialintelligenceact.eu/article/12/
Article 13: Transparency and Provision of Information to Deployers, EU Artificial Intelligence Act (artificialintelligenceact.eu), https://artificialintelligenceact.eu/article/13/
Implementation Timeline, EU Artificial Intelligence Act (artificialintelligenceact.eu), https://artificialintelligenceact.eu/implementation-timeline/
Regulation (EU) 2024/1689 (Artificial Intelligence Act), EUR-Lex, https://eur-lex.europa.eu/eli/reg/2024/1689/oj
Fourth Annual Study Finds AI Confidence Outpaces Readiness as Data Integrity Gaps Persist, Precisely and Drexel LeBow, 21 January 2026, https://www.precisely.com/press-release/fourth-annual-study-finds-ai-confidence-outpaces-readiness-as-data-integrity-gaps-persist/
OpenLineage project page, LF AI & Data Foundation, https://lfaidata.foundation/projects/openlineage/
Object Model, OpenLineage Specification, https://openlineage.io/docs/spec/object-model
Column Lineage Dataset Facet, OpenLineage Specification, https://openlineage.io/docs/spec/facets/dataset-facets/column_lineage_facet
OpenLineage Integration, Feast documentation, https://docs.feast.dev/reference/openlineage
Spark Queries (Time Travel and Snapshots Metadata Table), Apache Iceberg documentation, https://iceberg.apache.org/docs/latest/spark-queries/
Automatic Lineage Extraction, DataHub documentation, https://docs.datahub.com/docs/generated/lineage/automatic-lineage-extraction
View Data Lineage Using Unity Catalog, Databricks documentation, https://docs.databricks.com/aws/en/data-governance/unity-catalog/data-lineage
About Collibra Data Lineage, Collibra Product Resource Center, https://productresources.collibra.com/docs/collibra/latest/Content/CollibraDataLineage/co_collibra-data-lineage.htm
Trying Out the New Column Lineage Feature, Marquez Project blog, 25 October 2022, https://marquezproject.ai/blog/column-lineage-demo/
Integrations and Capability Matrix, OpenLineage documentation, https://openlineage.io/docs/integrations/
Data Mesh Principles and Logical Architecture, Zhamak Dehghani, martinfowler.com, https://martinfowler.com/articles/data-mesh-principles.html
Tax Administration Fined for Discriminatory and Unlawful Data Processing, Autoriteit Persoonsgegevens, https://www.autoriteitpersoonsgegevens.nl/en/current/tax-administration-fined-for-discriminatory-and-unlawful-data-processing
Identifying and Eliminating CSAM in Generative ML Training Data and Models, Stanford Internet Observatory, 23 December 2023, https://stacks.stanford.edu/file/druid:kh752sm9123/ml_training_data_csam_report-2023-12-23.pdf
A Large-Scale Audit of Dataset Licensing and Attribution in AI, Longpre et al., Nature Machine Intelligence, August 2024, https://www.nature.com/articles/s42256-024-00878-8
Tag Based Policies, Apache Ranger wiki, https://cwiki.apache.org/confluence/display/RANGER/Tag+Based+Policies
