Scalable Data Lineage for ML Pipelines: Track Metadata, Provenance, and Model Inputs in Production

Why scalable data lineage matters for ML pipelines

Modern ML systems rely on many moving parts: raw data, feature engineering, model training, hyperparameters, deployment artifacts and runtime inputs. Tracking lineage helps answer practical questions such as which dataset produced a given prediction, which feature version fed the model and which preprocessing step changed performance. In production, lineage that scales and stays queryable is a key part of observability and reproducibility.

Core concepts: metadata, provenance, inputs

Metadata is descriptive information about artifacts: schema, size, statistics, creation time. Provenance records the causal chain of transformations that produced an artifact: which job, which commit, which config. Model inputs are the actual features, parameters and external signals used at inference time. Designing lineage around these three concepts keeps records actionable.

Scalability challenges to anticipate

High cardinality of events when pipelines process millions of rows or thousands of daily runs
Heterogeneous sources: event logs, batch files, streaming topics, feature stores
Query performance for forensic or compliance queries across long time ranges
Retention and storage costs as lineage history grows

Architectural patterns

Some pragmatic patterns help balance flexibility and cost:

Event-first capture: emit lightweight lineage events at task boundaries and aggregate downstream
Graph model for relationships: represent artifact->process->artifact as edges for rich queries
Hybrid storage: store immediate event logs in a scalable append store and materialize summarized lineage into a graph DB or search index for fast queries
Sampling and summarization: keep full fidelity for a sliding window and compact older history via sketches or aggregated stats

Instrumentation points in a pipeline

Data ingestion: record source path, partition, record count, checksum
Feature transformation: log transformation id, code commit, feature version
Training runs: persist training dataset snapshot id, hyperparameters, model artifact id
Deployment and serving: capture model id, container digest, runtime feature versions used at inference
Monitoring alerts: link a drift or performance alert back to the exact input snapshot and model version

Concrete example: capture lineage with a simple Python snippet

The snippet below demonstrates emitting a compact lineage event as JSON to an object store or message queue. It contains dataset metadata, process info and produced artifact id. This pattern is portable and can be adapted to Airflow, Dagster, or any orchestration system.

import json
import hashlib
import time
from pathlib import Path

def checksum(path):
    h = hashlib.sha256()
    with open(path, "rb") as f:
        while chunk := f.read(8192):
            h.update(chunk)
    return h.hexdigest()

def emit_lineage_event(target, event_store_path):
    event = {
        "timestamp": int(time.time()),
        "process": {
            "name": "featurize_v2",
            "run_id": "run_20260225_001",
            "commit": "abc123"
        },
        "inputs": [
            {"type": "raw_csv", "path": str(target / \"data.csv\"), "checksum": checksum(target / \"data.csv\")}
        ],
        "outputs": [
            {"type": "feature_table", "id": "ft_users_v2", "path": str(target / \"features.parquet\")}
        ],
        "notes": "sample event for lineage capture"
    }
    # write to append log (could be Kafka, S3, or object store)
    with open(event_store_path, \"a\") as f:
        f.write(json.dumps(event) + \"\\n\")

# usage example
emit_lineage_event(Path(\"/tmp/job_123\"), \"/tmp/lineage_events.log\")

Choosing storage and query layers

There is no single right store. Tradeoffs matter:

Append logs (Kafka, S3) scale for ingestion and replayability but need a query layer
Graph DB (Neo4j, JanusGraph) models relationships naturally and supports deep lineage queries but can be costly at large scale
Search index (Elasticsearch) offers fast text and filtered queries for metadata and is useful for dashboards
Relational DB with indexes works for structured lookup and joins and fits teams familiar with SQL

A common approach is to keep raw events in an append store and run a daily job that materializes a graph or search index optimized for the queries your team runs most.

Query patterns to support

Find all models that used a given dataset snapshot
Trace predictions back to the exact preprocessing pipeline
List pipelines affected by a schema change or upstream code commit
Aggregate lineage by service owner, team, or data domain

Operational considerations

Latency: choose synchronous capture only where causal guarantees are needed; otherwise emit asynchronously
Retention: define retention tiers, for example full events for 90 days and aggregated lineage beyond that
Access control: protect lineage records as they may reveal sensitive sources or PII paths
Schema evolution: include versioned event schemas and a schema registry or compatibility checks

Integration ideas with common tools

Some integration patterns are practical without heavy engineering:

Hook into orchestration task success handlers to emit lineage events
Wrap feature store reads/writes with lightweight metadata logging
Use model registry hooks to record model input signatures and dataset ids
Connect monitoring alerts to the lineage index for automated root cause traces

Best practices checklist

Define minimal event schema that captures process id, input artifact ids, output artifact ids and context
Capture stable identifiers for datasets and features rather than file paths alone
Keep events small and emit frequently for observability, aggregate later for analytics
Instrument critical paths first: training, deployment and serving
Provide easy developer primitives so teams emit lineage without friction

Example mapping: from Airflow to serving

Imagine an Airflow DAG that ingests data, computes features, trains a model and registers it. Each task emits a lineage event with task id and artifact ids. The model registry adds model id and signature. The serving layer reads the model id plus a feature version manifest to ensure inference uses the same feature versions recorded at training. When a drift alert fires, you can query the lineage index to identify which upstream dataset partition changed and which tasks are implicated.

Practical tradeoffs and next steps

Start small: implement a compact event schema, instrument a few critical pipelines and ensure you can answer one or two key questions such as which dataset produced a model. Iterate by adding search and graph materializations for the queries that are slow. Expect to tune retention and indexing to control costs. Over time, lineage will reduce time-to-debug and improve collaboration between data, ML and production engineering teams.