data-lineage-tracker

license	Apache-2.0
name	data-lineage-tracker
description	OpenLineage, DataHub, Marquez for data lineage tracking and impact analysis. Activate on: data lineage, OpenLineage, DataHub, Marquez, impact analysis, data catalog, column lineage, data discovery. NOT for: data quality validation (use data-quality-guardian), dbt documentation (use dbt-analytics-engineer).
allowed-tools	Read,Write,Edit,Bash(npm:*,npx:*,python:*,docker:*)
category	Data & Analytics
tags	["data-lineage","openlineage","datahub","data-catalog","impact-analysis"]
pairs-with	[{"skill":"data-quality-guardian","reason":"Lineage traces quality issues to root cause"},{"skill":"dbt-analytics-engineer","reason":"dbt generates lineage metadata automatically"},{"skill":"airflow-dag-orchestrator","reason":"Airflow emits OpenLineage events per task"}]

Data Lineage Tracker

Implement end-to-end data lineage tracking using OpenLineage, DataHub, and Marquez for impact analysis, debugging, and compliance.

Activation Triggers

Activate on: "data lineage", "OpenLineage", "DataHub", "Marquez", "impact analysis", "data catalog", "column lineage", "data discovery", "data provenance", "downstream impact"

NOT for: Data quality checks → data-quality-guardian | dbt model documentation → dbt-analytics-engineer | Schema evolution → schema-evolution-manager

Quick Start

1. Choose platform — DataHub (full catalog + lineage), Marquez (lineage-focused), Amundsen, OpenMetadata
2. Instrument pipelines — integrate OpenLineage with Airflow, Spark, and dbt for automatic lineage
3. Capture column-level lineage — trace which source columns feed which target columns
4. Build impact analysis — before changing a table, see all downstream consumers
5. Integrate with incidents — link data quality alerts to lineage for root cause analysis

Core Capabilities

Domain	Technologies
Lineage Standard	OpenLineage 1.x (open standard, LFAI)
Catalogs	DataHub 0.14+, OpenMetadata 1.5+, Amundsen
Lineage Backend	Marquez (OpenLineage reference), DataHub lineage
Integrations	Airflow OpenLineage, Spark OpenLineage, dbt, Great Expectations
Visualization	DataHub lineage graph, Marquez UI, dbt docs

Architecture Patterns

OpenLineage Event Flow

Airflow Task              Spark Job                 dbt Model
     │                         │                         │
     ├─ START event ──→        ├─ START event ──→        ├─ START event ──→
     │                         │                         │
     ├─ RUNNING event ─→       ├─ RUNNING event ─→       │
     │                         │                         │
     ├─ COMPLETE event ─→      ├─ COMPLETE event ─→      ├─ COMPLETE event ─→
     │  (with I/O datasets)    │  (with I/O datasets)    │  (with I/O datasets)
     ↓                         ↓                         ↓
              ┌────────────────────────────────────┐
              │     OpenLineage Backend            │
              │  (Marquez / DataHub / custom)      │
              │                                    │
              │  Stores: job runs, datasets,       │
              │  input/output relationships,       │
              │  column-level lineage              │
              └────────────────────────────────────┘