| name | data-governance |
| description | Establish catalogs, lineage, classification, contracts, and regulatory controls across the data platform. Use when standing up a governance program or passing a GDPR/DPDP/HIPAA audit. |
Data Governance
Governance is automated metadata + enforced contracts, not a wiki. Catalog every dataset, classify every field, contract every interface, and prove lineage end to end.
Stack Baseline (2026)
| Concern | Recommended |
|---|
| Lakehouse catalog | Apache Polaris (Iceberg REST), Unity Catalog |
| Metadata catalog | OpenMetadata 1.4, DataHub 0.14, Atlan, Collibra |
| Lineage | OpenLineage 1.x, Marquez |
| Classification / PII | Macie, BigID, Immuta, OpenMetadata classifiers |
| Access policy | OPA/Rego, Apache Ranger 2.5, Lake Formation, Immuta |
| Contracts | Data Contract Specification 1.1, dbt contracts |
| Quality | Great Expectations, Soda, Monte Carlo |
| Privacy | Tokenization (Vault, Skyflow), DSAR tooling, k-anonymity |
| Regulations | GDPR, EU AI Act, India DPDP, HIPAA, PCI DSS 4.0, SOX |
When to Use
- New data platform or major migration.
- Audit, certification, or regulatory deadline.
- Cross-team data sharing or external data products.
- Adoption of data mesh / domain-owned data products.
Prerequisites
- Inventory of sources, datasets, and consumers.
- Named data product owners per domain.
- Identity provider with groups/roles for access policy.
- Object store + lakehouse with versioning enabled.
Instructions
1. Establish the operating model
Adopt federated governance per Data Mesh (Dehghani): a central platform team builds capabilities; domain teams own their data products and contracts.
flowchart LR
Platform[Platform team\nCatalog, lineage, policy] -->|capabilities| D1[Domain: Orders]
Platform --> D2[Domain: Catalog]
Platform --> D3[Domain: Customer]
D1 --> Mesh[(Data Product Marketplace)]
D2 --> Mesh
D3 --> Mesh
2. Catalog every dataset automatically
Connect OpenMetadata or DataHub to Postgres, Iceberg/Polaris, Kafka, dbt, BI tools. Require: owner, domain, classification, SLO, contract URL.
3. Classify fields and bind policy
table: customer
columns:
- name: email
tags: [PII.Email, GDPR.Article6]
- name: dob
tags: [PII.Sensitive, GDPR.Article9]
- name: country_code
tags: [BusinessTerm.Country]
# OPA: only finance + on-call may read PII.Sensitive
package data.access
default allow = false
allow {
input.action == "read"
some t; t := input.resource.tags[_]
t == "PII.Sensitive"
input.subject.groups[_] == "finance"
}
4. Publish a data contract per product
dataContractSpecification: 1.1.0
id: customer.gold.dim_customer
info: { owner: customer-domain, version: 3.0.0 }
terms:
usage: Internal analytics; external sharing prohibited.
limitations: No re-identification of pseudonymized fields.
models:
dim_customer:
fields:
customer_id: { type: string, required: true, classification: PII.Pseudonym }
email_hash: { type: string, required: true, classification: PII.Hashed }
servicelevels:
freshness: { threshold: 1h }
completeness: { threshold: 99.5 }
retention: { period: 7y, basis: legal }
5. Wire end-to-end lineage
Emit OpenLineage from ingestion (Airbyte/Debezium), transformation (dbt, Spark, Flink), and BI (Looker, Superset). Block deploys when lineage coverage drops below threshold.
6. Operate privacy controls
- DSAR runbook: locate, export, erase across all stores within statutory window.
- Tokenize at the edge (Skyflow, HashiCorp Vault Transit) for PCI/PII.
- Right-to-be-forgotten: Iceberg
DELETE + expire_snapshots + downstream propagation.
- Maintain a Records of Processing Activities (Article 30) generated from the catalog.
7. Audit and continuously improve
Quarterly: contract compliance, access reviews, classification recall sample, SLA breaches, cost-by-domain. Feed findings into platform backlog.
Common Pitfalls
| Pitfall | Why it hurts | Mitigation |
|---|
| Catalog without owners | Dies in 6 months | Owner field is mandatory |
| Manual lineage | Always stale | OpenLineage automation |
| PII discovered ad hoc | Audit failure | Automated classifiers + tags |
| Contracts as PDFs | Not enforceable | Machine-readable + CI gate |
| Centralized data team owns all | Bottleneck | Mesh + federated ownership |
| Erase only in warehouse | DSAR non-compliance | Propagate to all derived stores |
| Governance vs. velocity framing | Stalemate | Treat policy as code |
Output Format
Deliver: governance operating model, catalog connector inventory, classification taxonomy, OPA policy bundle, data contracts per product, lineage coverage report, DSAR runbook, and audit calendar.
Authoritative References
