data-governance

name	data-governance
description	Establish catalogs, lineage, classification, contracts, and regulatory controls across the data platform. Use when standing up a governance program or passing a GDPR/DPDP/HIPAA audit.

Data Governance

Governance is automated metadata + enforced contracts, not a wiki. Catalog every dataset, classify every field, contract every interface, and prove lineage end to end.

Stack Baseline (2026)

Concern	Recommended
Lakehouse catalog	Apache Polaris (Iceberg REST), Unity Catalog
Metadata catalog	OpenMetadata 1.4, DataHub 0.14, Atlan, Collibra
Lineage	OpenLineage 1.x, Marquez
Classification / PII	Macie, BigID, Immuta, OpenMetadata classifiers
Access policy	OPA/Rego, Apache Ranger 2.5, Lake Formation, Immuta
Contracts	Data Contract Specification 1.1, dbt contracts
Quality	Great Expectations, Soda, Monte Carlo
Privacy	Tokenization (Vault, Skyflow), DSAR tooling, k-anonymity
Regulations	GDPR, EU AI Act, India DPDP, HIPAA, PCI DSS 4.0, SOX

When to Use

• New data platform or major migration.
• Audit, certification, or regulatory deadline.
• Cross-team data sharing or external data products.
• Adoption of data mesh / domain-owned data products.

Prerequisites

• Inventory of sources, datasets, and consumers.
• Named data product owners per domain.
• Identity provider with groups/roles for access policy.
• Object store + lakehouse with versioning enabled.

Instructions

1. Establish the operating model

Adopt federated governance per Data Mesh (Dehghani): a central platform team builds capabilities; domain teams own their data products and contracts.

flowchart LR
    Platform[Platform team\nCatalog, lineage, policy] -->|capabilities| D1[Domain: Orders]
    Platform --> D2[Domain: Catalog]
    Platform --> D3[Domain: Customer]
    D1 --> Mesh[(Data Product Marketplace)]
    D2 --> Mesh
    D3 --> Mesh

2. Catalog every dataset automatically

Connect OpenMetadata or DataHub to Postgres, Iceberg/Polaris, Kafka, dbt, BI tools. Require: owner, domain, classification, SLO, contract URL.

3. Classify fields and bind policy

# OpenMetadata classification example
table: customer
columns:
  - name: email
    tags: [PII.Email, GDPR.Article6]
  - name: dob
    tags: [PII.Sensitive, GDPR.Article9]
  - name: country_code
    tags: [BusinessTerm.Country]

# OPA: only finance + on-call may read PII.Sensitive
package data.access
default allow = false
allow {
  input.action == "read"
  some t; t := input.resource.tags[_]
  t == "PII.Sensitive"
  input.subject.groups[_] == "finance"
}

4. Publish a data contract per product

dataContractSpecification: 1.1.0
id: customer.gold.dim_customer
info: { owner: customer-domain, version: 3.0.0 }
terms:
  usage: Internal analytics; external sharing prohibited.
  limitations: No re-identification of pseudonymized fields.
models:
  dim_customer:
    fields:
      customer_id: { type: string, required: true, classification: PII.Pseudonym }
      email_hash:  { type: string, required: true, classification: PII.Hashed }
servicelevels:
  freshness:    { threshold: 1h }
  completeness: { threshold: 99.5 }
  retention:    { period: 7y, basis: legal }

5. Wire end-to-end lineage

Emit OpenLineage from ingestion (Airbyte/Debezium), transformation (dbt, Spark, Flink), and BI (Looker, Superset). Block deploys when lineage coverage drops below threshold.

6. Operate privacy controls

• DSAR runbook: locate, export, erase across all stores within statutory window.
• Tokenize at the edge (Skyflow, HashiCorp Vault Transit) for PCI/PII.
• Right-to-be-forgotten: Iceberg DELETE + expire_snapshots + downstream propagation.
• Maintain a Records of Processing Activities (Article 30) generated from the catalog.

7. Audit and continuously improve

Quarterly: contract compliance, access reviews, classification recall sample, SLA breaches, cost-by-domain. Feed findings into platform backlog.

Common Pitfalls

Pitfall	Why it hurts	Mitigation
Catalog without owners	Dies in 6 months	Owner field is mandatory
Manual lineage	Always stale	OpenLineage automation
PII discovered ad hoc	Audit failure	Automated classifiers + tags
Contracts as PDFs	Not enforceable	Machine-readable + CI gate
Centralized data team owns all	Bottleneck	Mesh + federated ownership
Erase only in warehouse	DSAR non-compliance	Propagate to all derived stores
Governance vs. velocity framing	Stalemate	Treat policy as code

Output Format

Deliver: governance operating model, catalog connector inventory, classification taxonomy, OPA policy bundle, data contracts per product, lineage coverage report, DSAR runbook, and audit calendar.

Authoritative References

• Zhamak Dehghani, Data Mesh (O'Reilly, 2022)
• OpenLineage spec - https://openlineage.io/docs/
• OpenMetadata docs - https://docs.open-metadata.org/
• Apache Polaris - https://polaris.apache.org/
• Data Contract Specification - https://datacontract.com/
• EU GDPR text - https://gdpr-info.eu/
• India DPDP Act 2023 - https://www.meity.gov.in/data-protection-framework
• EU AI Act - https://artificialintelligenceact.eu/