| name | des-lineage-metadata-design |
| description | Use when designing data lineage, catalog metadata, business/technical/operational/reference metadata, run evidence, ownership, trust metadata, and auditability for data products. |
des-lineage-metadata-design
Purpose
Use this skill to create and validate the Lineage and Metadata Specification for any data engineering project.
This skill defines how business metadata, technical metadata, operational metadata, reference metadata, lineage, catalog entries, ownership, definitions, quality status, contracts, run evidence, semantic metadata, serving metadata, usage metadata, change metadata, version metadata, and audit metadata will be captured, organized, exposed, and governed.
The goal is to make the data product discoverable, explainable, auditable, debuggable, traceable, and governable across source, ingestion, Bronze, Silver, Gold, semantic, and serving layers.
Lineage and Metadata Design should answer:
Which assets need metadata?
What business meaning and ownership must be documented?
What technical schema and dependency metadata is required?
What run evidence proves how data was produced?
How far should lineage go?
Is column-level lineage required?
Which trust, quality, contract, and freshness metadata should be exposed?
Who maintains metadata?
How does metadata support governance, security, audit, troubleshooting, consumers, and AI/data agents?
In the Phase-Orchestrated Support Model, this phase is not Done when the Lineage and Metadata Specification is first written.
Phase 18 is Done only when:
Lineage and Metadata Specification exists
+ Phase 17 artifact and handoff are reviewed
+ lineage and metadata validation work is identified
+ metadata evidence is collected or waived with reason
+ artifact is revised from evidence
+ Phase 18 Done Gate passes
+ Phase 18 to Phase 19 handoff exists
When To Use
Use this skill when:
- Phase 17 Serving Layer Specification exists;
- Phase 17 handoff exists or the user explicitly accepts the risk of continuing without it;
- data products, datasets, metrics, contracts, transformations, quality rules, workflows, semantic objects, and serving outputs need lineage and metadata;
- users need a data catalog, data dictionary, glossary, ownership, field definitions, run metadata, quality status, trust status, or lineage graph;
- compliance, audit, troubleshooting, impact analysis, deletion requests, or incident response require traceability;
- AI/data-agent usage requires trustworthy semantic and lineage metadata;
- the workflow router selects Phase 18.
Do not use this skill to implement a catalog tool, write lineage extraction code, build metadata crawlers, deploy governance tools, configure Purview/DataHub/Collibra/dbt docs/OpenLineage, create metadata pipelines, create CI/CD workflows, or write implementation code.
Allowed in this phase:
metadata scope design
metadata category design
metadata inventory design
business metadata design
technical metadata design
operational metadata design
reference metadata design
dataset catalog requirements
field and schema metadata requirements
metric and KPI metadata requirements
contract metadata requirements
transformation lineage design
dataset lineage design
column-level lineage expectation design
semantic and serving lineage design
quality and trust metadata design
ownership and stewardship metadata design
usage and consumer metadata design
change and version metadata design
audit and compliance metadata design
metadata capture responsibility design
metadata freshness and maintenance policy design
Required Inputs
The agent should look for:
_des-output/planning-artifacts/01-business-discovery-brief.md;_des-output/planning-artifacts/02-business-question-catalog.md;_des-output/planning-artifacts/03-requirements-and-kpi-catalog.md;_des-output/planning-artifacts/04-data-product-specification.md;_des-output/planning-artifacts/05-data-source-inventory.md;_des-output/planning-artifacts/06-conceptual-domain-model.md;_des-output/planning-artifacts/08-ingestion-specification.md;_des-output/planning-artifacts/09-bronze-layer-specification.md;_des-output/planning-artifacts/10-silver-layer-specification.md;_des-output/planning-artifacts/11-gold-layer-specification.md;_des-output/planning-artifacts/12-data-contract-specification.md;_des-output/planning-artifacts/13-transformation-specification.md;_des-output/planning-artifacts/14-data-quality-specification.md;_des-output/planning-artifacts/15-orchestration-observability-specification.md;_des-output/planning-artifacts/16-semantic-model-specification.md;_des-output/planning-artifacts/17-serving-layer-specification.md;_des-output/phase-handoffs/phase-17-to-18-handoff.md;_des-output/evidence/phase-17/phase-17-evidence-pack.md, if available;_des-output/implementation-artifacts/phase-17-done-gate.md, if available;- workflow status file, if present;
- source inventory;
- Bronze/Silver/Gold dataset inventory;
- transformation dependency graph;
- contract inventory;
- quality rule inventory;
- workflow/run evidence expectations;
- semantic model inventory;
- serving output inventory;
- ownership and stewardship decisions.
If the Serving Layer Specification or Phase 17 handoff is missing or too weak, stop and ask whether to route back to des-serving-layer-design.
Output Files
Create or update the configured main output file:
_des-output/planning-artifacts/18-lineage-metadata-specification.md
Also create or update the Phase-Orchestrated Support outputs when using validated phase delivery:
_des-output/phase-support-plans/phase-18-support-plan.md
_des-output/evidence/phase-18/phase-18-evidence-pack.md
_des-output/implementation-artifacts/phase-18-artifact-revision.md
_des-output/implementation-artifacts/phase-18-done-gate.md
_des-output/phase-handoffs/phase-18-to-19-handoff.md
The main artifact must capture:
- lineage and metadata summary;
- metadata scope;
- metadata non-scope;
- metadata design principles;
- metadata categories;
- metadata inventory;
- business metadata;
- technical metadata;
- operational metadata;
- reference metadata;
- dataset catalog requirements;
- field and schema metadata;
- metric and KPI metadata;
- contract metadata;
- transformation lineage;
- dataset lineage;
- column-level lineage expectations;
- semantic and serving lineage;
- quality and trust metadata;
- ownership and stewardship metadata;
- usage and consumer metadata;
- change and version metadata;
- audit and compliance metadata;
- metadata capture responsibilities;
- metadata freshness and maintenance policy;
- lineage and metadata evidence summary;
- risks;
- assumptions;
- open questions;
- Phase 18 evidence summary;
- Phase 18 handoff notes;
- next skill recommendation.
On Activation
-
Read this SKILL.md completely.
-
Read customize.toml.
-
Identify output_file, template_file, checklist_file, status_file, and required upstream artifacts.
-
Identify Phase-Orchestrated Support files:
phase_support_plan_file;phase_evidence_pack_file;phase_artifact_revision_file;phase_done_gate_file;phase_handoff_file.
-
Load only steps/step-01-context-and-readiness.md.
-
Do not load step-02 or step-03 until the current step explicitly instructs you to continue.
-
Stop at every HALT point and wait for user input.
-
Do not invent owners, definitions, lineage paths, catalog fields, quality status, metadata maintenance processes, or compliance requirements.
-
Do not implement catalog, lineage extractor, crawler, OpenLineage, Purview, DataHub, Collibra, dbt docs, metadata pipelines, CI/CD, or code.
-
Before marking Phase 18 as Done, create or update the support plan, evidence pack, artifact revision notes, Done Gate, handoff, and workflow status.
Process Overview
The detailed execution procedure lives in steps/.
At a high level, this skill will:
- Confirm upstream dataset, transformation, contract, quality, semantic, serving, and ownership context.
- Confirm Phase 17 handoff readiness.
- Identify assets requiring metadata and lineage.
- Define metadata categories and inventory.
- Define business, technical, operational, and reference metadata requirements.
- Define dataset, transformation, and column-level lineage scope.
- Define trust, quality, contract, usage, version, and audit metadata.
- Define metadata capture and maintenance policies.
- Ask HALT questions for unresolved metadata decisions.
- Draft the Lineage and Metadata Specification.
- Create the Phase 18 Support Plan.
- Collect or reference Phase 18 evidence.
- Revise the Lineage and Metadata Specification using evidence.
- Run the Phase 18 Done Gate.
- Create the Phase 18 to Phase 19 Handoff.
- Update workflow status.
- Recommend the next skill.
Do not execute this overview directly. Follow the step files.
Phase-Orchestrated Support Model
Phase 18 uses lineage and metadata validation support work.
The purpose is not catalog implementation. The purpose is to make the data product discoverable, explainable, auditable, debuggable, governable, and ready for governance/security design.
Required Support Work
| Support Work | Purpose | Output |
|---|
| Phase 17 Handoff Review | Check metadata design follows serving requirements. | Evidence pack section |
| Metadata Scope Check | Validate metadata boundary. | Evidence pack section |
| Metadata Category Coverage Check | Validate business/technical/operational/reference coverage. | Evidence pack section |
| Metadata Asset Inventory Check | Ensure assets requiring metadata are listed. | Evidence pack section |
| Business Metadata Check | Validate definitions, glossary, rules, owners, usage. | Evidence pack section |
| Technical Metadata Check | Validate schema, fields, dependencies, constraints. | Evidence pack section |
| Operational Metadata Check | Validate run IDs, status, counts, errors, runtime, publish status. | Evidence pack section |
| Reference Metadata Check | Validate codes, units, calendars, mappings, lookups. | Evidence pack section |
| Dataset Catalog Requirement Check | Validate catalog fields. | Evidence pack section |
| Field/Schema Metadata Check | Validate field-level meaning/type/constraint metadata. | Evidence pack section |
| Metric/KPI Metadata Check | Validate metric definition, formula, owner, status, version. | Evidence pack section |
| Contract Metadata Check | Validate contract/version/SLA/schema/owner metadata. | Evidence pack section |
| Transformation Lineage Check | Validate input/output/rule lineage. | Evidence pack section |
| Dataset Lineage Check | Validate source→Bronze→Silver→Gold→Semantic→Serving paths. | Evidence pack section |
| Column-Level Lineage Expectation Check | Validate scope and feasibility of column-level lineage. | Evidence pack section |
| Semantic/Serving Lineage Check | Validate semantic and consumer-facing lineage. | Evidence pack section |
| Quality/Trust Metadata Check | Validate quality status, trust badge, freshness, warnings. | Evidence pack section |
| Ownership/Stewardship Metadata Check | Validate owner, steward, support contact, accountability. | Evidence pack section |
| Usage/Consumer Metadata Check | Validate downstream usage, consumers, API/export/dashboard/agent usage. | Evidence pack section |
| Change/Version Metadata Check | Validate schema, contract, metric, and model version metadata. | Evidence pack section |
| Audit/Compliance Metadata Check | Validate access/change/quality/incident/deletion traceability. | Evidence pack section |
| Metadata Capture Responsibility Check | Validate who captures what and when. | Evidence pack section |
| Metadata Freshness/Maintenance Check | Validate metadata update policy. | Evidence pack section |
| Phase 18 Done Gate | Decide whether Phase 18 is Done, Done with risks, or Blocked. | phase-18-done-gate.md |
| Phase 18 Handoff | Tell Phase 19 what governance/security must control. | phase-18-to-19-handoff.md |
HALT Policy
This skill must stop when a required decision cannot be safely inferred.
Stop especially when:
- upstream contexts are unknown;
- required inputs are missing.
Detailed HALT checkpoints are defined in steps/.
Guardrails
The agent must not:
- treat metadata as only table descriptions;
- treat lineage as a static diagram only;
- ignore operational metadata from workflows and runs;
- ignore reference metadata such as code lists, units, calendars, and mappings;
- ignore ownership and accountability;
- expose undocumented or uncertified datasets as trusted;
- claim column-level lineage exists unless scope and capture method are defined;
- ignore metadata maintenance and freshness;
- ignore downstream usage metadata;
- ignore compliance/audit traceability;
- hide trust, quality, contract, or freshness signals from catalog/consumer metadata where needed;
- implement catalog, scanner, lineage, crawler, OpenLineage, Purview, DataHub, Collibra, dbt docs, metadata pipelines, CI/CD, or code in this phase;
- mark lineage design Done if P1 outputs cannot trace back to upstream datasets, transformations, contracts, quality status, serving output, and owners.
Handoff To The Next Skill
Recommend des-governance-security-design only after:
Lineage and Metadata Specification exists
+ Phase 18 support plan exists or is waived with reason
+ Phase 18 evidence pack exists or evidence is waived with reason
+ Phase 18 Done Gate is Pass or Pass with risks
+ Phase 18 to Phase 19 handoff is Ready or Ready with Risks
If the phase is Draft or Blocked, recommend one of:
continue des-lineage-metadata-design
return to Step 02 lineage, metadata, and catalog design
resolve HALT question
route back to des-serving-layer-design
route back to des-semantic-model-design
route back to des-orchestration-observability
route back to des-data-quality
des-wise
des-correct-course
