| name | des-orchestration-observability |
| description | Use when designing schedules, triggers, dependencies, quality gates, retries, failure handling, recovery, backfills, run evidence, alerting, incident behavior, and operational monitoring for data pipelines. |
des-orchestration-observability
Purpose
Use this skill to create and validate the Orchestration and Observability Specification for any data engineering project.
This skill defines workflow scheduling, triggers, dependencies, retry policy, failure handling, quality gate integration, publish gates, alerting, incident handling, monitoring signals, run evidence, operational metadata, SLA tracking, backfill operations, and recovery expectations.
The goal is to make data pipelines operable, observable, recoverable, and auditable before implementation begins.
Orchestration and Observability should answer:
What workflows exist?
What triggers them?
What must run before what?
Where are quality gates placed?
When is an output safe to publish?
What can be retried safely?
What happens on failure?
Who is alerted?
How are incidents escalated?
What signals prove freshness, quality, runtime, volume, and cost?
What operational evidence is captured for audit/debugging?
In the Phase-Orchestrated Support Model, this phase is not Done when the Orchestration and Observability Specification is first written.
Phase 15 is Done only when:
Orchestration and Observability Specification exists
+ Phase 14 artifact and handoff are reviewed
+ operational validation work is identified
+ operational evidence is collected or waived with reason
+ artifact is revised from evidence
+ Phase 15 Done Gate passes
+ Phase 15 to Phase 16 handoff exists
When To Use
Use this skill when:
- Phase 14 Data Quality Specification exists;
- Phase 14 handoff exists or the user explicitly accepts the risk of continuing without it;
- ingestion, Bronze, Silver, Gold, transformation, and quality rules need to be orchestrated;
- the project needs scheduling, dependency management, retries, alerts, SLAs, observability, backfill, or recovery design;
- quality gates need to block or warn before publishing outputs;
- pipeline health, freshness, runtime, row counts, quality results, cost, or incidents need monitoring;
- the user asks to create Airflow DAGs, Fabric pipelines, Databricks workflows, Prefect flows, Dagster assets, GitHub Actions, or orchestration logic;
- the workflow router selects Phase 15.
Do not use this skill to implement orchestration code, write DAGs, create Fabric pipeline JSON, create monitoring dashboards, write CI/CD workflows, deploy infrastructure, or implement alerting integrations.
Allowed in this phase:
workflow inventory design
logical dependency graph design
schedule and trigger strategy
source readiness check design
ingestion orchestration design
Bronze/Silver/Gold orchestration design
quality gate placement
publish and release gate design
retry and timeout policy
failure recovery policy
backfill and replay operation design
late data and correction operation design
alerting and notification policy
incident and escalation policy
observability signal inventory
freshness/SLA monitoring design
volume/completeness monitoring design
quality result monitoring design
runtime/performance monitoring design
cost/usage monitoring design
run evidence and audit metadata design
operational ownership design
Required Inputs
The agent should look for:
_des-output/planning-artifacts/01-business-discovery-brief.md;_des-output/planning-artifacts/02-business-question-catalog.md;_des-output/planning-artifacts/03-requirements-and-kpi-catalog.md;_des-output/planning-artifacts/04-data-product-specification.md;_des-output/planning-artifacts/05-data-source-inventory.md;_des-output/planning-artifacts/06-conceptual-domain-model.md;_des-output/planning-artifacts/07-architecture-decision-record.md;_des-output/planning-artifacts/08-ingestion-specification.md;_des-output/planning-artifacts/09-bronze-layer-specification.md;_des-output/planning-artifacts/10-silver-layer-specification.md;_des-output/planning-artifacts/11-gold-layer-specification.md;_des-output/planning-artifacts/12-data-contract-specification.md;_des-output/planning-artifacts/13-transformation-specification.md;_des-output/planning-artifacts/14-data-quality-specification.md;_des-output/phase-handoffs/phase-14-to-15-handoff.md;_des-output/evidence/phase-14/phase-14-evidence-pack.md, if available;_des-output/implementation-artifacts/phase-14-done-gate.md, if available;- workflow status file, if present;
- approved architecture orchestration boundary;
- ingestion frequency and trigger;
- transformation dependency graph;
- data quality gates;
- freshness/SLA expectations;
- contract validation expectations;
- failure handling expectations;
- alerting and owner expectations.
If the Data Quality Specification or Phase 14 handoff is missing or too weak, stop and ask whether to route back to des-data-quality.
Output Files
Create or update the configured main output file:
_des-output/planning-artifacts/15-orchestration-observability-specification.md
Also create or update the Phase-Orchestrated Support outputs when using validated phase delivery:
_des-output/phase-support-plans/phase-15-support-plan.md
_des-output/evidence/phase-15/phase-15-evidence-pack.md
_des-output/implementation-artifacts/phase-15-artifact-revision.md
_des-output/implementation-artifacts/phase-15-done-gate.md
_des-output/phase-handoffs/phase-15-to-16-handoff.md
The main artifact must capture:
- orchestration and observability summary;
- orchestration scope;
- orchestration non-scope;
- orchestration design principles;
- workflow inventory;
- dependency graph;
- schedule and trigger strategy;
- source readiness checks;
- ingestion orchestration;
- Bronze/Silver/Gold transformation orchestration;
- quality gate integration;
- publish and release gates;
- retry and timeout policy;
- failure handling and recovery policy;
- backfill and replay operations;
- late-arriving and corrected data handling;
- alerting and notification policy;
- incident and escalation policy;
- observability signal inventory;
- freshness and SLA monitoring;
- volume and completeness monitoring;
- quality result monitoring;
- runtime and performance monitoring;
- cost and usage monitoring;
- run evidence and audit metadata;
- operational ownership;
- operational evidence summary;
- risks;
- assumptions;
- open questions;
- Phase 15 evidence summary;
- Phase 15 handoff notes;
- next skill recommendation.
On Activation
-
Read this SKILL.md completely.
-
Read customize.toml.
-
Identify output_file, template_file, checklist_file, status_file, and required upstream artifacts.
-
Identify Phase-Orchestrated Support files:
phase_support_plan_file;phase_evidence_pack_file;phase_artifact_revision_file;phase_done_gate_file;phase_handoff_file.
-
Load only steps/step-01-context-and-readiness.md.
-
Do not load step-02 or step-03 until the current step explicitly instructs you to continue.
-
Stop at every HALT point and wait for user input.
-
Do not invent schedules, retries, alerts, owners, SLAs, incident paths, or publish gates.
-
Do not write Airflow, Fabric, Databricks, Dagster, Prefect, GitHub Actions, SQL, Python, monitoring dashboard, CI/CD, or infrastructure code.
-
Before marking Phase 15 as Done, create or update the support plan, evidence pack, artifact revision notes, Done Gate, handoff, and workflow status.
Process Overview
The detailed execution procedure lives in steps/.
At a high level, this skill will:
- Confirm upstream transformation, contract, Gold, Silver, Bronze, and requirement context.
- Confirm Phase 14 handoff readiness.
- Identify datasets and outputs requiring quality rules.
- Define data quality dimensions and rule categories.
- Map rules to datasets, contracts, transformations, and layers.
- Define thresholds, severity, failure handling, and quality gates.
- Define evidence, reporting, ownership, and monitoring expectations.
- Ask HALT questions for unresolved quality decisions.
- Draft the Data Quality Specification.
- Create the Phase 14 Support Plan.
- Collect or reference Phase 14 evidence.
- Revise the Data Quality Specification using evidence.
- Run the Phase 14 Done Gate.
- Create the Phase 14 to Phase 15 Handoff.
- Update workflow status.
- Recommend the next skill.
Do not execute this overview directly. Follow the step files.
Phase-Orchestrated Support Model
Phase 15 uses operational validation support work.
The purpose is not DAG implementation. The purpose is to make workflows operable, observable, recoverable, auditable, and safe for downstream semantic modeling and serving phases.
Required Support Work
| Support Work | Purpose | Output |
|---|
| Phase 14 Handoff Review | Check operational design follows quality gate expectations. | Evidence pack section |
| Workflow Scope Check | Validate operational boundary. | Evidence pack section |
| Workflow Inventory Check | Ensure workflows are listed. | Evidence pack section |
| Dependency Graph Check | Validate logical ordering and dependency risks. | Evidence pack section |
| Schedule/Trigger Check | Validate workflow start behavior. | Evidence pack section |
| Source Readiness Check | Validate source availability/pre-checks. | Evidence pack section |
| Ingestion Orchestration Check | Validate source-to-Bronze orchestration. | Evidence pack section |
| Transformation Orchestration Check | Validate Bronze/Silver/Gold execution order. | Evidence pack section |
| Quality Gate Integration Check | Validate DQ gate placement and behavior. | Evidence pack section |
| Publish/Release Gate Check | Validate consumer-facing publish behavior. | Evidence pack section |
| Retry/Timeout Check | Validate safe retry and timeout policy. | Evidence pack section |
| Failure/Recovery Check | Validate failure behavior and recovery path. | Evidence pack section |
| Backfill/Replay Check | Validate historical reprocessing operations. | Evidence pack section |
| Late Data/Correction Check | Validate late/corrected data operations. | Evidence pack section |
| Alerting/Notification Check | Validate alert routing and severity. | Evidence pack section |
| Incident/Escalation Check | Validate incident path. | Evidence pack section |
| Observability Signal Check | Validate monitoring signals. | Evidence pack section |
| Freshness/SLA Monitoring Check | Validate freshness measurement. | Evidence pack section |
| Volume/Completeness Monitoring Check | Validate volume/completeness monitoring. | Evidence pack section |
| Quality Result Monitoring Check | Validate DQ result monitoring. | Evidence pack section |
| Runtime/Performance Monitoring Check | Validate runtime and performance signals. | Evidence pack section |
| Cost/Usage Monitoring Check | Validate cost/usage signals where needed. | Evidence pack section |
| Run Evidence/Audit Check | Validate audit/debug metadata. | Evidence pack section |
| Operational Ownership Check | Validate owners and escalation. | Evidence pack section |
| Phase 15 Done Gate | Decide whether Phase 15 is Done, Done with risks, or Blocked. | phase-15-done-gate.md |
| Phase 15 Handoff | Tell Phase 16 what semantic modeling can rely on operationally. | phase-15-to-16-handoff.md |
HALT Policy
This skill must stop when a required decision cannot be safely inferred.
Stop especially when:
- upstream contexts are unknown;
- required inputs are missing.
Detailed HALT checkpoints are defined in steps/.
Guardrails
The agent must not:
- treat orchestration as only scheduling;
- ignore dependencies between ingestion, transformation, quality, and publish;
- ignore data quality gates;
- retry unsafe or non-idempotent steps without approval;
- continue publishing when P1 quality gates fail unless explicitly approved;
- alert nobody;
- create noisy alerts without severity;
- omit evidence needed for audit and debugging;
- ignore backfill/replay operations;
- ignore SLA/freshness monitoring;
- implement DAGs or pipeline code in this phase;
- mark the artifact Done if P1 workflows lack schedule/trigger, dependency, retry/failure handling, quality gate, alerting, and ownership.
Handoff To The Next Skill
Recommend des-semantic-model-design only after:
Orchestration and Observability Specification exists
+ Phase 15 support plan exists or is waived with reason
+ Phase 15 evidence pack exists or evidence is waived with reason
+ Phase 15 Done Gate is Pass or Pass with risks
+ Phase 15 to Phase 16 handoff is Ready or Ready with Risks
If the phase is Draft or Blocked, recommend one of:
continue des-orchestration-observability
return to Step 02 workflow scheduling and monitoring
resolve HALT question
route back to des-data-quality
route back to des-transformation-design
route back to des-data-contracts
des-wise
des-correct-course
