Menu
Data pipeline architecture, ETL/ELT patterns, data quality, batch vs stream processing, orchestration, and data governance principles.
Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.
Based on SOC occupation classification
Structured fault tolerance for coordinator agents. 5-level escalation ladder (Retry → Replace → Skip → Redistribute → Degrade), dead-man timers, degraded completion protocol, and cross-level escalation format. Load when orchestrating agents that may fail.
Structured code review protocol for inspecting code quality against the full rule set. Use when auditing code written by yourself or another agent, during the /audit workflow, or when the user asks for a code review.
Reusable convergence protocol for coordinator agents. Defines the BRIEFING → ITERATE → GATE → CONVERGE loop, context hygiene rules, self-succession protocol, turn budget, and handoff compression. Load when orchestrating multi-iteration workflows.
Pre-flight checklist and post-implementation self-review protocol. Use before generating any code (pre-flight) and after writing code but before verification (self-review) to catch issues early.
MECE task decomposition, file ownership enforcement, DAG-based execution, and safe merge protocol for intra-domain parallel dispatch. The safety invariants that prevent merge chaos when multiple agents write in parallel. Applies recursively at every nesting depth.
Shared protocols for all agents in the multi-agent pipeline: recursive nesting, pre-implementation restatement, parallel dispatch format, and agent definition cascade. Load this skill instead of inlining these protocols in every agent file.
| name | data-engineering |
| description | Data pipeline architecture, ETL/ELT patterns, data quality, batch vs stream processing, orchestration, and data governance principles. |
Guidelines for building reliable, scalable data pipelines and platforms.
| Pattern | When to Use |
|---|---|
| Batch ETL | Scheduled, high volume, latency-tolerant |
| Streaming | Real-time, event-driven, low latency |
| Lambda | Both batch and stream (complexity trade-off) |
| Kappa | Stream-only, reprocessing via replay |
| Medallion | Bronze (raw) → Silver (cleaned) → Gold (curated) |
Source → Validate (schema, nulls, types) → Transform → Validate (business rules) → Load → Verify (counts, checksums)
| Tool | Strength |
|---|---|
| Apache Airflow | Most mature, Python-native, DAG-based |
| Dagster | Type-safe, asset-oriented, modern |
| Prefect | Pythonic, flow-based, cloud-native |
| Model | When |
|---|---|
| Star schema | Analytics, BI dashboards, simple queries |
| Data Vault | Enterprise, auditability, multiple sources |
| Dimensional | Aggregated reporting, OLAP |