Menu
Data pipeline architecture, ETL/ELT patterns, data quality, batch vs stream processing, orchestration, and data governance principles.
Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.
Based on SOC occupation classification
| name | data-engineering |
| description | Data pipeline architecture, ETL/ELT patterns, data quality, batch vs stream processing, orchestration, and data governance principles. |
Guidelines for building reliable, scalable data pipelines and platforms.
| Pattern | When to Use |
|---|---|
| Batch ETL | Scheduled, high volume, latency-tolerant |
| Streaming | Real-time, event-driven, low latency |
| Lambda | Both batch and stream (complexity trade-off) |
| Kappa | Stream-only, reprocessing via replay |
| Medallion | Bronze (raw) → Silver (cleaned) → Gold (curated) |
Source → Validate (schema, nulls, types) → Transform → Validate (business rules) → Load → Verify (counts, checksums)
| Tool | Strength |
|---|---|
| Apache Airflow | Most mature, Python-native, DAG-based |
| Dagster | Type-safe, asset-oriented, modern |
| Prefect | Pythonic, flow-based, cloud-native |
| Model | When |
|---|---|
| Star schema | Analytics, BI dashboards, simple queries |
| Data Vault | Enterprise, auditability, multiple sources |
| Dimensional | Aggregated reporting, OLAP |
Session bootstrap + workflows for Pathfinder semantic navigation tools. Covers: discovery protocol, tool chaining patterns (explore, impact, audit, debug), search optimization, LSP degraded mode, and error recovery.
Playwright browser automation via MCP. Covers E2E testing, UI review, web scraping, screenshot capture, and general browser interaction. MCP-first — CLI is fallback only.
Safe command execution: input sanitization, timeout handling, output capture, error propagation. For spawning processes, shell commands, system calls.
Git conventions: conventional commits, branch naming, PR hygiene, release tagging.
Structured incident workflow: severity classification, triage, diagnosis, mitigation, postmortem, and prevention. Template-driven with blameless review.
Constructs, validates, and traverses a Directed Acyclic Graph (DAG) from scope cards for safe level-based parallel dispatch. Determines execution order via topological sort. Detects cycles and invalid dependencies.