Menu
Build and manage ETL pipelines for data migration with transformation, CDC, and monitoring
Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.
Based on SOC occupation classification
Reference for querying the Atlas knowledge graph through its MCP tools — the SECONDARY enrichment/comparison layer that adds best-practice context to systems you have ALREADY scanned from your real sources (`az`, repos, dirs). Use when you need to look up nodes, edges, kinds, clusters, stats, or wiki pages in Atlas to compare against your real inventory. (atlas graph, query atlas, atlas mcp, search the graph, graph neighbors, atlas record, atlas kinds, enrichment layer)
Atlas turns your STATED NEED into a real systems atlas by SCANNING your actual sources (Azure via `az`, git repos, local dirs) and process/data mining them, THEN enriching against the Atlas knowledge graph. Use this skill when asked to inventory/map your real systems, scan your cloud + repos + directories, mine the real processes or data they contain, or collect their real constraints/gotchas. (atlas, scan my systems, inventory our azure account, map my repos, real systems atlas, process mining, data mining, collect nuances, system discovery)
This skill should be used when the user asks to "find skills in the wild", "assimilate popular workflows", "discover SKILL.md files in repos", "research external skills", "find workflow patterns", "survey the skill landscape", "what skills exist out there", or wants to investigate public repositories for extractable processes, babysitter plugins, and reusable procedural insights. Searches GitHub for SKILL.md files, classifies repos by archetype, and maintains structured research under docs/reference-repos/.
JavaScript and TypeScript documentation generation using JSDoc and TSDoc. Parse source code, generate API documentation, validate coverage, and integrate with TypeDoc for comprehensive developer documentation.
Structured debugging methodology using hypothesis-driven investigation, log analysis, and bisection to isolate and resolve defects.
Zero-knowledge circuit development using Circom and Noir languages. Supports constraint optimization, ZK-friendly cryptographic primitives, proof generation (Groth16, PLONK), and Merkle tree implementations.
| name | etl-pipeline-builder |
| description | Build and manage ETL pipelines for data migration with transformation, CDC, and monitoring |
| allowed-tools | ["Bash","Read","Write","Grep","Glob","Edit"] |
| graph | {"domains":["domain:software-engineering"],"specializations":["specialization:code-migration-modernization"],"skillAreas":["skill-area:etl-pipelines","skill-area:database-migrations-zero-downtime"],"roles":["role:data-engineer","role:backend-engineer"],"workflows":["workflow:technical-debt-reduction"],"topics":["topic:refactoring"]} |
Builds and manages ETL (Extract, Transform, Load) pipelines for data migration, supporting incremental loads, CDC, and comprehensive monitoring.
Enable data pipeline creation for:
| Tool | Type | Integration Method |
|---|---|---|
| Apache Airflow | Orchestration | Python |
| dbt | Transformation | CLI |
| Airbyte | Data integration | API |
| Fivetran | SaaS ETL | API |
| AWS DMS | Cloud migration | CLI |
| Debezium | CDC | Config |
{
"pipelineId": "string",
"timestamp": "ISO8601",
"pipeline": {
"name": "string",
"source": {},
"target": {},
"mappings": [],
"transformations": [],
"schedule": "string"
},
"artifacts": {
"dagFile": "string",
"configFile": "string",
"sqlFiles": []
},
"deployment": {
"status": "string",
"url": "string"
}
}
data-migration-validator: Validationschema-comparator: Schema mappingdatabase-migration-orchestrator: Pipeline orchestrationdata-architect-agent: Pipeline design