data-pipelines

name

data-pipelines

description

Umbrella skill for moving data from source to destination — end-to-end ETL/ELT design, DAG orchestration, real-time streaming, and system-to-system migration. Use this whenever the user is building, scheduling, debugging, or migrating a pipeline and it isn't yet clear which sub-area dominates. This skill ROUTES to the focused sub-skills (pipeline-design, orchestration-patterns, streaming-patterns, data-migration) and pulls in more than one when a task spans them (e.g. "design a streaming pipeline and orchestrate its backfill"). Trigger on: new ingestion job, batch vs. streaming choice, Airflow/Prefect/Dagster DAGs, CDC, backfill, cutover, dual-write, or "move data from X to Y".

Data Pipelines (Router)

This is a router skill. It groups the four skills that deal with getting data from a source into a destination and keeping it flowing. Diagnose which sub-area(s) the task touches, then invoke the matching sub-skill(s) with the Skill tool. For a task that spans areas, invoke several and combine their guidance.

How to route

If the task is about…	Invoke sub-skill
Designing a new ingestion job end-to-end: extraction strategy, idempotency, load patterns, raw/staging/marts layers, ELT vs ETL	pipeline-design
Scheduling multi-step pipelines, task dependencies, retries, SLAs, sensors, failure recovery, choosing between Airflow/Prefect/Dagster	orchestration-patterns
Real-time / near-real-time processing, Kafka/Flink/Kinesis/Spark Structured Streaming, consumer lag, windowing, exactly-once, late data	streaming-patterns
Moving between systems safely: cutover planning, backfill, dual-write, shadow reads, validation, rollback, retiring a legacy pipeline	data-migration

Routing rules

• Default new-pipeline questions to pipeline-design first — it frames source shape, volume, change pattern, and downstream contract, which the others build on.
• Batch vs. streaming undecided? Invoke streaming-patterns (it contains the stream-vs-batch decision) before committing to an architecture.
• Anything scheduled with >2 dependent steps also pulls in orchestration-patterns.
• Replacing or moving off an existing system always pulls in data-migration, usually alongside pipeline-design.
• Invoke via the Skill tool by name, e.g. call Skill(skill="orchestration-patterns"). Combine outputs; don't paraphrase from memory.

Related groups

• Modeling the data once it lands → [[data-modeling]]
• Validating and governing it → [[data-reliability]]
• Where it runs and what it costs → [[cloud-data-infra]]