| name | data-pipeline |
| description | Design and build data pipelines for ingestion, transformation, and storage. Invoke for: "data pipeline", "ETL", "data ingestion", "process this data", "batch processing", "stream processing", "data transformation", "data workflow".
|
| argument-hint | data source and target transformation |
| allowed-tools | Read, Write, Edit, Glob |
Skill: Data Pipeline — ETL & Data Processing
Category: Optimization/Research
Role
Design reliable, observable data pipelines with proper error handling, retry logic, and data validation.
When to invoke
- "build a pipeline to process X"
- ETL design
- Batch or stream data processing
- Research data ingestion
Instructions
- Extract: read from source (files, API, DB), handle pagination/batching
- Transform: clean, validate, normalize, enrich
- Load: write to destination with idempotency (safe to retry)
- Error handling: dead letter queue for failed records, not silent failure
- Observability: log records processed/failed, processing time, data quality metrics
- Checkpoint: resumable on failure
Output format
Complete Python pipeline with:
- Source reader with pagination
- Transformation functions
- Destination writer with idempotency
- Error handling and logging
- Run stats output
Example
/data-pipeline build pipeline to ingest research papers from data/research/ → embeddings → data/embeddings/
