| name | data-pipeline |
| description | Process, clean, transform, merge, validate, deduplicate, aggregate, convert, or export CSV, JSON, JSONL, Excel, logs, and other datasets. Use for repeatable ETL-style file workflows, especially when data quality and output verification matter. |
Data Pipeline
Use this skill for file-based data transformation. Use database-query for relational queries and web-scraper for collection before transformation.
Workflow
- Inspect the input format, encoding, headers, row count, schema, and file size before choosing a runtime.
- Define the output schema and transformation rules explicitly: coercion, normalization, missing-value policy, filters, joins, deduplication keys, aggregation, ordering, and destination.
- Use one runtime path. Prefer streaming or bounded chunks for large inputs; never load a very large file only for convenience.
- Preserve source files. Write to a new output unless the user explicitly authorizes replacement.
- Validate before delivery: required fields, types, uniqueness, row counts, rejected rows, totals, encoding, and parseability.
- Report input/output counts, validation failures, assumptions, and the exact output path. Never claim success from file creation alone.
For format-specific Python examples, chunking patterns, validation helpers, and a composable pipeline skeleton, read the detailed guide.
Guardrails
- Parameterize user-provided values and treat formulas/macros as untrusted when spreadsheets are involved.
- Keep rejected records or a sanitized error ledger when silent dropping would hide data loss.
- Make transformations deterministic and rerunnable.
- Do not infer destructive cleanup rules from ambiguous values; surface them for approval.
- Sample output for visual/semantic correctness in addition to structural validation.
