data-pipeline-engineering

## Overview Data pipelines fail silently. Bad data flows downstream, corrupts reports, poisons ML training sets, and violates compliance requirements — often for days before anyone notices. This skill builds pipelines with the discipline to catch failures at the source. ## When to Use - Before building any ETL/ELT pipeline - When designing data transformations - When adding a new data source to a pipeline - When a downstream consumer reports incorrect data ## Process ### Step 1: Define the data contract Before any code: document the contract for every data source: - Schema (columns, types, nullability) - Volume expectations (rows per batch/day) - Freshness SLA (how old can data be?) - Quality invariants (column X is always positive, column Y is always a valid email) The contract is your test suite specification. ### Step 2: Implement contract validation at ingestion Validate incoming data against the contract as the first step of every pipeline. Fail loudly on schema violations, out-of-range values, and unexpected nulls. Never silently drop bad records without alerting. ### Step 3: Design for idempotency Every pipeline run must be idempotent: running it twice on the same input produces the same output. Implement with: append-only writes + deduplication, or upserts with stable primary keys, or overwrite semantics with explicit partitions. ### Step 4: Implement data lineage Track: where did each record come from? What transformations were applied? When was it processed? This is required for debugging, compliance (GDPR right-to-erasure), and impact analysis. ### Step 5: Design the quality gate After transformation, before writing to the destination: - Row count check (within expected range?) - Key quality metrics (null rates, uniqueness, referential integrity) - Distribution check (not wildly different from yesterday?) - Business rule validation (revenue never negative, user_id always set) Fail the pipeline and alert if quality gates fail. ### Step 6: Handle late-arriving and out-of-order data Define: what is the watermark? How long do you wait for late data? What happens to a record that arrives after the window closes? ### Step 7: Design the error handling strategy For each failure mode: - Source unavailable → retry with backoff, alert after N failures - Schema change in source → fail loudly, alert, do not silently drop - Quality gate failure → fail loudly, quarantine bad data, alert - Transform error → log with context, route to dead letter queue, alert ### Step 8: Implement observability Track: records in, records out, records failed, processing latency, source freshness, pipeline run duration. Alert on: any of these metrics going outside normal range. ### Step 9: Test the pipeline - Unit tests for each transformation - Integration tests with representative sample data - Tests for the quality gate (does it catch bad data?) - Tests for late-arriving data handling - Load tests (does it handle 10x volume?) ### Step 10: Document the pipeline For each pipeline: what does it do, what are its inputs, what are its outputs, what are its dependencies, what do you do when it fails? On-call engineers must be able to debug it without the original author. ## Anti-Rationalizations **"The source is reliable — we don't need contract validation"** Every source eventually sends bad data. The question is whether you catch it. **"Data lineage is a nice-to-have"** Data lineage becomes a requirement the first time you need to answer: "which records were affected by this bug?" or "which users' data do we need to delete?" ## Verification Requirements - [ ] Data contracts defined for all sources - [ ] Contract validation runs at ingestion - [ ] Pipeline is idempotent - [ ] Data lineage tracked - [ ] Quality gates implemented before destination writes - [ ] Late data strategy defined - [ ] Error handling routes to dead letter queue with alerts - [ ] Observability metrics tracked - [ ] Tests for transformations and quality gates - [ ] Runbook written