| name | Data Quality |
| description | Use when defining data quality rules, schema contracts, or SLA monitoring for a dataset. Triggers on 'data quality', 'great expectations', 'schema contract', 'data SLA', 'data validation', 'null check'. |
Data Quality
When to invoke
- "Add data quality checks to this pipeline."
- "Define a schema contract for this dataset."
- "Our dashboard broke because upstream changed a column."
The six dimensions
Every dataset should have explicit tests covering:
- Completeness - expected rows present, no unexpected nulls.
- Uniqueness - primary key actually unique.
- Validity - values conform to type, range, regex, enum.
- Consistency - cross-field rules (end_date ≥ start_date), cross-dataset (foreign keys resolve).
- Timeliness - data arrived within its SLA.
- Accuracy - spot checks against a source of truth (sampled).
Workflow
- Write the contract first - a YAML/JSON schema describing columns, types, constraints, freshness SLA, ownership.
- Enforce at boundaries - validate on write into Silver, not only on read.
- Separate hard-fail from soft-warn:
- Hard fail → PK duplication, schema drift, missing partition → pipeline stops, pages on-call.
- Soft warn → row count outside ±20% of rolling baseline → ticket, does not block.
- Quarantine bad rows, don't drop them silently. Keep the rejection reason.
- Dashboard the SLA: freshness, volume anomaly, failed-check rate by dataset.
Tools
- Great Expectations / Soda - declarative tests, good for batch
- dbt tests - if you're in the dbt ecosystem
- Deequ / PyDeequ - Spark-native
- Monte Carlo / Anomalo - observability / anomaly detection layer on top
Anti-patterns
- Checks only in production - learn at the worst time.
- No owner on the dataset - nobody fixes alerts.
- Warn-only checks forever - noise becomes ignored.
- Testing the output of a transform instead of the input - you've already corrupted downstream.
References
