data-quality

name	data-quality
description	Guides agents through data quality design using AWS Glue Data Quality and DQDL. Use when defining quality rules, thresholds, anomaly detection, quarantine patterns, or integrating DQ checks into Glue ETL pipelines and the Data Catalog.

Data Quality

Overview

Use this skill when data quality is the design concern — not just "add a null check" but deciding what to measure, where to enforce it, what thresholds are acceptable, and what happens when data fails. Built on AWS Glue Data Quality and the Data Quality Definition Language (DQDL).

When to Use

• defining data quality rules for lake tables or ETL pipelines
• choosing between Data Catalog DQ (at rest) and ETL DQ (in transit)
• designing quarantine and remediation patterns for bad records
• setting up anomaly detection and dynamic thresholds
• integrating DQ scores into governance, alerting, or publish gates
• recommending rule coverage for Bronze, Silver, or Gold layers

Do not use this to validate application-layer input forms or API request schemas. Those belong in application code, not lake DQ.

Stack Context (this workspace)

Component	Detail
Engine	AWS Glue Data Quality (serverless, built on DeeQu)
Rule language	DQDL — domain-specific, open, version-controllable
Entry points	Data Catalog (rules on cataloged tables, scheduled) and ETL jobs (inline in Glue Studio or script)
Format support	S3, Iceberg, Hudi, Delta, Redshift, JDBC, Lake Formation managed tables
Anomaly detection	ML-based, uses analyzers + historical statistics, GA since Aug 2024
Record-level results	Supported in ETL jobs — identify which rows failed
Results storage	S3 (default) or Iceberg tables in Glue Catalog (July 2026+)
Integration	EventBridge (alerts), CloudWatch (metrics), CloudFormation (IaC)

Two Entry Points — When to Use Each

	Data Catalog DQ	ETL Job DQ
When	Rules on data at rest — scheduled monitoring of existing tables	Rules on data in transit — inline in the pipeline before writing
Who	Data stewards, analysts, governance teams	Data engineers building ETL
Record identification	Not supported	Supported — flag or quarantine specific failing rows
Rule recommendations	Supported (auto-suggest)	Not supported
Scaling	Fixed	Auto-scaling, Flex supported
Use case	"Is my Gold table still healthy today?"	"Don't let bad records reach Silver"

Use both together: ETL DQ catches problems before they land; Catalog DQ monitors drift after they land.

Workflow

1. Define what "quality" means for each layer. Quality expectations differ by medallion layer:

   Layer	DQ focus	Typical rules
   Bronze	Structural integrity	File not empty, expected columns present, parseable format, row count within expected range
   Silver	Business validity	Null checks on required fields, type conformance, referential integrity, uniqueness on keys, freshness
   Gold	Publish readiness	Aggregate completeness, no negative revenue, metric ranges, freshness SLA, row count vs. prior run

2. Write rules in DQDL. Structure:

   Rules = [
     Completeness "customer_id" > 0.99,
     Uniqueness "customer_id" > 0.99,
     ColumnValues "age" between 0 and 120,
     ColumnValues "email" matches "[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,}",
     IsComplete "order_date",
     IsPrimaryKey "order_id",
     ReferentialIntegrity "customer_id" "dim_customer.customer_id" > 0.98,
     RowCount between 1000 and 10000000,
     Freshness "ingestion_timestamp" <= 25 hours
   ]
   

   Key DQDL capabilities:

   • where clause to filter before applying rules
   • NOT operator for negation
   • Composite rules with nested logic
   • Dynamic thresholds: RowCount > avg(last(10)) — compare against historical
   • Constants for reusable values across large rulesets
   • Labels for organizing rules by team, domain, or category

3. Decide enforcement behavior. What happens when rules fail:

   Strategy	When	Implementation
   Warn and continue	Bronze (raw is raw)	Log DQ score, publish to EventBridge, don't stop the job
   Quarantine bad rows	Silver (protect downstream)	Route failing rows to s3://<bucket>/_quarantine/<table>/, continue with clean rows
   Fail the job	Gold (publish gate)	If DQ score < threshold, job fails, Gold not updated, SNS alert fires
   Conditional publish	Gold (soft gate)	Write results but mark table as "degraded" in catalog metadata

DQDL Rule Categories

Category	Example rules	Use for
Completeness	Completeness "col" > 0.99, IsComplete "col"	Required fields must not be null
Uniqueness	Uniqueness "col" > 0.99, IsPrimaryKey "col"	Key columns must not have duplicates
Validity	ColumnValues "col" between X and Y, ColumnValues "col" in ["A","B"]	Values within expected range or set
Pattern	ColumnValues "col" matches "regex"	Format validation (emails, phone numbers, IDs)
Freshness	Freshness "timestamp_col" <= N hours	Data arrived within expected window
Volume	RowCount between X and Y, RowCount > avg(last(10))	Table not empty, not unexpectedly large/small
Referential	ReferentialIntegrity "fk" "other_table.pk" > 0.98	Foreign key relationships hold
File-level	FileFreshness, FileSize, FileUniqueness, FileMatch	Source file checks before parsing
Anomaly	Analyzers + dynamic rules	ML-detected drift over time

Quarantine Pattern

When Silver DQ fails on specific rows:

Bronze table (all rows)
    │
    ▼
┌─────────────────────────┐
│  EvaluateDataQuality    │  DQDL ruleset applied
└────────┬────────────────┘
         │
    ┌────┴────┐
    │         │
    ▼         ▼
 PASS      FAIL
    │         │
    ▼         ▼
Silver    Quarantine
table     s3://<bucket>/_quarantine/<table>/dt=<date>/

Quarantined rows:

• Include the original row plus which rule(s) failed
• Are retained for investigation and replay
• Are NOT included in Silver or Gold
• Trigger an alert if count exceeds a threshold

Integration with Other Skills

Skill	How DQ connects
data-lake-config	Pipeline configs carry DQ rules in the quality_checks section; generate_configs.py emits them from the STTM
data-lake-infrastructure	Terraform provisions the quarantine path prefix, EventBridge rule, and CloudWatch alarms
data-lake-architecture	DQ is the enforcement mechanism for the "Bronze permissive, Gold disciplined" principle
data-lake-open-table-format	DQ runs after MERGE; results stored in Iceberg tables (July 2026+)
glue-data-catalog-and-lake-formation-governance	DQ scores visible in catalog metadata; publish gate enforces governance

Common Rationalizations

Rationalization	Reality
"We'll add data quality checks later."	DQ debt compounds: bad data in Silver propagates to Gold, dashboards show wrong numbers, and trust erodes faster than you can fix it.
"100% completeness is the only acceptable threshold."	Some fields are legitimately nullable. Set thresholds per column based on business meaning, not blanket perfection.
"A null check is enough."	Null checks catch only one failure mode. Type conformance, referential integrity, freshness, and volume drift are equally common.
"DQ is the platform team's responsibility."	Domain teams define what "quality" means for their data. The platform provides the framework; domains author the rules.
"Anomaly detection replaces static rules."	Anomaly detection catches drift; static rules catch invariants. You need both. An anomaly detector won't flag a missing customer_id if it's never seen one before.

Red Flags

• No DQ rules on Silver tables — bad data propagates unchecked to Gold
• DQ rules exist but failures don't block the pipeline or alert anyone
• Thresholds are all 100% — overly strict rules cause false failures and get disabled
• Quarantine path exists but nobody reviews or replays quarantined data
• Anomaly detection enabled without enough history (< 10 runs) — produces noise
• DQ scores not visible in catalog metadata — consumers can't assess trust
• Rules are hardcoded in job scripts rather than version-controlled DQDL files
• Record-level failure identification not used — you know "something failed" but not which rows

Verification

• Each medallion layer has DQ rules appropriate to its quality bar
• Enforcement behavior is explicit: warn, quarantine, or fail per layer
• DQDL rules are version-controlled, not embedded in job scripts
• Quarantine path exists and is monitored (not a black hole)
• Anomaly detection is enabled on Silver and Gold with sufficient history
• DQ score is recorded per run and visible in catalog metadata
• Gold publish gate uses DQ score as a go/no-go threshold
• Alerts fire to domain owners (not just platform) on failure