data-quality-guardian

license	Apache-2.0
name	data-quality-guardian
description	Great Expectations, dbt tests, anomaly detection, and data contracts for data quality. Activate on: data quality, data validation, Great Expectations, data contract, anomaly detection, SLA, freshness check, schema validation. NOT for: dbt model structure (use dbt-analytics-engineer), schema evolution (use schema-evolution-manager).
allowed-tools	Read,Write,Edit,Bash(npm:*,npx:*,python:*,dbt:*)
category	Data & Analytics
tags	["data-quality","testing","great-expectations","data-contracts","anomaly-detection"]
pairs-with	[{"skill":"dbt-analytics-engineer","reason":"dbt tests are a primary data quality enforcement tool"},{"skill":"data-lineage-tracker","reason":"Quality issues traced via lineage to root cause"},{"skill":"airflow-dag-orchestrator","reason":"Quality checks run as orchestrated pipeline steps"}]

Data Quality Guardian

Implement comprehensive data quality frameworks with automated validation, anomaly detection, data contracts, and SLA monitoring.

Activation Triggers

Activate on: "data quality", "data validation", "Great Expectations", "data contract", "anomaly detection", "SLA", "freshness check", "schema validation", "data observability", "Soda", "elementary"

NOT for: dbt project layout → dbt-analytics-engineer | Schema evolution strategy → schema-evolution-manager | Data migration validation → data-migration-specialist

Quick Start

1. Define data contracts — agree on schema, freshness, volume, and value ranges with upstream producers
2. Implement tests — dbt tests for SQL models, Great Expectations for raw/external data
3. Monitor freshness — alert when source tables are stale beyond SLA
4. Detect anomalies — statistical checks for volume, distribution, and null rate changes
5. Build quality dashboard — centralized view of all quality metrics across pipelines

Core Capabilities

Domain	Technologies
Testing	dbt tests, Great Expectations 1.x, Soda Core 3.x
Observability	elementary (dbt), Monte Carlo, Anomalo
Contracts	Soda data contracts, dbt model contracts, Protobuf schemas
Anomaly Detection	elementary anomaly monitors, custom z-score, Prophet
Alerting	Slack/PagerDuty integration, SLA miss alerts

Architecture Patterns

Multi-Layer Quality Checks

Raw Data (Landing)
    ↓
Layer 1: Schema Validation
    - Column types match contract
    - No unexpected NULLs in required fields
    - Row count within expected range
    ↓
Layer 2: Business Rule Validation
    - Referential integrity (FK relationships hold)
    - accepted_values constraints
    - Custom SQL assertions (e.g., revenue >= 0)
    ↓
Layer 3: Statistical Anomaly Detection
    - Row count deviation from 7-day rolling average
    - Null rate spike detection
    - Distribution shift (KL divergence)
    ↓
Layer 4: Freshness & SLA Monitoring
    - Source loaded within 2 hours
    - Downstream models built within 4 hours
    - Dashboard data <6 hours old