self-healing-models

name	self-healing-models
description	This skill should be used when the user asks about "concept drift", "data drift", "model degradation", "model staleness", "production model monitoring", "self-healing", "automatic retraining", "shadow model", "champion-challenger", "blue-green deployment", "canary deployment", "Kolmogorov-Smirnov test", "PSI", "Population Stability Index", "model refresh", "continuous learning", "online learning", or when a production model's performance has degraded over time due to distribution shift.
version	1.0.0

Self-Healing Models — Drift Detection and Autonomous Recovery

Provides the complete framework for detecting statistical distribution shifts in production ML systems and autonomously responding with shadow retraining and zero-downtime model updates. Implements the "Self-Healing Model" concept where concept drift triggers an end-to-end recovery cycle without manual intervention.

Concept Drift Taxonomy

Drift Type	Description	Detection Method
Sudden drift	Abrupt change at a specific time point (e.g., system change, world event)	Immediate KS test comparison
Gradual drift	Slow, continuous shift (e.g., user behavior evolution)	Rolling window PSI trend
Recurrent drift	Seasonal or cyclic patterns reappear (e.g., holiday behavior)	Season-aware baseline comparison
Incremental drift	Monotonic directional drift over time	Linear trend test on PSI
Feature drift	Input distribution changes but relationship to target is stable	Input-only KS/PSI (re-calibrate, don't retrain)
Label drift	Relationship between features and target changes (true concept drift)	Requires label monitoring + performance tracking

Statistical Tests

Kolmogorov-Smirnov (KS) Test for continuous features:

• D = max|F_ref(x) − F_current(x)| (maximum CDF difference)
• p-value < 0.05 → statistically significant drift
• Sensitive to any shape of distribution change (shift, scale, shape)

Chi-Square Test for categorical features:

• χ² = Σ (O_i − E_i)² / E_i
• O_i = observed frequency in current period; E_i = expected frequency from reference
• df = k - 1 (k = number of categories); p-value < 0.05 → drift

Population Stability Index (PSI) for magnitude quantification:

• PSI = Σ (P_current_i − P_reference_i) · ln(P_current_i / P_reference_i)
• PSI < 0.1: negligible shift — no action
• 0.1 ≤ PSI < 0.2: moderate shift — increase monitoring frequency, alert team
• PSI ≥ 0.2: major shift — trigger retraining pipeline

Drift aggregation rule (avoids false positives from individual feature noise):

• Trigger retraining only when: ≥ 3 features have KS p < 0.05 AND ≥ 1 feature has PSI ≥ 0.2, OR performance metric degrades > 5% relative.

See references/drift-detection.md for complete test implementations, window size selection, and rolling-window monitoring patterns.

Self-Healing Pipeline Sequence

1. Drift Detected (KS + PSI thresholds exceeded)
          ↓
2. Shadow Training Triggered (new data distribution)
          ↓
3. Challenger Model Evaluated vs. Champion
          ↓
4. Statistical Significance Test (paired t-test, p < 0.05)
          ↓
5. Blue-Green Deployment (10% canary → 100% cutover)
          ↓
6. Champion model archived (24-hour rollback window)

Shadow Training

Shadow training runs the full training pipeline on the drifted data distribution without disrupting the production model:

• Isolate: use the current data window (typically last 30–90 days) as the new training distribution
• Architecture: reuse the same architecture unless drift analysis suggests a fundamentally different data signature
• HPO: reuse the last best hyperparameters; re-run HPO only if drift severity is "critical" (PSI ≥ 0.5)
• Parallelism: shadow model trains on a separate compute partition — no production impact
• Hardware: prefer spot instances for cost efficiency; configure checkpointing for preemption resilience

Champion-Challenger Evaluation

Before any deployment, the challenger must beat the champion on the current data distribution:

1. Set aside 20% of the current data window as a holdout evaluation set (after shadow training).
2. Evaluate both champion and challenger on the holdout set.
3. Run a paired t-test (or permutation test) on per-sample losses:
   • H₀: challenger performance = champion performance
   • Reject H₀ if p < 0.05 → challenger is statistically better
4. Require: challenger primary metric improvement ≥ 2% relative AND p < 0.05.
5. Check all quality gates: latency SLA, fairness thresholds, no regression on reference distribution.

See references/deployment-strategies.md for blue-green, canary, and shadow traffic routing patterns with Kubernetes ingress configurations.

Rollback Protocol

If post-deployment monitoring detects regression within 24 hours:

1. Immediately route 100% traffic back to the previous champion (now in "green" slot).
2. Log the regression with full diagnostics: which metrics degraded, by how much.
3. File an incident report and increase shadow training data window (e.g., 30 → 90 days).
4. Re-evaluate whether drift was correctly classified (feature drift vs. true label drift).