mlops

name	mlops
description	Design and implement ML operations — model registry, serving patterns, deployment strategies (shadow/canary/blue-green), drift detection, feature stores, retraining triggers, and prediction monitoring. Use when asked to "deploy a model", "model registry", "MLflow", "feature store", "drift detection", "retrain trigger", "shadow mode", "model versioning", "serving infrastructure", or "ML pipeline". Do NOT use for: prompt engineering or RAG pipelines — see prompt-engineering and rag-architect skills. Do NOT use for: general API deployment without an ML component.
origin	yamtam-original
license	MIT © 2026 Vũ Văn Tâm
version	1.0.0
compatibility	MLflow ≥ 2.x, BentoML ≥ 1.x, Ray Serve ≥ 2.x. Patterns are framework-agnostic.

When to Use

• Use when: shipping a trained model as a production API endpoint
• Use when: tracking experiments and promoting models to production
• Use when: detecting model quality degradation without ground-truth labels
• Use when: building a retraining pipeline triggered by drift or schedule
• Do NOT use for: prompt/RAG pipelines — those are LLM patterns, not classical MLOps
• Do NOT use for: data engineering ETL — model serving starts after features are ready

---

Model Registry

A registry gives every model artifact a version, status, and audit trail.

import mlflow

# Log a training run
with mlflow.start_run():
    mlflow.log_params({"learning_rate": 0.01, "n_estimators": 100})
    mlflow.log_metrics({"auc": 0.94, "f1": 0.87})
    mlflow.sklearn.log_model(model, "model", registered_model_name="churn-predictor")

# Promote to production after review
client = mlflow.MlflowClient()
client.transition_model_version_stage(
    name="churn-predictor", version="3", stage="Production"
)
# Stages: None → Staging → Production → Archived

Never deploy directly from a training run — always register, validate in staging, then promote.

---

Serving Patterns

Pattern	Use when	Latency	Throughput
REST endpoint	Single predictions, interactive	< 100 ms	Medium
Batch inference	Scoring millions of rows offline	N/A	High
Streaming inference	Kafka/Kinesis event scoring	< 500 ms	High
Shadow mode	Validate new model against live traffic	Async	—

# BentoML REST endpoint
import bentoml
from bentoml.io import JSON

runner = bentoml.mlflow.get("churn-predictor:production").to_runner()
svc = bentoml.Service("churn-api", runners=[runner])

@svc.api(input=JSON(), output=JSON())
async def predict(input_data: dict) -> dict:
    features = extract_features(input_data)
    score = await runner.predict.async_run([features])
    return {"churn_probability": float(score[0])}

---

Deployment Strategies

Shadow mode   → new model receives same traffic, results discarded
               → compare predictions vs champion offline
               → zero user risk, full production signal

Canary        → new model serves 5% of traffic
               → monitor error rate + business metric
               → ramp: 5% → 20% → 50% → 100%

Blue/green    → two identical stacks, instant cutover via load balancer
               → instant rollback by flipping DNS/LB
               → costs 2× infra during transition

Default path: shadow → canary → full rollout. Skip shadow only if model is a minor retrain with identical feature schema.

---

Drift Detection

Drift = distribution of inputs or outputs diverges from training distribution.

Type	What shifts	Detection method
Data drift	Input feature distribution	PSI, KS test, Jensen-Shannon divergence
Concept drift	Relationship between X and y	Monitor prediction accuracy on labeled sample
Prediction drift	Output distribution	Score distribution histogram shift

# PSI (Population Stability Index) — industry standard for data drift
def psi(expected, actual, buckets=10):
    e, _ = np.histogram(expected, bins=buckets, density=True)
    a, _ = np.histogram(actual, bins=np.histogram(expected, bins=buckets)[1], density=True)
    e = np.clip(e, 1e-6, None); a = np.clip(a, 1e-6, None)
    return np.sum((a - e) * np.log(a / e))

# PSI < 0.1: no drift. 0.1–0.25: moderate. > 0.25: retrain.

Run drift checks on a daily sample — not on every prediction (too expensive).

---

Feature Store

Solves training-serving skew — training and serving must compute features identically.

Offline store  → historical features for training (S3, BigQuery, Delta Lake)
Online store   → low-latency feature lookup for serving (Redis, DynamoDB)
Feature server → materializes offline → online on a schedule

Request time:
  features = feature_store.get_online_features(
      feature_refs=["user:age", "user:churn_score_30d"],
      entity_rows=[{"user_id": user_id}]
  )

If you can't afford a feature store (Feast, Tecton, Hopsworks): at minimum, share the same feature-computation code between training and serving via a Python package — no copy-paste.

---

Retraining Triggers

Trigger	When to use
Scheduled	Stable distribution, weekly/monthly cadence
Drift threshold	PSI > 0.25 on key features
Performance degradation	Accuracy drops below SLO on labeled holdout
Data volume	New data volume exceeds X% of training set

# Example Airflow DAG trigger
retrain_dag:
  schedule: "@weekly"
  on_failure_callback: alert_slack
  tasks:
    - check_drift        # abort if PSI < 0.1 (no drift → no need)
    - run_training
    - evaluate_vs_champion  # only promote if new model beats current
    - promote_to_staging
    - notify_ml_team

Never auto-promote to production — a human or automated gate must compare champion vs challenger.

---

Prediction Monitoring

Log every prediction with enough context to debug incidents and detect drift.

# Minimum prediction log schema
{
  "request_id":   str,   # for deduplication
  "model_name":   str,
  "model_version": str,
  "timestamp":    ISO8601,
  "features":     dict,  # input snapshot — enables drift recomputation
  "prediction":   float | int | str,
  "latency_ms":   int,
  "ground_truth": None   # filled in later via feedback loop
}

Store prediction logs separately from app logs — they need long retention for drift analysis.

---

Anti-Fake-Pass Rules

Before claiming an ML deployment is done, you MUST show:

• Model registered with version + stage — not deployed from a raw artifact path
• Serving schema validated — feature names/types match training schema exactly
• Shadow or canary deployed first — not straight to 100% traffic
• Drift detection scheduled — PSI or equivalent running on daily sample
• Retraining trigger defined — schedule or drift threshold, not manual-only
• Prediction logs captured — include features snapshot, model version, latency
• Rollback path tested — old model version can be restored without a redeploy

Reference: gates/anti-fake-pass-gate.md