Use this skill when deploying ML models to production, setting up model monitoring, implementing A/B testing for models, or managing feature stores. Triggers on model deployment, model serving, ML pipelines, feature engineering, model versioning, data drift detection, model registry, experiment tracking, and any task requiring machine learning operations infrastructure.
Use this skill when deploying ML models to production, setting up model monitoring, implementing A/B testing for models, or managing feature stores. Triggers on model deployment, model serving, ML pipelines, feature engineering, model versioning, data drift detection, model registry, experiment tracking, and any task requiring machine learning operations infrastructure.
When this skill is activated, always start your first response with the 🧢 emoji.
ML Ops
A production engineering framework for the full machine learning lifecycle. MLOps
bridges the gap between model experimentation and reliable production systems by
applying software engineering discipline to ML workloads. This skill covers model
deployment strategies, experiment tracking, feature stores, drift monitoring, A/B
testing, and versioning - the infrastructure that makes models trustworthy over time.
Think of it as DevOps for models: automate everything, measure what matters, and
treat reproducibility as a first-class constraint.
When to use this skill
Trigger this skill when the user:
Deploys a trained model to a production serving endpoint
Sets up experiment tracking for training runs (parameters, metrics, artifacts)
Implements canary or shadow deployments for a new model version
Designs or integrates a feature store for online/offline feature serving
Sets up monitoring for data drift, prediction drift, or model degradation
Runs A/B or champion/challenger tests across model versions in production
Versions models, datasets, or pipelines with DVC or a model registry
Builds or migrates to an automated training/retraining pipeline
Do NOT trigger this skill for:
Core model research, architecture design, or hyperparameter search (use an ML
research skill instead - MLOps starts after a candidate model exists)
General software observability (logs, metrics, traces for non-ML services - use
the backend-engineering skill)
Key principles
Reproducibility is non-negotiable - Every training run must be reproducible
from scratch: fixed seeds, pinned dependency versions, tracked data splits, and
logged hyperparameters. If you cannot reproduce a model, you cannot debug it,
audit it, or roll back to it safely.
Automate the training pipeline - Manual training is a one-way door to
undocumented models. Build an automated pipeline (data ingestion -> preprocessing
-> training -> evaluation -> registration) from day one. Humans should only
approve a model for promotion, not run the steps.
Monitor data, not just models - Model metrics degrade because the input data
changes. Track feature distributions in production against training baselines.
Data drift is usually the root cause; model drift is the symptom.
Version everything - Models, datasets, feature definitions, pipeline code,
and environment configs all deserve version control. An unversioned artifact is
a liability. Use DVC for data/models, a model registry for lifecycle state, and
git for code.
Treat ML code like production code - Tests, code review, CI/CD, and on-call
rotation apply to training pipelines and serving code. The "it works in the
notebook" standard is not a production standard.
Core concepts
ML lifecycle describes the end-to-end journey of a model:
Each stage has gates: an experiment produces a candidate; training on full data
with tracked params produces an artifact; validation gates on held-out metrics;
deployment chooses a serving strategy; monitoring decides when retraining is needed.
Model registry is the source of truth for model lifecycle state. A model moves
through stages: Staging -> Production -> Archived. The registry stores metadata,
metrics, lineage, and the artifact URI. MLflow Model Registry, Vertex AI Model
Registry, and SageMaker Model Registry are the main options.
Feature stores decouple feature computation from model training and serving.
They have two serving paths: an offline store (columnar, batch-oriented,
used for training and batch inference) and an online store (low-latency key-value
lookup, used at prediction time). The critical guarantee is point-in-time
correctness - training features must only use data available before the label
timestamp to prevent target leakage.
Data drift occurs when the statistical distribution of input features in
production diverges from the training distribution. Concept drift occurs when
the relationship between features and labels changes even if feature distributions
are stable (e.g., user behavior shifts after a product change).
Shadow deployment runs the new model in parallel with the live model, receiving
the same traffic, but its predictions are not served to users. Used to compare
behavior before any real traffic exposure.
Common tasks
Design an ML pipeline
Structure pipelines as discrete, testable stages with explicit inputs/outputs:
Data ingestion -> Validation -> Preprocessing -> Training -> Evaluation -> Registration
| | | | |
raw data schema check feature eng model go/no-go
versioned + stats artifact artifact gate
Orchestration choices:
Need
Tool
Python-native, simple DAGs
Prefect, Apache Airflow
Kubernetes-native, reproducible
Kubeflow Pipelines, Argo Workflows
Managed, minimal infra
Vertex AI Pipelines, SageMaker Pipelines
Git-driven, code-first
ZenML, Metaflow
Gate evaluation: define a go/no-go threshold before training starts. A model that
does not beat baseline (or the current production model) should never reach
the registry.
Set up experiment tracking
Track every training run with: parameters (hyperparams, data version), metrics
(loss curves, eval metrics), artifacts (model weights, plots), and environment
(library versions, hardware).
Define rollback triggers before deploying: error rate > X%, prediction latency
p99 > Y ms, or business metric (e.g., conversion rate) drops > Z%.
Implement model monitoring
Monitor three layers - input data, predictions, and business outcomes:
Layer
Signal
Method
Input data
Feature distribution drift
PSI, KS test, chi-squared
Predictions
Output distribution drift
PSI on prediction histogram
Business outcome
Actual vs expected labels
Delayed feedback loop
Population Stability Index (PSI) thresholds:
PSI < 0.1 -> No significant change, model stable
PSI 0.1-0.2 -> Moderate drift, investigate
PSI > 0.2 -> Significant drift, retrain or escalate
Monitoring setup pattern:
# On each prediction batch, compute and log feature stats
baseline_stats = load_training_stats() # saved during training
production_stats = compute_stats(current_batch_features)
for feature in monitored_features:
psi = compute_psi(baseline_stats[feature], production_stats[feature])
metrics.gauge(f"drift.psi.{feature}", psi)
if psi > 0.2:
alert(f"Significant drift on feature: {feature}")
Set up scheduled monitoring jobs (hourly/daily depending on traffic volume) rather
than per-prediction to avoid overhead. Load the references/tool-landscape.md for
monitoring platform options.
Build a feature store
Separate feature computation from model code to enable reuse and prevent leakage.
Architecture:
Raw data sources
|
Feature computation (Spark, dbt, Flink)
|
+-----------> Offline store (Parquet/BigQuery) -> Training jobs
|
+-----------> Online store (Redis, DynamoDB) -> Real-time serving
Point-in-time correctness - the most critical correctness property:
# WRONG: uses future data at training time (target leakage)
features = feature_store.get_features(entity_id=user_id)
# CORRECT: fetch features as they existed at the event timestamp
features = feature_store.get_historical_features(
entity_df=events_df, # includes entity_id + event_timestamp
feature_refs=["user:age", "user:30d_spend", "user:country"]
)
Feature naming convention:<entity>:<feature_name> (e.g., user:30d_spend,
product:avg_rating_7d). Version feature definitions in a registry (Feast, Tecton,
Vertex Feature Store). Never hardcode feature transformations in training scripts.
A/B test models in production
A/B testing models requires statistical rigor. A "better offline metric" does not
guarantee better business outcomes.
Setup:
Define the primary metric (business metric, not model metric) and a guardrail
metric before the test
Calculate required sample size for desired power (typically 80%) and significance
level (typically 5%)
Randomly assign users/sessions to treatment/control - sticky assignment (same
user always gets the same model) prevents contamination
Run for full business cycles (minimum 1-2 weeks for weekly seasonality)
Stopping criteria: Do not peek at p-values daily. Pre-register the minimum
runtime and only stop early for clearly harmful outcomes (guardrail breach). Use
sequential testing methods (mSPRT) if early stopping is required by business needs.
A model that improves AUC by 2% but reduces revenue is not a better model.
Always tie model tests to business metrics.
Version models and datasets
Dataset versioning with DVC:
# Track a dataset in DVC
dvc add data/training/users_2024q1.parquet
git add data/training/users_2024q1.parquet.dvc .gitignore
git commit -m "Track Q1 2024 training dataset"# Push dataset to remote storage
dvc push
# Reproduce dataset at a specific git commit
git checkout <commit-hash>
dvc pull
Model registry lifecycle:
Training pipeline produces artifact
-> Registers as version N in "Staging"
-> QA + validation passes
-> Promoted to "Production" (previous Production -> "Archived")
-> On rollback: restore previous version from "Archived"
Lineage tracking: A model version should link to: the training dataset version,
the pipeline code commit, the feature definitions version, and the evaluation report.
Without lineage, auditing and debugging become guesswork.
Anti-patterns / common mistakes
Mistake
Why it's wrong
What to do instead
Training and serving skew
Features computed differently at train vs serve time - silent accuracy loss
Share feature computation code; use a feature store for consistency
No baseline comparison
Deploying a new model without comparing to the current production model or a simple baseline
Always register the current production model as the benchmark; gate on relative improvement
Testing on test data during development
Inflated metrics, model does not generalize; test set is contaminated
Use train/validation/test splits; touch test set only for final reporting
Monitoring only model metrics, not inputs
Drift in input data causes silent degradation - you notice it in business metrics weeks later
Monitor feature distributions against training baseline as a first-class signal
Manual deployment steps
Undocumented, unrepeatable process; impossible to roll back reliably
Automate the full promote-to-production flow in CI/CD; humans approve, machines execute
A/B testing without sufficient sample size
Statistically underpowered tests produce false positives; teams ship regressions confidently
Calculate sample size upfront using power analysis; commit to minimum runtime before launch
Gotchas
Training-serving skew is silent and deadly - If the feature engineering code that runs during training differs even slightly from what runs at inference time (different library versions, different null handling, different normalization order), the model receives inputs it was never trained on. The model silently produces worse predictions. Share the exact same feature transformation code between training and serving; a feature store enforces this by design.
PSI drift alerts fire on expected seasonal changes, not just real drift - A retail model will always show PSI > 0.2 on Black Friday vs. a July training baseline. Alerting on raw PSI without seasonality context produces alert fatigue and trains teams to ignore drift signals. Baseline your monitoring against the same calendar period from the prior year, or use rolling baselines updated monthly.
DVC pull on a different machine requires remote storage credentials - dvc pull fetches data from the configured remote (S3, GCS, Azure). A teammate who clones the repo and runs dvc pull without configuring remote credentials gets a cryptic access-denied error that looks like a DVC bug. Document remote storage setup in the repo's README and use environment-based credential configuration.
MLflow autologging captures too much and inflates experiment storage - mlflow.autolog() is convenient for notebooks but logs every parameter, metric, and artifact from every library it supports. In training pipelines running thousands of experiments, this creates massive metadata storage and slow UI queries. Enable autologging selectively with mlflow.sklearn.autolog(log_models=False) or log manually with mlflow.log_params/metrics.
A/B tests on models need sticky user assignment, not session assignment - If a user is randomly assigned to the control or treatment model on each request, they experience inconsistent behavior within the same session. This contaminates the experiment (users implicitly see both models) and inflates variance. Hash on user ID to ensure consistent model assignment for the duration of the experiment.
References
For detailed platform comparisons and tool selection guidance, read the relevant
file from the references/ folder:
references/tool-landscape.md - MLflow vs W&B vs Vertex AI vs SageMaker,
feature store comparison, model serving options
Load references/tool-landscape.md when the task involves selecting or comparing
MLOps platforms - it is detailed and will consume context, so only load it when
needed.
Companion check
On first activation of this skill in a conversation: check which companion skills are installed by running ls ~/.claude/skills/ ~/.agent/skills/ ~/.agents/skills/ .claude/skills/ .agent/skills/ .agents/skills/ 2>/dev/null. Compare the results against the recommended_skills field in this file's frontmatter. For any that are missing, mention them once and offer to install:
