serve-deploy-ml-model

name	serve-deploy-ml-model
description	Deploys a trained ML model to production — packaging it with the identical training-time preprocessing, registering a versioned model+code+data triple, serving via batch or online REST/gRPC behind a runtime (BentoML/TorchServe/Triton/ONNX), with autoscaling/warmup and canary/shadow rollout — so served predictions reproducibly match offline scoring.
when_to_use	Taking a trained model to production to generate predictions (package, register, serve, scale, roll out). Distinct from train-evaluate-ml-model (building/evaluating the model), monitor-ml-drift (post-deployment drift/quality monitoring), and deploy-release (generic application deploys with no model artifact).

When to Use

Reach for this skill when a working model needs to serve predictions in production, not when it's still being built:

• "Deploy this model so the app/service can call it"
• "Stand up a REST/gRPC inference endpoint for model.pkl/model.pt"
• "Run nightly batch scoring over the warehouse table"
• "Roll the new model out behind the old one (shadow/canary) before cutting over"
• "Our predictions in prod don't match what we got in the notebook" (train/serve skew)
• "Speed up / scale the inference service" (ONNX export, autoscaling, warmup)

NOT this skill:

• Training, hyperparameter search, offline metrics, choosing the model → train-evaluate-ml-model
• Watching the live model for input drift, label delay, quality decay, alerting → monitor-ml-drift
• Shipping a normal app/service with no model artifact (web app, API, worker) → deploy-release
• Percentage ramps, kill switches, sticky bucketing for any change → feature-flags-rollout (this skill uses it for the model rollout)
• Latency/cost tuning of an LLM prompt/provider path → optimize-llm-cost-latency

Steps

1. Package the model WITH its exact preprocessing — this is the #1 cause of train/serve skew. The artifact must contain the same feature/transform code that produced training inputs, not a reimplementation. Fit transforms on train data, serialize the fitted objects, and apply the identical pipeline at serve time.

   # train.py — ONE fitted pipeline = preprocessing + model, saved as a unit
   from sklearn.pipeline import Pipeline
   from sklearn.compose import ColumnTransformer
   import mlflow, mlflow.sklearn
   
   pipe = Pipeline([("prep", ColumnTransformer(...)), ("model", clf)]).fit(X_tr, y_tr)
   
   with mlflow.start_run():
       mlflow.sklearn.log_model(
           pipe, "model",
           registered_model_name="churn",
           input_example=X_tr.iloc[:5],          # captures schema + dtypes
           signature=mlflow.models.infer_signature(X_tr, pipe.predict(X_tr)),
           pip_requirements="requirements.lock",  # pinned, == training env
       )
   

   Rules: never re-derive features in the serving codebase; serve the fitted prep+model as one object. For deep nets, save the transform graph (e.g. torchvision/torchaudio transforms or a tf.function preprocessing layer) inside the exported module so the runtime applies it. Stateful features (counts, embeddings, aggregates) computed from a feature store at train time must be read from the same store online — recomputing them in app code drifts.

2. Register and pin model + code + data together. A model version is meaningless without the code and data snapshot that produced it. Push to a registry (MLflow Model Registry, SageMaker, Vertex, or a tagged OCI artifact) and record, in the run/version metadata: git SHA, training-data version/hash (DVC/Delta/snapshot id), and the locked dependency file. Use registry stages (Staging → Production) or aliases; deploy by immutable version, never "latest".

3. Pick the serving pattern by latency need — decide, don't hedge.

Common Errors

• Reimplementing preprocessing in the serving code. The serving normalizer/encoder/tokenizer drifts from the training one → skew. Serialize and serve the fitted pipeline as one artifact; never rewrite the transforms.
• Fitting a transform at serve time (e.g. StandardScaler().fit(request_batch), or imputing with the request's own mean). Must use stats fitted on training data, frozen in the artifact.
• Deploying "latest"/an unpinned stage. A retrain silently swaps the model under prod. Deploy an immutable version id; promote via alias (churn:prod), not churn:latest.
• Env mismatch between train and serve. Different numpy/sklearn/torch/CUDA or ONNX op-set changes outputs. Serve from the exact locked requirements; pin the image by digest.
• ONNX/TensorRT export assumed equivalent. Quantization, fused ops, or op-set bumps shift predictions. Always diff converted vs original outputs on a fixed sample before shipping.
• No warmup → readiness flaps. First requests hit an unloaded/un-JIT'd model and time out; the cold pod is added to the pool before it can serve. Warm at startup and gate readiness on it.
• Online endpoint for a batch problem. Standing up a low-latency REST service to score a table on a schedule wastes cost and adds failure modes. Use a batch job.
• Hard cutover with no shadow/canary. A skew or perf regression hits 100% of traffic instantly. Shadow, then ramp, with auto-rollback.
• Single replica / scale-to-zero on a latency path. Any restart or scale event becomes a user-visible cold start. Keep minReplicas ≥ 2.
• Autoscaling GPU models on CPU%. CPU sits low while the GPU saturates → it never scales and latency explodes. Scale on concurrency/queue depth/GPU util.
• Stateful features recomputed in app code. Online aggregates/counts computed differently from the training feature store drift per request. Read from the same store.
• No rollback artifact. The previous version was torn down, so "revert" means a rebuild. Keep the prior registered version deployed and one alias-flip away.

Verify

1. Parity (the skew gate): Take a fixed holdout sample, score it three ways — training notebook, the batch job, and the online endpoint — and assert predictions match within tolerance (exact for classification labels; atol≤1e-4 for probabilities/regression). Any mismatch blocks the deploy. This is the single most important check.
2. ONNX/quantized parity: If exported, diff converted-runtime outputs vs the original framework on the same sample within tolerance.
3. Schema/contract: Send a malformed/missing-field request → a clean 4xx, not a 500 or a silently wrong prediction. The logged input signature matches the registered one.
4. Latency/throughput: Under the target arrival rate (load-stress-test), p95/p99 and sustained QPS meet the documented SLO with warmup applied — measure warm, not cold.
5. Warmup/readiness: A freshly started replica reports ready only after a successful synthetic prediction; first real request is not a cold spike.
6. Autoscaling: Drive load past the per-replica knee → replicas scale up on the chosen signal and back down after the stabilization window; minReplicas is honored at idle.
7. Shadow: New version receives mirrored traffic and logs predictions while users still get the incumbent's response; their distributions are comparable before any canary.
8. Rollback: Flip the alias to the previous version and confirm traffic serves the old model within seconds, no rebuild.
9. Reproducibility pin: The deployed image digest, model version, training-data hash, and git SHA are all recorded together and resolvable from the running service.

Done = served predictions match offline scoring on the fixed sample within tolerance, latency/throughput meet the SLO warm, shadow/canary ran with guardrails, and a one-command rollback to the prior registered version is proven.