serve-deploy-ml-model

name	serve-deploy-ml-model
description	Deploys a trained ML model to production — packaging it with the identical training-time preprocessing, registering a versioned model+code+data triple, serving via batch or online REST/gRPC behind a runtime (BentoML/TorchServe/Triton/ONNX), with autoscaling/warmup and canary/shadow rollout — so served predictions reproducibly match offline scoring.
when_to_use	Taking a trained model to production to generate predictions (package, register, serve, scale, roll out). Distinct from train-evaluate-ml-model (building/evaluating the model), monitor-ml-drift (post-deployment drift/quality monitoring), and deploy-release (generic application deploys with no model artifact).

When to Use

Reach for this skill when a working model needs to serve predictions in production, not when it's still being built:

• "Deploy this model so the app/service can call it"
• "Stand up a REST/gRPC inference endpoint for model.pkl/model.pt"
• "Run nightly batch scoring over the warehouse table"
• "Roll the new model out behind the old one (shadow/canary) before cutting over"
• "Our predictions in prod don't match what we got in the notebook" (train/serve skew)
• "Speed up / scale the inference service" (ONNX export, autoscaling, warmup)

NOT this skill:

• Training, hyperparameter search, offline metrics, choosing the model → train-evaluate-ml-model
• Watching the live model for input drift, label delay, quality decay, alerting → monitor-ml-drift
• Shipping a normal app/service with no model artifact (web app, API, worker) → deploy-release
• Percentage ramps, kill switches, sticky bucketing for any change → feature-flags-rollout (this skill uses it for the model rollout)
• Latency/cost tuning of an LLM prompt/provider path → optimize-llm-cost-latency

Steps

1. Package the model WITH its exact preprocessing — this is the #1 cause of train/serve skew. The artifact must contain the same feature/transform code that produced training inputs, not a reimplementation. Fit transforms on train data, serialize the fitted objects, and apply the identical pipeline at serve time.

   # train.py — ONE fitted pipeline = preprocessing + model, saved as a unit
   from sklearn.pipeline import Pipeline
   from sklearn.compose import ColumnTransformer
   import mlflow, mlflow.sklearn
   
   pipe = Pipeline([("prep", ColumnTransformer(...)), ("model", clf)]).fit(X_tr, y_tr)
   
   with mlflow.start_run():
       mlflow.sklearn.log_model(
           pipe, "model",
           registered_model_name="churn",
           input_example=X_tr.iloc[:5],          # captures schema + dtypes
           signature=mlflow.models.infer_signature(X_tr, pipe.predict(X_tr)),
           pip_requirements="requirements.lock",  # pinned, == training env
       )
   

   Rules: never re-derive features in the serving codebase; serve the fitted prep+model as one object. For deep nets, save the transform graph (e.g. torchvision/torchaudio transforms or a tf.function preprocessing layer) inside the exported module so the runtime applies it. Stateful features (counts, embeddings, aggregates) computed from a feature store at train time must be read from the same store online — recomputing them in app code drifts.

2. Register and pin model + code + data together. A model version is meaningless without the code and data snapshot that produced it. Push to a registry (MLflow Model Registry, SageMaker, Vertex, or a tagged OCI artifact) and record, in the run/version metadata: git SHA, training-data version/hash (DVC/Delta/snapshot id), and the locked dependency file. Use registry stages (Staging → Production) or aliases; deploy by immutable version, never "latest".

3. Pick the serving pattern by latency need — decide, don't hedge.

   Pattern	Use when	Interface	Default runtime
   Batch / offline	No realtime need; score a table/file on a schedule	Job writes predictions to warehouse/S3	Spark / Ray / a plain container in cron/Airflow
   Online (sync)	A user request blocks on the prediction; p99 budget < ~200 ms	REST (simple, debuggable) default; gRPC when p99 < 20 ms or high QPS	BentoML / TorchServe / Triton
   Streaming	React to an event flow (clicks, transactions) continuously	Consume Kafka/Kinesis → predict → emit	Flink / Faust / a Ray Serve consumer

   Defaults: batch unless something blocks on the result — it's cheaper, simpler, and trivially reproducible. For online, start with REST + JSON and only move to gRPC/protobuf when a measured latency budget forces it. Do not build an online endpoint for a nightly report.

4. Choose the runtime; export to ONNX/TensorRT only when you need the speed. Server defaults: BentoML (Python-first, easy custom logic, batching) for most teams; Triton for multi-framework, GPU, dynamic batching at scale; TorchServe for pure PyTorch shops. Convert to ONNX Runtime (CPU) or TensorRT (GPU) when profiling shows the framework runtime is the bottleneck — and re-verify outputs match the original within tolerance (atol≈1e-4) before trusting it; quantization/op-set changes silently alter predictions.

   # bento service.py — load a PINNED model version (never "latest"), server-side batching
   import bentoml
   from bentoml.io import JSON
   runner = bentoml.mlflow.get("churn:prod").to_runner()   # alias -> immutable version; never churn:latest
   svc = bentoml.Service("churn", runners=[runner])
   
   @svc.api(input=JSON(), output=JSON())   # set batchable=True + max_batch_size on the runner config for throughput
   async def predict(rows: list[dict]) -> list[dict]:
       return await runner.predict.async_run(rows)
   

5. Add warmup, resource limits, and autoscaling — in that order. Cold models cause p99 spikes: run a synthetic prediction at startup (load weights, JIT/CUDA-warm, fill caches) and gate the readiness probe on it so traffic only arrives warm. Set CPU/memory/GPU requests and limits from a load test (see load-stress-test), not by guessing. Autoscale on the right signal — request concurrency / queue depth / GPU util, not CPU% for GPU models — with minReplicas ≥ 2 (no cold-start on scale-from-zero for latency-critical paths) and a scale-down stabilization window so it doesn't flap. Pin threads (OMP_NUM_THREADS) to avoid oversubscription under the container limit.

6. Roll out shadow → canary against the current model; keep an instant rollback. Never hard-cut. Shadow first: mirror live traffic to the new version, log its predictions, serve the old model's response to users — compares behavior on real traffic at zero user risk. Then canary: route 1% → 10% → 50% → 100% by sticky hashed bucketing, watching guardrail metrics (latency, error rate, and prediction distribution vs the incumbent); auto-halt and revert on breach. Drive the ramp/kill switch with feature-flags-rollout. Rollback = repoint the alias/route to the previous registered version (still deployed) — must be one command, seconds, no rebuild.

7. Lock inference reproducibility end to end. Serve from the locked requirements captured at registration (same library versions, same op-set), pin the base image by digest, set seeds where any stochasticity exists, and freeze the feature-store read path. The contract: the same input row produces a bit-identical (or within-tolerance) prediction in the notebook, the batch job, and the online endpoint.

Common Errors

• Reimplementing preprocessing in the serving code. The serving normalizer/encoder/tokenizer drifts from the training one → skew. Serialize and serve the fitted pipeline as one artifact; never rewrite the transforms.
• Fitting a transform at serve time (e.g. StandardScaler().fit(request_batch), or imputing with the request's own mean). Must use stats fitted on training data, frozen in the artifact.
• Deploying "latest"/an unpinned stage. A retrain silently swaps the model under prod. Deploy an immutable version id; promote via alias (churn:prod), not churn:latest.
• Env mismatch between train and serve. Different numpy/sklearn/torch/CUDA or ONNX op-set changes outputs. Serve from the exact locked requirements; pin the image by digest.
• ONNX/TensorRT export assumed equivalent. Quantization, fused ops, or op-set bumps shift predictions. Always diff converted vs original outputs on a fixed sample before shipping.
• No warmup → readiness flaps. First requests hit an unloaded/un-JIT'd model and time out; the cold pod is added to the pool before it can serve. Warm at startup and gate readiness on it.
• Online endpoint for a batch problem. Standing up a low-latency REST service to score a table on a schedule wastes cost and adds failure modes. Use a batch job.
• Hard cutover with no shadow/canary. A skew or perf regression hits 100% of traffic instantly. Shadow, then ramp, with auto-rollback.
• Single replica / scale-to-zero on a latency path. Any restart or scale event becomes a user-visible cold start. Keep minReplicas ≥ 2.
• Autoscaling GPU models on CPU%. CPU sits low while the GPU saturates → it never scales and latency explodes. Scale on concurrency/queue depth/GPU util.
• Stateful features recomputed in app code. Online aggregates/counts computed differently from the training feature store drift per request. Read from the same store.
• No rollback artifact. The previous version was torn down, so "revert" means a rebuild. Keep the prior registered version deployed and one alias-flip away.

Verify

1. Parity (the skew gate): Take a fixed holdout sample, score it three ways — training notebook, the batch job, and the online endpoint — and assert predictions match within tolerance (exact for classification labels; atol≤1e-4 for probabilities/regression). Any mismatch blocks the deploy. This is the single most important check.
2. ONNX/quantized parity: If exported, diff converted-runtime outputs vs the original framework on the same sample within tolerance.
3. Schema/contract: Send a malformed/missing-field request → a clean 4xx, not a 500 or a silently wrong prediction. The logged input signature matches the registered one.
4. Latency/throughput: Under the target arrival rate (load-stress-test), p95/p99 and sustained QPS meet the documented SLO with warmup applied — measure warm, not cold.
5. Warmup/readiness: A freshly started replica reports ready only after a successful synthetic prediction; first real request is not a cold spike.
6. Autoscaling: Drive load past the per-replica knee → replicas scale up on the chosen signal and back down after the stabilization window; minReplicas is honored at idle.
7. Shadow: New version receives mirrored traffic and logs predictions while users still get the incumbent's response; their distributions are comparable before any canary.
8. Rollback: Flip the alias to the previous version and confirm traffic serves the old model within seconds, no rebuild.
9. Reproducibility pin: The deployed image digest, model version, training-data hash, and git SHA are all recorded together and resolvable from the running service.

Done = served predictions match offline scoring on the fixed sample within tolerance, latency/throughput meet the SLO warm, shadow/canary ran with guardrails, and a one-command rollback to the prior registered version is proven.