ml-model-deployment-checklist

# ML Model Deployment Checklist ## Phase 1: Model Validation 1. Validate model before deployment - [ ] Model performance metrics meet acceptance criteria - [ ] Evaluated on held-out test set (not training data) - [ ] Tested on edge cases and failure modes - [ ] Bias and fairness metrics reviewed - [ ] Model size and inference latency within requirements - [ ] Comparison with current production model (if exists) - [ ] Model card/documentation completed 2. Get model review sign-off from stakeholders ### Model Performance Summary | Metric | Training | Validation | Test | Production Baseline | Threshold | |--------|---------|-----------|------|-------------------|-----------| | Primary metric | | | | | > | | Secondary metric | | | | | > | | Latency (P50) | N/A | N/A | ms | ms | < ms | | Latency (P99) | N/A | N/A | ms | ms | < ms | | Model size | MB | N/A | N/A | MB | < MB | ## Phase 2: Infrastructure Setup 1. Prepare serving infrastructure - [ ] Compute resources provisioned (GPU/CPU, memory) - [ ] Auto-scaling configured (min/max replicas) - [ ] Load balancer and health checks configured - [ ] Model artifact stored in versioned registry - [ ] Feature store connected (if real-time features needed) - [ ] Caching layer configured (if applicable) - [ ] Network security (API authentication, rate limiting) 2. Validate infrastructure handles expected load ### Resource Configuration | Resource | Specification | Auto-Scale Min | Auto-Scale Max | Cost/hour | |----------|-------------|----------------|----------------|-----------| | Compute | | | | $ | | GPU | | | | $ | | Memory | GB | | | | | Storage | GB | N/A | N/A | $ | ## Phase 3: Serving Configuration 1. Configure model serving - [ ] Model serialization format (ONNX, TensorFlow SavedModel, PyTorch, pickle) - [ ] Input validation and preprocessing pipeline - [ ] Output post-processing and formatting - [ ] Batching configuration (dynamic batching for throughput) - [ ] Timeout configuration for inference requests - [ ] Fallback behavior on model errors - [ ] API versioning for model versions 2. Test serving endpoint thoroughly ### API Contract | Field | Request | Response | Required | Validation | |-------|---------|----------|----------|-----------| | Input features | | N/A | | Type, range checks | | Predictions | N/A | | | | | Confidence scores | N/A | | | | | Model version | N/A | | | | | Request ID | | | | UUID format | ## Phase 4: Monitoring Setup 1. Configure model monitoring - [ ] Prediction latency (P50, P95, P99) - [ ] Prediction throughput (requests/sec) - [ ] Error rate and error types - [ ] Input data distribution (feature drift detection) - [ ] Output distribution (prediction drift detection) - [ ] Model performance degradation (if ground truth available) - [ ] Resource utilization (CPU, GPU, memory) - [ ] Feature store freshness (if applicable) 2. Set up alerts for anomalies ### Monitoring Alert Configuration | Alert | Condition | Severity | Action | |-------|-----------|----------|--------| | Latency spike | P95 > threshold | Warning | Investigate | | Error rate | > 1% | Critical | Page on-call | | Data drift | Distribution shift > threshold | Warning | Investigate model | | Prediction drift | Output shift > threshold | Warning | Retrain pipeline | | Resource saturation | CPU/GPU > 90% | Warning | Scale up | ## Phase 5: Deployment Strategy 1. Select deployment strategy - [ ] Shadow deployment (run alongside, compare outputs) - [ ] Canary deployment (gradual traffic shift) - [ ] A/B test (controlled experiment with metrics) - [ ] Blue-green (instant switch with rollback) 2. Execute deployment - [ ] Deploy new model version - [ ] Route initial traffic percentage - [ ] Monitor metrics during rollout - [ ] Gradually increase traffic if healthy - [ ] Full rollout or rollback decision ### Rollout Plan | Phase | Traffic % | Duration | Success Criteria | Rollback Trigger | |-------|----------|----------|-----------------|-----------------| | Shadow | 0% (duplicate) | 24 hours | Output matches expectations | N/A | | Canary | 5% | 4 hours | Error rate < 0.5%, latency OK | Error > 2% | | Expand | 25% | 12 hours | Metrics stable | Performance degradation | | Expand | 50% | 24 hours | Business metrics OK | Business impact | | Full | 100% | - | Stable for 72 hours | Any regression | ## Phase 6: Post-Deployment 1. Post-deployment validation - [ ] Verify model version serving correctly - [ ] Validate end-to-end prediction quality - [ ] Confirm monitoring and alerting working - [ ] Document deployment in model registry - [ ] Archive previous model version (keep for rollback) - [ ] Schedule model retraining pipeline - [ ] Set up data pipeline for ground truth collection ## Counter-Rationalizations | Shortcut | Counter | Why | |----------|---------|-----| | "We can skip some steps for this case" | Adapt the workflow steps, don't skip them | Skipped steps are where incidents and oversights originate | | "The user seems to already know what to do" | Complete all workflow phases with the user | The workflow catches blind spots that experience alone misses | | "This is a minor case, full process is overkill" | Scale the process down, don't turn it off | Minor cases become major when unstructured; the process scales, not disappears | | "I'll fill in the details later" | Complete each section before moving on | Deferred details are forgotten; real-time capture is more accurate | | "The template output isn't necessary" | Always produce the structured output format | Structured output enables comparison, audit trails, and handoff to other teams | ## Output Format - **Model Card**: Performance, limitations, and intended use - **Deployment Runbook**: Step-by-step deployment procedures - **Monitoring Dashboard**: Model health and drift metrics - **Rollback Procedure**: Steps to revert to previous version - **Retraining Schedule**: Automated retraining cadence ## Action Items - [ ] Validate model meets performance thresholds - [ ] Provision and test serving infrastructure - [ ] Configure monitoring and drift detection - [ ] Execute phased deployment with monitoring - [ ] Validate post-deployment metrics - [ ] Set up automated retraining pipeline - [ ] Document model version and deployment details