Deploy ML models on Kubernetes with KServe (formerly KFServing) and NVIDIA Triton Inference Server. Includes canary deployments, autoscaling, model versioning, A/B testing, and GPU resource management for production model serving.
Instalar com Codex ou Claude Copie este prompt, cole no Codex, Claude ou outro assistente e deixe que ele revise a página da skill e instale para você.
Um comando direto ignora o prompt de revisão. Verifique a origem antes de executá-lo.
Instruções da origem · Visualização somente leitura
name
model-serving-kubernetes
description
Deploy ML models on Kubernetes with KServe (formerly KFServing) and NVIDIA Triton Inference Server. Includes canary deployments, autoscaling, model versioning, A/B testing, and GPU resource management for production model serving.
license
MIT
metadata
{"author":"devops-skills","version":"1.0"}
Model Serving on Kubernetes
Production ML model serving with KServe and Triton — canary deployments, autoscaling, and GPU-aware scheduling.
When to Use This Skill
Use this skill when:
Serving scikit-learn, PyTorch, TensorFlow, or ONNX models at scale
Implementing canary deployments and A/B testing for ML models
Autoscaling inference pods based on request rate or GPU metrics
Deploying LLMs with Triton or KServe on Kubernetes
Managing multiple model versions with traffic splitting
apiVersion:serving.kserve.io/v1beta1kind:InferenceServicemetadata:name:llama-3-8bnamespace:modelsspec:predictor:canaryTrafficPercent:20# 20% to new version, 80% to stablecontainers:-name:vllm-containerimage:vllm/vllm-openai:latestargs:-"--model"-"meta-llama/Llama-3.1-8B-Instruct-v2"# new model versionresources:limits:nvidia.com/gpu:"1"
apiVersion:apps/v1kind:Deploymentmetadata:name:triton-servernamespace:modelsspec:replicas:2selector:matchLabels:app:tritontemplate:metadata:labels:app:tritonspec:containers:-name:tritonimage:nvcr.io/nvidia/tritonserver:24.05-py3args:-"tritonserver"-"--model-store=s3://my-model-store/models"-"--model-control-mode=poll"# auto-load new model versions-"--repository-poll-secs=30"-"--metrics-port=8002"ports:-containerPort:8000# HTTP-containerPort:8001# gRPC-containerPort:8002# Metricsresources:limits:nvidia.com/gpu:"1"readinessProbe:httpGet:path:/v2/health/readyport:8000initialDelaySeconds:30
# List loaded models (Triton)
curl http://triton:8000/v2/models
# Load a new model version
curl -X POST http://triton:8000/v2/repository/models/text-classifier/load
# Unload a model
curl -X POST http://triton:8000/v2/repository/models/text-classifier/unload
# KServe — watch rollout status
kubectl rollout status deployment/llama-3-8b-predictor -n models
kubectl get inferenceservice llama-3-8b -n models -w
Common Issues
Issue
Cause
Fix
InferenceService not ready
Model loading or OOM
Check predictor pod logs; increase memory limits
Canary stuck at 0%
KNative routing issue
Check kubectl get ksvc -n models
Triton missing model
S3 permissions or path
Verify IAM role; check --model-store path
Low GPU utilization
Dynamic batching off
Enable dynamic_batching in Triton config
Autoscaler not triggering
Prometheus query wrong
Test query in Prometheus UI
Best Practices
Use canary deployments for all model updates — roll back in seconds if metrics degrade.
Enable Triton dynamic batching — it can increase GPU throughput 5–10× for small models.
Store models in S3/GCS with versioned paths (s3://bucket/model/v1/, v2/).
Pin GPU node selectors to prevent model pods landing on CPU-only nodes.
Monitor p99 latency and error rates per model version during canary rollouts.
