model-serving

name	model-serving
version	2.0.0
sasmp_version	1.3.0
description	Master model serving - inference optimization, scaling, deployment, edge serving
bonded_agent	05-model-serving
bond_type	PRIMARY_BOND
category	deployment
difficulty	intermediate_to_advanced
estimated_hours	35
prerequisites	["mlops-basics","training-pipelines"]
validation	{"pre_conditions":["Completed prerequisite skills","Trained model available"],"post_conditions":["Can deploy models with BentoML/Triton","Can optimize inference latency","Can configure auto-scaling"]}
observability	{"metrics":["models_deployed","inference_latency","optimization_speedup"]}

Model Serving Skill

Learn: Deploy ML models for production inference with optimization.

Skill Overview

Attribute	Value
Bonded Agent	05-model-serving
Difficulty	Intermediate to Advanced
Duration	35 hours
Prerequisites	mlops-basics, training-pipelines

---

Learning Objectives

1. Deploy models with BentoML and Triton
2. Optimize inference with quantization and ONNX
3. Configure auto-scaling policies
4. Implement batch and streaming inference
5. Deploy to edge devices

---

Topics Covered

Module 1: Serving Platforms (8 hours)

Platform Comparison:

Platform	Multi-framework	Dynamic Batching	Kubernetes
TorchServe	PyTorch only	✅	✅
Triton	✅	✅	✅
BentoML	✅	✅	✅
Seldon	✅	⚠️	✅

---

Module 2: BentoML Deployment (10 hours)

Service Definition:

import bentoml
from bentoml.io import JSON, NumpyNdarray

@bentoml.service(resources={"gpu": 1, "memory": "4Gi"})
class ModelService:
    def __init__(self):
        .model = bentoml.pytorch.load_model()


      () -> :
         torch.no_grad():
            predictions = .model(input_array)
         {: predictions.tolist()}