model-serving-infrastructure

name	model-serving-infrastructure
description	Load when designing, costing, or debugging production model inference infrastructure — choosing managed API vs serverless GPU vs dedicated cluster, GPU utilization economics and break-even math, autoscaling and cold starts, token streaming/SSE and retry semantics, multi-model and LoRA serving, model rollout, or serving observability (TTFT, queue depth, KV cache).

Model Serving Infrastructure

The standard doctrine, compressed

A strong model already produces this cold; anchors only. You buy GPU-time, sell tokens; utilization is the exchange rate and goodput (SLO-meeting throughput) the metric. Autoscale on work waiting (vllm:num_requests_waiting, in-flight per replica, KV usage), never GPU-util % — DCGM's util is a kernel-residency duty cycle that reads ~100% during memory-stalled decode. Cold start is the scale-to-zero tax: bake/stream weights (never pull from a hub at boot), GPU-state snapshotting (Modal-class: 7B vLLM ~2 min → ~10s), warm pools; scale-up thresholds derived from measured cold-start time; thundering herd (N replicas pulling the same 30 GB through one path) needs jittered starts, capped step size, region-local weight cache. Streaming: SSE with heartbeats, buffering disabled on every hop (X-Accel-Buffering: no), timeout per gap (TTFT ~30–60s + inter-chunk ~10–30s), retries only on provably-pre-generation failures with idempotency keys and a retry budget — retried 90-second GPU calls are metastable-failure fuel, never web-service milliseconds. Client disconnects must propagate to engine abort (10–30% of GPU spend at chat/agent abandonment rates; kill a client mid-stream and watch the running gauge). Per-customer fine-tunes = multi-LoRA on a shared base (vLLM --enable-lora; same base + tokenizer; adapter-cache thrash if the working set exceeds headroom) — full weights per customer is a 10–100× mistake. Shared prefix cache is a cross-tenant timing oracle: per-tenant cache_salt or pools. MIG = hardware isolation for small latency-sensitive models; time-slicing never behind an SLO; fractional GPUs are irrelevant for big LLMs. Rollout: offline eval gate → shadow (validates latency/OOM/distributions, structurally cannot measure user impact) → canary by user/session, never per-request; keep N−1 warm. max-model-len to what you support; per-endpoint max_tokens; readiness only after a synthetic warm-up generation. Right-size before platform: 40k req/day ≈ 2–3 req/s = a 1–3 GPU problem, not a k8s+llm-d problem.

Corrections and sharpenings

• Break-even math: compute GPU-seconds per request, don't guess aggregate throughput. The baseline reflex assumes ~1,000–1,500 tok/s "combined" and lands at 50–77% break-even utilization. Do it properly (2026 ballparks, re-verify): 70B FP8 saturated ≈ 450–500 output tok/s aggregate + prefill ~15k tok/s. Request 2,000 in/400 out → 400/460 + 2,000/15,000 ≈ 1.0 GPU-s. H100 at ~$2.50/hr = $0.00069/req at saturation vs API ~$0.90/M × 2,400 tok = $0.00216 → break-even ≈ 32% sustained duty cycle, not 50–77%. Sensitivities: output-heavy shapes push break-even up (decode eats GPU-time, API bills all tokens flat); reserved pricing (−30–40%) and prefix-cache hits push it down. Prices: specialized clouds ~$2–3.3/hr H100, hyperscalers 2–5× that; serverless ~1.5–3× on-demand per-second → serverless-vs-dedicated break-even ≈ 12 busy h/day at 2× premium (8 h at 3×), which is why a floor-replica + serverless burst usually beats either pure option for 9-to-5 traffic.
• The 2026 fleet layer, by name (rots fast — verify): llm-d and KServe on k8s for distributed serving; NVIDIA Dynamo 1.0 GA (March 2026) for disaggregated prefill/decode; AIBrix; Ray Serve for many-model multiplexing; Replicate is Cloudflare-owned since early 2026. NVIDIA's DRA driver was donated to the CNCF in 2026 and is the direction of travel for GPU allocation, but DRA still does not re-partition MIG dynamically — MIG geometry is chosen per node pool ahead of time; re-partitioning is a drain-the-node lifecycle operation. Below fleet scale, session-sticky routing + per-replica automatic prefix caching captures ~80% of KV-aware routing's value with none of the machinery.
• Long-context is the launch-day time bomb. Tuned at 2k-token prompts, --gpu-memory-utilization 0.9; weeks later one 100k-token paste forces mass preemption or OOMs the engine. Corrections: --max-model-len = product limit, gateway-level input validation/truncation, load tests at max context × max concurrency, alert on preemption counters (storms precede OOM) and on p99 input-length shifts — token-shape changes are cost and stability changes before they're incidents.
• Shadow-mode conclusions about quality are a category error — nothing acted on the outputs, so "shadow looked fine" cannot license promotion; the canary needs an outcome metric (e.g. edit distance on drafts agents actually used). And an engine-version bump is a model change: same weights, new engine can change outputs — canary it like a model.
• Quality drift needs its own detector: serving metrics stay pristine while a rollout/quantization/prompt regression tanks output quality — continuously sample production outputs into an eval pipeline per model version.
• Prefix stability is an economic invariant: someone prepends Current time: 09:41 to the system prompt → hit rate ~0, prefill cost doubles, TTFT jumps, nobody connects it for a week. Volatile content at the end; alert on hit-rate step changes.
• Warm floors follow the clock, not the mean. Weekend scale-to-zero is fine for an internal tool; weekday-overnight zero gives the 7am user a daily cold start — schedule minReplicas to rise before business hours. Scale-to-zero is a product decision per endpoint: cold-starts/day × latency cost vs idle burn saved.

Verification / self-check

• Cost comparison uses the real in:out shape, measured duty cycle, and this-quarter prices, dated — and the GPU-time-per-request method above, not a guessed aggregate throughput.
• SLO stated first; batch/scaling derived from it; autoscaling signal is queue/concurrency/KV; thundering-herd path analyzed.
• Streaming tested through the full proxy chain; disconnect-abort verified by killing a client mid-stream; retry policy idempotent with a budget.
• Long-input load-tested at max × max; gateway caps and per-endpoint max_tokens in place.
• Rollout: eval gate + entity-randomized canary + incumbent kept warm; every stack component justified by a number, not fashion.
• Stopping rule: SLO met at peak with 20–30% headroom, cost model beats the rejected alternative with shown arithmetic, and the failure drills (cold start, spike, long prompt, mid-stream disconnect, bad-model rollback) each run once. Revisit when traffic shape, model size, or prices move materially.

Delta notes (vs Opus 4.8 baseline, audited 2026-07)

• Probed 13 claims: 11 baseline (cut/compressed — DCGM-util fallacy + correct vLLM signals, cold-start techniques incl. snapshotting magnitudes, streaming timeout/retry semantics, disconnect-abort with spend estimate, multi-LoRA architecture and limits, cache_salt cross-tenant fix, MIG/time-slicing/MPS placement, shadow-vs-canary reasoning, thundering herd, right-sizing rejection of k8s-for-3-req/s, observability set), 2 partial (sharpened), 0 fully wrong.
• Biggest baseline gaps: break-even arithmetic — Opus's throughput assumption (counting prefill+decode as one aggregate rate) lands at 50–77% break-even vs ~32% by GPU-time-per-request decomposition, a decision-flipping error; fleet-layer currency (Dynamo 1.0 GA, DRA→CNCF donation, Replicate/Cloudflare) is post-cutoff.
• Kept as likely-delta despite unprobed: input-length-distribution alerting, engine-bump-as-model-change canary rule, clock-scheduled warm floors.