llm-serving-optimizer

name

llm-serving-optimizer

description

Tune self-hosted LLM serving for efficiency — continuous batching configuration, KV-cache sizing, quantized serving, speculative decoding, replica right-sizing, and GPU utilization per served token. Use this skill whenever the user self-hosts models (vLLM, TGI, TensorRT-LLM, Ollama at scale), shares serving configs or GPU utilization data, or asks how to serve more tokens per GPU. Part of Lean Agentic AI Skills; emits lean-findings.json.

LLM Serving Optimizer

Subject type: emit subject.type: "ai-inference" in findings.

Producer skill. Input: serving configs (vLLM/TGI/TensorRT-LLM engine args, replica counts), GPU utilization and throughput metrics, traffic patterns. Output: lean-findings.json.

model-right-sizer decides what runs; this skill tunes how it runs. A serving stack at 20% GPU utilization wastes the same silicon whether the model choice was right or not — and serving defaults are tuned for demos, not efficiency.

Signatures

• No continuous batching / tiny batch ceilings — sequential or small-batch serving where the engine supports continuous batching; tokens-per-GPU-second far below the hardware's envelope. High. Evidence: throughput metrics vs engine capability. Fix: enable/raise batching limits; measure latency SLO impact.
• KV-cache misconfiguration — cache sized so small that batches evict and recompute, or so large that replica count is inflated for memory rather than compute. High. Fix: size to the measured context-length distribution (from logs), not the model's maximum.
• fp16 serving where quantized holds quality — 8-bit/4-bit serving variants untested despite mature support. Medium-high: same GPU serves materially more tokens. Fix: quality eval on the user's tasks before/after — mandatory, quantization loss is task-dependent.
• No speculative decoding for long-output workloads with an available draft model. Medium; gains are workload-dependent — recommend a measured trial, never a promise.
• Replica overprovisioning — replica count sized to peak with no autoscaling; nights and weekends at flat capacity. High for bursty traffic. Fix: scale-to-load, with cold-start trade-off named; pairs with right-size-compute patterns at the fleet level.
• Long max-tokens defaults — server-side generation ceilings far beyond the application's real outputs, holding batch slots hostage. Medium. Pairs with prompt-carbon-optimizer's output caps.
• Prefix caching disabled — shared system prompts recomputed per request where the engine supports prefix/prompt caching. Medium-high. Pairs with llm-cache-designer layer 1.
• Wrong hardware for the model — model sharded across GPUs it doesn't need, or squeezed onto cards forcing tiny batches. Medium; route fleet choices to right-size-compute, record the serving-level finding here.

Cost signal (countable)

Cost driver: GPU-hours per million served tokens. Both terms countable from serving metrics; this ratio is the skill's whole scoreboard. observed = current tokens/GPU-second and replica-hours; direction = qualitative ("continuous batching typically multiplies tokens-per-GPU-second on concurrent workloads — measure on your traffic").

Honesty rules

Every throughput change trades against latency — p95 latency before/after is the mandatory companion measurement. Quantization requires the user's own quality eval. Vendor-published speedups are not your evidence; the user's metrics are.

Not this skill's job

Choosing the model tier (model-right-sizer), response/semantic caching (llm-cache-designer), edge-vs-cloud placement (inference-placement-advisor), buying the GPUs (hardware-lifecycle-advisor).