agentsop-llm-engine-selection

# LLM Engine Selection SOP > **State-of-the-art warning.** This skill is dated **May 2026**. The inference-engine landscape moves in 3–6 month cycles (TGI exited maintenance into deprecation in late 2025; SGLang's RadixAttention regressed vLLM's lead in 2024; TensorRT-LLM dropped its proprietary-engine requirement in 2025; Ollama added concurrent-request support mid-2025). Re-verify before betting a quarter of eng budget on any choice below. --- ## 1. 何时激活 (When to activate) Activate this skill any time a coder-agent must: - **Pick** a serving stack for a new project (production hosting / batch / edge / dev laptop / multi-tenant SaaS / structured-output service). - **Defend** an existing stack against a "let's switch to X" pressure. - **Migrate**: justify or block a swap (e.g. TGI → vLLM, Ollama → vLLM, vLLM → TensorRT-LLM). - **Mix**: design a multi-tier deployment (e.g. premium tier on TensorRT-LLM, free tier on vLLM-AWQ, dev on Ollama). - **Audit** a recommendation that smells like benchmark-cherry-picking ("X is 5× faster"). Do **not** activate for: - Tuning a single chosen engine — defer to the dedicated skill (`vllm`, `sglang`, `tensorrt-llm`, `llama-cpp`). - Training/fine-tuning runtime selection — different problem class (`accelerate`, `deepspeed`, `axolotl`). - Hosted-API procurement (OpenAI / Anthropic / Bedrock) — engine choice doesn't apply. --- ## 2. 核心心智模型 (Core Mental Model) **Engine choice is a function of (hardware × workload × constraint), not "which is fastest".** There is no global ranking. Every "X beats Y by N%" headline holds only inside an unstated (hardware, batch size, ISL/OSL, model, quantization, concurrency) tuple. Change any axis and the ranking flips. ### 2.1 The four-axis decision space 1. **Hardware axis** — NVIDIA H100/A100 (NVLink) ≠ NVIDIA L40S/RTX (PCIe-only) ≠ AMD MI300 ≠ Apple Silicon ≠ CPU-only. The interconnect topology matters as much as raw FLOPS [spheron.network 2026]. PCIe-only tensor parallelism collapses; NVLink rescues it. 2. **Workload axis** — production multi-user (throughput) vs latency-bound single-stream vs offline batch vs edge single-user vs structured-output service vs multi-LoRA SaaS. Each has a different winner. 3. **Constraint axis** — license (Apache vs proprietary), vendor lock-in tolerance, engineering budget (1 day vs 2 weeks setup), commercial-use clauses, on-prem vs cloud, P50 vs P99 SLA. 4. **Maturity axis** — the engine's coverage of YOUR model family. A 2025-launched MoE may run on vLLM day-1 but need a 3-month wait for TensorRT-LLM, and may never get a stable GGUF. ### 2.2 The default in 2026 For **GPU-backed, multi-user, open-weights** serving, the default is **vLLM**. It owns the production slot because it is vendor-neutral (NVIDIA/AMD/Intel/TPU/Apple-experimental), supports 200+ architectures including MoE/multimodal, ships an OpenAI-compatible API, and HuggingFace themselves recommend it over their own (now-maintenance-mode) TGI [yottalabs.ai 2026; vllm-project README]. You only reach past vLLM when one of three conditions binds: - **Workload-binding** (heavy prefix sharing or structured generation → SGLang). - **Hardware-binding** (NVIDIA-only + max throughput goal → TensorRT-LLM; CPU/edge/Apple → llama.cpp/MLX/Ollama). - **Operator-binding** (dev laptop, want 5-min setup → Ollama; ≤1 concurrent user → llama.cpp). ### 2.3 "Fastest" is a category error Throughput-per-GPU, throughput-per-dollar, P50 TTFT, P99 ITL, and developer-time-to-first-request are **five different goals**, and the engines optimize for different combinations: | Engine | What it optimizes for | |---|---| | vLLM | Throughput-per-GPU across mixed traffic, model breadth | | SGLang | Throughput when requests share prefix; structured-gen TPS | | TensorRT-LLM | Peak throughput on NVIDIA at saturation; per-token cost at scale | | TGI | Was generic; now mostly a migration source | | llama.cpp | Single-user TPS on CPU/Apple/edge; minimal-deps install | | Ollama | Developer-time-to-first-request (5 min) | | MLX | Apple Silicon throughput and Apple-native dev UX | Pick by **which axis your project is binding on**, not by which engine has the most stars. --- ## 3. SOP 工作流 (SOP Workflow) ``` [Step 0] Define the four-axis constraint vector ├─ Hardware: GPU vendor, count, interconnect (NVLink? PCIe?), VRAM/GPU ├─ Workload: # concurrent users, ISL/OSL distribution, shared-prefix %, structured-out % ├─ Constraint: license, vendor-lock tolerance, eng-days budget, P50/P99 SLA └─ Model: family (Llama/Qwen/Mixtral/DeepSeek/Mamba/...), size, quantization preference [Step 1] Eliminate incompatible engines (hard filters) ├─ No NVIDIA GPU? → drop TensorRT-LLM ├─ CPU/Apple/edge only? → drop vLLM (production), TGI, TensorRT-LLM ├─ Need OSS-permissive only? → drop TensorRT-LLM (NVIDIA license) ├─ Mamba / brand-new arch? → check vLLM+SGLang coverage; likely drop others └─ Multi-LoRA hot-swap? → vLLM (best), TensorRT-LLM (good), SGLang (good); others drop [Step 2] Map workload to engine strength ├─ Throughput + mixed traffic → vLLM ├─ Shared prefixes (chatbot, agent) → SGLang (≈29% over vLLM on shared-context [n1n.ai 2026]) ├─ Structured JSON/regex at scale → SGLang (RadixAttention + state machines) ├─ Peak throughput, NVIDIA-only, big budget → TensorRT-LLM (+30–50% over vLLM [n1n.ai 2026]) ├─ Single user, dev laptop → Ollama ├─ CPU/Apple/edge/embedded → llama.cpp / MLX └─ Multi-tenant SaaS with LoRA fleet → vLLM (multi-LoRA mature) or SGLang [Step 3] Sanity-check hardware topology ├─ TP requires NVLink (≈900 GB/s) — not PCIe Gen4 (≈32 GB/s, ~28× slower) ├─ PCIe-only box → smaller TP + replicas, or PP, or switch engines ├─ Cross-NUMA across the same node → as bad as PCIe; pin to NUMA-local GPUs └─ Multi-node → require IB / RoCE, not Ethernet [Step 4] Benchmark top 2 on YOUR workload ├─ Replay representative ISL/OSL distribution (NOT MMLU prompts) ├─ Measure P50 + P99 TTFT, P50 + P99 ITL, tokens/s/GPU, $/M-tokens ├─ Watch saturation: queue depth, KV occupancy, preemption count └─ Decide; document the constraint vector that justified the pick [Step 5] Plan the escape hatch ├─ Note the workload threshold that would force a switch ├─ Keep the OpenAI-compatible API layer so swaps are mechanical └─ Re-evaluate every 6 months — engines evolve in quarters, not years ``` --- ## 4. 操作模型 (Operation Model) ### OP-1: Decision matrix lookup by (hardware, workload) - **Trigger**: User asks "what should we serve X on?" and gives hardware + workload. - **Action**: Look up the row in §7 table; eliminate by Step 1 hard filters; pick by Step 2 strength match. - **Output**: A primary engine + a fallback, with the constraint vector that justified the pick. - **Evidence**: §7 cross-framework matrix; [yottalabs.ai 2026]; [aimadetools.com 2026]. ### OP-2: Detect "wrong-tool-for-the-job" symptoms - **Trigger**: A team is using engine E but reporting one of: low throughput despite tuning, unsupported model architecture, structured-output workarounds, single-user pain on a multi-user engine. - **Action**: Match symptom → engine mismatch: - vLLM stuck below expected throughput on PCIe-only L40S → switch to PP or 2× independent TP=2 replicas (engine isn't the problem; topology is). - vLLM with heavy `guided_decoding` overhead on agent traffic → A/B SGLang. - TGI deployment, HF themselves now recommend vLLM/SGLang → plan migration. - Ollama in "production" with ≥10 concurrent users → graduate to vLLM. - llama.cpp on a serving-many-users box → graduate to vLLM. - **Output**: Migration recommendation with cited reason. - **Evidence**: [yottalabs.ai 2026]; [contracollective.com 2026]. ### OP-3: Quantization-driven engine pick - **Trigger**: Quantization format dictates engine; team has GGUF / AWQ / GPTQ / FP8 weights already. - **Action**: - **GGUF only** → llama.cpp / Ollama / LM Studio. vLLM has experimental GGUF; not production-grade. - **AWQ / GPTQ** → vLLM (first-class), SGLang (good), TensorRT-LLM (supported, needs engine rebuild). - **FP8 (Hopper/Ada+)** → vLLM, SGLang, TensorRT-LLM all support; near-lossless [arxiv 2411.02355]. - **MLX format (Apple)** → MLX / mlx-lm; not portable to other engines. - **Output**: Engine shortlist gated by available weights format. - **Evidence**: [docs.vllm.ai/quantization]; [arxiv.org/abs/2411.02355]. ### OP-4: Structured-output workload routing - **Trigger**: ≥30% of requests need JSON / regex / grammar-constrained output, OR the project is an agent with tool-call state machines. - **Action**: Default to **SGLang**. RadixAttention + first-class state machines beat vLLM's `guided_decoding` at scale. If staying on vLLM, expect throughput hit on constrained requests; budget for it. - **Output**: SGLang shortlisted as primary; vLLM as fallback if SGLang doesn't support the model. - **Evidence**: [sglang.ai blog]; [n1n.ai 2026 SGLang +29% on shared-context]. ### OP-5: Apple Silicon / edge routing - **Trigger**: Target platform is M-series Mac, iPhone/iPad, Jetson, Raspberry Pi, or other ARM/edge. - **Action**: - **M-series Mac, dev/local**: Ollama (best UX) or MLX (best perf, Apple-native). - **M-series Mac, production server**: llama.cpp Metal backend or MLX. vLLM Metal/MPS is experimental as of 2026; do not ship on it [aimadetools.com 2026; contracollective.com 2026]. - **Jetson / ARM Linux**: llama.cpp (broadest hardware support) or TensorRT-LLM if Jetson Orin. - **CPU-only x86 server**: llama.cpp with AVX-512; expect single-digit users. - **Output**: Apple/edge-specific engine pick with format requirement (GGUF / MLX). - **Evidence**: [contracollective.com 2026]; [aimadetools.com 2026]. ### OP-6: Multi-tier deployment design - **Trigger**: Project has both interactive (premium, latency-bound) and batch (free, throughput-bound) tiers. - **Action**: Run heterogeneous engines: - Premium / interactive on **TensorRT-LLM** (if NVIDIA-locked + budget) or **vLLM FP8** with prefix caching. - Free / batch on **vLLM AWQ-4** (high concurrency, lower cost) or **SGLang** if prefix-sharing. - Dev / staging on **Ollama** for fast model swaps. - Edge / on-device client on **llama.cpp** (GGUF). - **Output**: 2–3 engine deployment plan with a single OpenAI-compatible gateway in front. - **Evidence**: Common production pattern documented in [yottalabs.ai 2026; contracollective.com 2026]. ### OP-7: Bench-then-pick when the choice is close - **Trigger**: Two engines both pass Step 1 + 2; team disagrees. - **Action**: Run a 1-day benchmark with replayed traffic. Required metrics: P50 + P99 TTFT, P50 + P99 ITL, tokens/s/GPU, $/M-tokens, peak memory, model-load time. **Do not benchmark on MMLU-style prompts**; use real production ISL/OSL. - **Output**: A decision memo with the constraint vector + measured numbers + the threshold that would flip the decision. - **Evidence**: Standard practice; all cited comparison reports note that workload-specific benchmarks override published numbers. --- ## 5. 困境决策案例 (Dilemma Cases) ### Dilemma 1: Team wants vLLM, but only has a PCIe-only 4-GPU box **Situation**: A team standardized on vLLM but their cluster is 4× L40S on PCIe Gen4 (no NVLink). They configure `--tensor-parallel-size 4` for Llama-3-70B-FP8 and see throughput collapse — single-replica TPS is ~30% of what the H100×4 benchmark advertises. **Tension**: - vLLM is the right *engine*, but TP=4 over PCIe is the wrong *topology*. The all-reduce after every layer chokes on the ~32 GB/s PCIe link vs the ~900 GB/s NVLink the benchmarks assumed [spheron.network 2026]. - Switching engines doesn't help — the same all-reduce penalty hits TensorRT-LLM and SGLang. **Resolution**: - **First**: drop to `TP=2 × 2 replicas` (each replica uses an NVLink-paired or NUMA-local pair, if any), or use pipeline parallelism `PP=4, TP=1`. Replicas eliminate the cross-GPU all-reduce on the hot path [developers.redhat.com 2026 step-5]. - **Second**: if 70B doesn't fit at TP=2 with FP8 on L40S (48 GB/GPU), step down to AWQ-4 or use Llama-3-8B until hardware upgrades. - **Third**: if neither works, swap to TGI (similar PCIe penalty but lower TP-coupling overhead in some configs) or accept a switch to llama.cpp for low-concurrency workloads. - **Key lesson**: the bottleneck was hardware topology, not engine choice. Always check interconnect before blaming the engine. **Evidence**: [docs.vllm.ai parallelism_scaling]; [developers.redhat.com 2026]; [spheron.network 2026]. ### Dilemma 2: Structured JSON output at scale — SGLang vs vLLM **Situation**: An agent platform serves Qwen-2.5-32B-Instruct with ~70% of requests demanding strict JSON schema output (tool-calling, function arguments). On vLLM with `guided_decoding`, throughput drops ~40% vs unconstrained baseline; P99 TTFT regresses. **Tension**: - vLLM has the broader model + production track record; team already operates it. - SGLang's RadixAttention + native state-machine compilation gives ≈29% higher throughput on shared-context workloads and meaningful gains on constrained decoding [n1n.ai 2026]; agent traffic has both properties. - Migration cost is real: deployment scripts, observability, ops runbooks. **Resolution**: - **A/B**: run SGLang as a sidecar on a fraction of traffic. Measure constrained-output TPS + correctness rate on real JSON schemas (not toy schemas). - **If SGLang wins ≥25%** on YOUR traffic → migrate the structured-output path; keep vLLM for unconstrained traffic, OR consolidate on SGLang if it covers all your models. - **If SGLang model coverage misses** a key model in your fleet → stay on vLLM, optimize `guided_decoding` (xgrammar backend), and re-evaluate next quarter. - **Heuristic**: structured-output workload share above ~30% justifies SGLang shortlisting; above ~50% it usually wins. **Evidence**: [sglang.ai/blog]; [n1n.ai 2026]; [yottalabs.ai 2026]. ### Dilemma 3: NVIDIA shop with 2-week budget — vLLM or TensorRT-LLM? **Situation**: NVIDIA-only cluster (8×H100, NVLink-full), single-tenant, throughput-per-GPU is the dominant cost line. Team has 2 weeks of senior eng time. TensorRT-LLM promises +30–50% throughput [n1n.ai 2026] but requires engine build per (model × precision × max-batch × max-seq) tuple. **Tension**: - **TensorRT-LLM wins** on saturated throughput and is the right tool when GPU-hours are the binding cost. - **It loses** on agility: every model update or context-length change requires an engine rebuild (10 min – 2 hours); architecture support lags vLLM by 3–6 months for new releases; vendor lock-in if you ever consider AMD/TPU. - vLLM tolerates new models on day-1 and supports any NVIDIA + AMD + Intel hardware. **Resolution**: - **If model is stable for ≥3 months and throughput dominates cost** → TensorRT-LLM is correct. Spend the 2 weeks. - **If model churn is monthly OR you ship new models often OR you want vendor optionality** → vLLM. The +30–50% throughput is rarely worth the agility loss. - **Hybrid**: vLLM during model-fit phase; freeze a winning model into a TensorRT-LLM engine for steady-state production. **Evidence**: [n1n.ai 2026]; [yottalabs.ai 2026]; [spheron.network 2026]. ### Dilemma 4: "We want one engine for laptop dev → cloud prod" **Situation**: A small team wants the same engine for local dev on M-series Macs AND production on cloud NVIDIA. They are tempted to standardize on Ollama (since it runs everywhere) or vLLM (since it's the prod default). **Tension**: - **Ollama everywhere**: great dev UX, but tops out at single-digit concurrent users; vLLM-class throughput is unreachable. Using Ollama in cloud production wastes 10–20× the GPU spend [contracollective.com 2026]. - **vLLM everywhere**: production-grade, but Metal/MPS is experimental on Mac; dev velocity is worse than Ollama; single-user latency is no better than llama.cpp. **Resolution**: - Do **not** standardize on one engine. Standardize on the **API contract** (OpenAI-compatible HTTP). - **Local dev**: Ollama (M-series) or vLLM (if devs have NVIDIA workstations). The OpenAI compatibility means application code is identical. - **Production**: vLLM (default) or SGLang/TensorRT-LLM (if Step 2 binds). - **Edge / on-device**: llama.cpp with GGUF. - The "one engine" goal is the wrong abstraction; the right abstraction is "one API, multiple runtimes" — same pattern as JVM languages, same pattern as POSIX [yottalabs.ai 2026 common pattern]. **Evidence**: [contracollective.com 2026]; [yottalabs.ai 2026]. --- ## 6. 反模式与边界 (Anti-patterns & Boundaries) ### Anti-pattern 1: Pick by GitHub stars or community vibes