ai-llm-inference

name	ai-llm-inference
description	Operational patterns for LLM inference: latency budgeting, tail-latency control, caching, batching/scheduling, quantization/compression, parallelism, and reliable serving at scale. Emphasizes production-grade performance, cost control, and observability.

LLMOps – Inference & Optimization – Production Skill Hub

Modern Best Practices (January 2026):

• Treat inference as a systems problem: SLOs, tail latency, retries, overload, and cache strategy.
• Use continuous batching / smart scheduling when serving many concurrent requests (Orca scheduling: https://www.usenix.org/conference/osdi22/presentation/yu).
• Use KV-cache aware serving (PagedAttention/vLLM: https://arxiv.org/abs/2309.06180) and efficient attention kernels (FlashAttention: https://arxiv.org/abs/2205.14135).
• Use speculative decoding when latency is critical and draft-model quality is acceptable (speculative decoding: https://arxiv.org/abs/2302.01318).
• Quantize only with measured quality impact and rollback plan (quantization must be validated on your eval set).

This skill provides production-ready operational patterns for optimizing LLM inference performance, cost, and reliability. It centralizes decision rules, optimization strategies, configuration templates, and operational checklists for inference workloads.

No theory. No narrative. Only what Claude can execute.

---

When to Use This Skill

Claude should activate this skill whenever the user asks for:

• Optimizing LLM inference latency or throughput
• Choosing quantization strategies (FP8/FP4/INT8/INT4)
• Configuring vLLM, TensorRT-LLM, or DeepSpeed inference
• Scaling LLM inference across GPUs (tensor/pipeline parallelism)
• Building high-throughput LLM APIs
• Improving context window performance (KV cache optimization)
• Using speculative decoding for faster generation
• Reducing cost per token
• Profiling and benchmarking inference workloads
• Planning infrastructure capacity
• CPU/edge deployment patterns
• High availability and resilience patterns

Scope Boundaries (Use These Skills for Depth)

• Prompting, tuning, datasets → ai-llm
• RAG pipeline construction → ai-rag
• Deployment, APIs, monitoring → ai-mlops
• Safety, governance → ai-mlops

---

Quick Reference

Task	Tool/Framework	Command/Pattern	When to Use
Latency budget	SLO + load model	TTFT/ITL + P95/P99 under load	Any production endpoint
Tail-latency control	Scheduling + timeouts	Admission control + queue caps + backpressure	Prevent p99 explosions
Throughput	Batching + KV-cache aware serving	Continuous batching + KV paging	High concurrency serving
Cost control	Model tiering + caching	Cache (prefix/response) + quotas	Reduce spend and overload risk
Long context	Prefill optimization	Chunked prefill + prompt compression	Long inputs and RAG-heavy apps
Parallelism	TP/PP/DP	Choose by model size and interconnect	Models that do not fit one device
Reliability	Resilience patterns	Timeouts + circuit breakers + idempotency	Avoid cascading failures

---

Decision Tree: Inference Optimization Strategy

Need to optimize LLM inference: [Optimization Path]
    ├─ Primary constraint: Throughput?
    │   ├─ Many concurrent users? → batching + KV-cache aware serving + admission control
    │   └─ Mostly batch/offline? → batch inference jobs + large batches + spot capacity
    │
    ├─ Primary constraint: Cost?
    │   ├─ Can accept lower quality tier? → model tiering (small/medium/large router)
    │   └─ Must keep quality? → caching + prompt/context reduction before quantization
    │
    ├─ Primary constraint: Latency?
    │   ├─ Draft model acceptable? → speculative decoding
    │   └─ Long context? → prefill optimizations + attention kernels + context budgets
    │
    ├─ Large model (>70B)?
    │   ├─ Multiple GPUs? → Tensor parallelism (NVLink required)
    │   └─ Deep model? → Pipeline parallelism (minimize bubbles)
    │
    └─ Edge deployment?
        └─ CPU + quantization → Optimized for constrained resources

---

Core Concepts & Practices

Core Concepts (Vendor-Agnostic)

• Latency components: queueing + prefill + decode; optimize the largest contributor first.
• Tail latency: p99 is dominated by queuing and long prompts; fix with admission control and context budgets.
• Retries: retries can multiply load; bound retries and use hedged requests only with strict budgets.
• Caching: prefix caching helps repeated system/tool scaffolds; response caching helps repeated questions (requires invalidation).
• Security & privacy: prompts/outputs can contain sensitive data; scrub logs, enforce auth/tenancy, and rate-limit abuse (OWASP LLM Top 10: https://owasp.org/www-project-top-10-for-large-language-model-applications/).

Implementation Practices (Tooling Examples)

• Measure under load: benchmark TTFT/ITL and p95/p99 with realistic concurrency and prompt lengths.
• Separate environments: dev/stage/prod model configs; promote only after passing the inference review checklist.
• Export telemetry: request-level tokens, TTFT/ITL, queue depth, GPU memory headroom, and error classes (OpenTelemetry GenAI semantic conventions: https://opentelemetry.io/docs/specs/semconv/gen-ai/).

Do / Avoid

Do

• Do enforce max_input_tokens and max_output_tokens at the API boundary.
• Do cap concurrency and queue depth; return overload errors quickly.
• Do validate quality after any quantization or kernel change.

Avoid

• Avoid unbounded retries (amplifies outages).
• Avoid unbounded context windows (OOM + latency spikes).
• Avoid benchmarking on single requests; always test with realistic concurrency.

---

Resources (Detailed Operational Guides)

For comprehensive guides on specific topics, see:

Infrastructure & Serving

• Infrastructure Tuning - OS, container, Kubernetes optimization for GPU workloads
• Serving Architectures - Production serving stack patterns
• Resilience & HA Patterns - Multi-region, failover, traffic management

Performance Optimization

• Quantization Patterns - FP8/FP4/INT8/INT4 decision trees and validation
• KV Cache Optimization - PagedAttention, FlashAttention, prefix caching
• Parallelism Patterns - Tensor/pipeline/expert parallelism strategies
• Optimization Strategies - Throughput, cost, memory optimization
• Batching & Scheduling - Continuous batching and throughput patterns

Deployment & Operations

• Edge & CPU Optimization - llama.cpp, GGUF, mobile/browser deployment
• GPU Optimization Checklists - Hardware-specific tuning
• Speculative Decoding Guide - Advanced generation acceleration
• Profiling & Capacity Planning - Benchmarking, SLOs, replica sizing

---

Templates

Inference Configs

Production-ready configuration templates for leading inference engines:

• vLLM Configuration - Continuous batching, PagedAttention setup
• TensorRT-LLM Configuration - NVIDIA kernel optimizations
• DeepSpeed Inference - PyTorch-friendly inference

Quantization & Compression

Model compression templates for reducing memory and cost:

• GPTQ Quantization - GPU post-training quantization
• AWQ Quantization - Activation-aware weight quantization
• GGUF Format - CPU/edge optimized formats

Serving Pipelines

High-throughput serving architectures:

• LLM API Server - FastAPI + vLLM production setup
• High-Throughput Setup - Multi-replica scaling patterns

Caching & Batching

Performance optimization templates:

• Prefix Caching - KV cache reuse strategies
• Batching Configuration - Continuous batching tuning

Benchmarking

Performance measurement and validation:

• Latency & Throughput Testing - Load testing framework

Checklists

• Inference Performance Review Checklist - Baseline, bottlenecks, rollout readiness

Navigation

Resources

• resources/serving-architectures.md
• resources/profiling-and-capacity-planning.md
• resources/gpu-optimization-checklists.md
• resources/speculative-decoding-guide.md
• resources/resilience-ha-patterns.md
• resources/optimization-strategies.md
• resources/kv-cache-optimization.md
• resources/batching-and-scheduling.md
• resources/quantization-patterns.md
• resources/parallelism-patterns.md
• resources/edge-cpu-optimization.md
• resources/infrastructure-tuning.md

Templates

• templates/serving/template-llm-api.md
• templates/serving/template-high-throughput-setup.md
• templates/inference/template-vllm-config.md
• templates/inference/template-tensorrtllm-config.md
• templates/inference/template-deepspeed-inference.md
• templates/quantization/template-awq.md
• templates/quantization/template-gptq.md
• templates/quantization/template-gguf.md
• templates/batching/template-batching-config.md
• templates/caching/template-prefix-caching.md
• templates/benchmarking/template-latency-throughput-test.md
• templates/checklists/inference-review-checklist.md

Data

• data/sources.json — Curated external references

---

Related Skills

This skill focuses on inference-time performance. For related workflows:

• ai-llm - Prompting, fine-tuning, application architecture
• ai-rag - RAG pipeline construction and optimization
• ai-mlops - Deployment, monitoring, safety, and governance
• qa-observability - Performance monitoring and optimization
• ops-devops-platform - Infrastructure and platform operations

---

External Resources

See data/sources.json for:

• Serving frameworks (vLLM, TensorRT-LLM, DeepSpeed-MII)
• Quantization libraries (GPTQ, AWQ, bitsandbytes, LLM Compressor)
• FlashAttention, FlashInfer, xFormers
• GPU hardware guides and optimization docs
• Benchmarking frameworks and tools

---

Use this skill whenever the user needs LLM inference performance, cost reduction, or serving architecture guidance.