| name | llm-caching |
| description | Optimize LLM costs and latency through KV caching and prompt caching. Use when (1) structuring prompts for cache hits, (2) configuring API cache_control for Anthropic/Cohere/OpenAI/Gemini, (3) setting up self-hosted inference with vLLM/SGLang/Ollama, (4) building agentic workflows with prefix reuse, (5) designing batch processing pipelines, or (6) understanding cache pricing and tradeoffs. |
LLM Caching
Maximize KV cache reuse to reduce costs and latency.
Core Concept
LLMs compute Key (K) and Value (V) vectors for each token during inference. These encode the model's "understanding" of context. Caching avoids recomputation.
Level 1: KV Cache (inference) - Within one generation, reuse previous tokens' K,V
Level 2: Prompt Cache (API) - Across requests, persist KV state server-side
Level 3: Prefix Sharing (batch) - Across users/requests, share common prefixes
The Golden Rule
Static content first, variable content last.
[System prompt] <- cacheable, same every request
[Tool definitions] <- cacheable
[Few-shot examples] <- cacheable (same order!)
[Reference documents] <- cacheable if stable
[User message] <- variable, at the end
Cache hits require the prefix (beginning) to match exactly. Any difference breaks caching for everything after.
Prompt Structure Template
┌─────────────────────────────────────┐
│ 1. System instructions (static) │ <- cache_control
├─────────────────────────────────────┤
│ 2. Tool definitions (static) │ <- cache_control
├─────────────────────────────────────┤
│ 3. Few-shot examples (static) │ <- cache_control
├─────────────────────────────────────┤
│ 4. Documents/context (semi-static) │ <- cache_control if reused
├─────────────────────────────────────┤
│ 5. Conversation history (growing) │ <- cache after N turns
├─────────────────────────────────────┤
│ 6. Current user message (variable) │ <- no caching
└─────────────────────────────────────┘
