Caching strategies for LLM prompts including Anthropic prompt caching, response caching, and Cache-Augmented Generation (CAG) to cut latency and cost. USE WHEN you want to reduce LLM cost/latency by caching prompt prefixes or responses.
Caching strategies for LLM prompts including Anthropic prompt caching, response caching, and Cache-Augmented Generation (CAG) to cut latency and cost. USE WHEN you want to reduce LLM cost/latency by caching prompt prefixes or responses.
cluster
ai-agents-meta
version
1.0.0
origin
antigravity-awesome-skills (MIT)
risk
none
source
vibeship-spawner-skills (Apache 2.0)
date_added
"2026-02-27T00:00:00.000Z"
Prompt Caching
Caching strategies for LLM prompts including Anthropic prompt caching, response caching, and CAG (Cache Augmented Generation)
Capabilities
prompt-cache
response-cache
kv-cache
cag-patterns
cache-invalidation
Prerequisites
Knowledge: Caching fundamentals, LLM API usage, Hash functions
Pre-cache documents in prompt instead of RAG retrieval
When to use: Document corpus is stable and fits in context
// CAG: Pre-compute document context, cache in prompt
// Better than RAG when:
// - Documents are stable
// - Total fits in context window
// - Latency is critical
class CAGSystem {
private cachedContext: string | null = null;
private lastUpdate: number = 0;
async buildCachedContext(documents: Document[]): Promise<void> {
// Pre-process and format documents
const formatted = documents.map(d =>
`## ${d.title}\n${d.content}`
).join('\n\n');
// Store with timestamp
this.cachedContext = formatted;
this.lastUpdate = Date.now();
}
async query(userQuery: string): Promise<string> {
// Use cached context directly in prompt
const response = await client.messages.create({
model: "claude-sonnet-4-20250514",
max_tokens: 1024,
system: [
{
type: "text",
text: "You are a helpful assistant with access to the following documentation.",
cache_control: { type: "ephemeral" }
},
{
type: "text",
text: this.cachedContext!, // Pre-cached docs
cache_control: { type: "ephemeral" }
}
],
messages: [{ role: "user", content: userQuery }]
});
return response.content[0].text;
}
// Periodic refresh
async refreshIfNeeded(documents: Document[]): Promise<void> {
const stale = Date.now() - this.lastUpdate > 3600000; // 1 hour
if (stale) {
await this.buildCachedContext(documents);
}
}
}
// CAG vs RAG decision matrix:
// | Factor | CAG Better | RAG Better |
// |------------------|------------|------------|
// | Corpus size | < 100K tokens | > 100K tokens |
// | Update frequency | Low | High |
// | Latency needs | Critical | Flexible |
// | Query specificity| General | Specific |
Sharp Edges
Cache miss causes latency spike with additional overhead
Severity: HIGH
Situation: Slow response when cache miss, slower than no caching
Symptoms:
Slow responses on cache miss
Cache hit rate below 50%
Higher latency than uncached
Why this breaks:
Cache check adds latency.
Cache write adds more latency.
Miss + overhead > no caching.
// Non-blocking cache check
const cachedPromise = this.cache.get(cacheKey);
const llmPromise = this.queryLLM(prompt);
// Race: use cache if available before LLM returns
const cached = await Promise.race([
cachedPromise,
sleep(50).then(() => null) // 50ms cache timeout
]);
if (cached) {
// Cancel LLM request if possible
return cached;
}
// Cache miss: continue with LLM
const response = await llmPromise;
// Async cache write (don't block response)
this.cache.set(cacheKey, response).catch(console.error);
return response;
}
}
// Alternative: Probabilistic caching
// Only cache if query matches known high-frequency patterns
class SelectiveCache {
private patterns: Map<string, number> = new Map();
