Implement caching for OpenRouter API responses to reduce cost and latency. Use when optimizing repeat queries, building RAG systems, or reducing API spend. Triggers: 'openrouter cache', 'cache llm responses', 'openrouter caching', 'reduce openrouter cost'.
Instrucciones de origen · Vista previa de solo lectura
name
openrouter-caching-strategy
description
Implement caching for OpenRouter API responses to reduce cost and latency. Use when optimizing repeat queries, building RAG systems, or reducing API spend. Triggers: 'openrouter cache', 'cache llm responses', 'openrouter caching', 'reduce openrouter cost'.
OpenRouter charges per token, so caching identical or similar requests can dramatically cut costs. Deterministic requests (temperature=0) with the same model and messages produce identical outputs -- these are safe to cache. This skill covers in-memory caching, persistent caching with TTL, and Anthropic prompt caching via OpenRouter.
Prerequisites
An OpenRouter API key (sk-or-v1-...) exported as OPENROUTER_API_KEY — see the openrouter-install-auth skill for setup
Python 3.8+ with the OpenAI SDK, plus the redis client package for the persistent cache; Node.js 18+ with the OpenAI SDK for the TypeScript variant in the references
A Redis server reachable at localhost:6379 for Persistent Cache with Redis (the in-memory LLMCache needs no infrastructure)
Deterministic request settings — caching is only safe at temperature=0
Instructions
Confirm the requests you want to cache are deterministic (temperature=0); non-zero temperatures produce different outputs each call and must never be cached.
Start with the In-Memory Cache: LLMCache plus cached_completion() gives you TTL expiry and hit/miss counters in a single process.
For multi-instance deployments, switch to Persistent Cache with Redis — redis_cached_completion() stores results under or:<sha256> keys with r.setex TTL expiry and falls through to a direct API call on a miss.
Build keys per Cache Key Design: include the model ID (with variants like :floor), messages, temperature, max_tokens, and top_p; exclude stream and the HTTP-Referer/X-Title headers.
For large static system prompts (RAG context), add cache_control: {"type": "ephemeral"} per Anthropic Prompt Caching via OpenRouter — cache reads bill at 0.1x the input rate.
Wire the Cache Invalidation table: flush per-model keys on model version updates, flush everything on system prompt changes, and let TTL handle the rest.
import redis, json, hashlib
r = redis.Redis(host="localhost", port=6379, db=0)
defredis_cached_completion(messages, model="openai/gpt-4o-mini", ttl=3600, **kwargs):
"""Cache in Redis with automatic TTL expiry."""
kwargs["temperature"] = 0# Must be deterministic
key = f"or:{hashlib.sha256(json.dumps({'m': model, 'msgs': messages, **kwargs}, sort_keys=True).encode()).hexdigest()}"
cached = r.get(key)
if cached:
return json.loads(cached)
response = client.chat.completions.create(model=model, messages=messages, **kwargs)
result = {
"content": response.choices[0].message.content,
"model": response.model,
"tokens": response.usage.prompt_tokens + response.usage.completion_tokens,
}
r.setex(key, ttl, json.dumps(result))
return result
Anthropic Prompt Caching via OpenRouter
Anthropic models on OpenRouter support prompt caching -- large system prompts are cached server-side, reducing input cost by 90% on cache hits.
# Mark large static content blocks with cache_control
response = client.chat.completions.create(
model="anthropic/claude-3.5-sonnet",
messages=[
{
"role": "system",
"content": [
{
"type": "text",
"text": "You are an expert. Here is the full source:\n" + large_context,
"cache_control": {"type": "ephemeral"}, # Cache this block
}
],
},
{"role": "user", "content": "What does the main() function do?"},
],
max_tokens=1024,
)
# First call: cache_creation_input_tokens charged at 1.25x# Subsequent: cache_read_input_tokens charged at 0.1x (90% savings)
Cached completion payloads returned without an API round-trip: {"content", "model", "usage"} from the in-memory cache or {"content", "model", "tokens"} from Redis
Redis keys of the form or:<sha256-of-canonical-request> that expire automatically via TTL
Hit/miss counters and a hit_rate figure you can use to justify the caching infrastructure
On Anthropic models, cache_creation_input_tokens billed at 1.25x on the first call and cache_read_input_tokens at 0.1x (90% savings) on subsequent hits
Examples
Two identical deterministic calls through the ResponseCache from the references — the second returns instantly from cache:
result1 = cached_completion("What is Python?") # [Cache MISS] key=3f8a92c1... (stored)
result2 = cached_completion("What is Python?") # [Cache HIT] key=3f8a92c1...print(f"Hit rate: {cache.hit_rate:.0%}") # Hit rate: 50%
More worked examples, including a TypeScript Redis-style cache: references/examples.md.
Error Handling
Error
Cause
Fix
Stale cache response
TTL too long
Reduce TTL or version cache keys
Cache miss storm
Cold start or invalidation
Warm cache with common queries at deploy
Redis connection error
Redis down
Fall through to direct API call
Non-deterministic cache
temperature > 0 cached
Only cache when temperature=0
Enterprise Considerations
Only cache deterministic requests (temperature=0) -- non-zero temperatures produce different outputs each time
Use Anthropic prompt caching for large system prompts (RAG context) -- 90% cost reduction on cache hits
Set TTL based on content freshness needs (30 min for dynamic, 24h for reference data)
Track cache hit rate to justify caching infrastructure cost
Use Redis or Memcached for multi-instance deployments; in-memory only works for single-process
Version cache keys when updating system prompts or switching model versions
