implement-caching

name	implement-caching
source	botcore
description	Guides caching strategy selection and implementation across the full stack including HTTP caching, application-level caching (Redis, in-memory), frontend data caching (SWR, TanStack Query), LLM response caching (prompt caching, semantic caching), database query caching, cache invalidation patterns, and distributed cache architectures. Covers cache-aside, read-through, write-through, write-behind patterns, eviction policies (LRU/LFU), and agentic workflow caching considerations. Use when adding caching to an application, choosing a caching strategy, debugging stale data, optimizing API response times, reducing LLM costs, or designing distributed cache topologies. Triggers: cache, caching, Redis, CDN, TTL, cache invalidation, stale data, Cache-Control, ETag, SWR, TanStack Query, prompt caching, semantic cache, LRU, write-through, cache-aside, materialized view.
version	1.0.0
triggers	["cache","caching","Redis","CDN","TTL","cache invalidation","stale data","Cache-Control","ETag","SWR","TanStack Query","prompt caching","semantic cache","LRU","LFU","write-through","write-behind","cache-aside","read-through","materialized view","in-memory cache","distributed cache","eviction policy"]
portable	true

Implementing Caching

Expert guidance for caching strategy selection, implementation, and invalidation across HTTP, application, frontend, database, and LLM layers.

Capabilities

1. HTTP Caching -- Cache-Control headers, ETags, stale-while-revalidate, CDN edge caching, and s-maxage configuration
2. Application-Level Caching -- Redis patterns, in-memory caching, two-tier cache architectures, eviction policies (LRU/LFU), and connection pooling
3. Caching Patterns -- Cache-aside (lazy loading), read-through, write-through, write-behind (write-back), and refresh-ahead strategies
4. Frontend Data Caching -- SWR, TanStack Query, service workers, browser storage APIs, and optimistic updates
5. LLM Response Caching -- Anthropic prompt caching, semantic caching with embeddings, agentic plan caching, and workflow-level caching
6. Database Query Caching -- Materialized views, query result caching, connection pooling, and read replicas
7. Cache Invalidation -- TTL-based, event-driven, tag-based, versioned keys, and hybrid invalidation strategies
8. Distributed Caching -- Cache topology, replication, partitioning, consistency models, and failure handling

Routing Logic

Request type	Load reference
HTTP headers, CDN, Cache-Control, ETags, browser caching	references/http-and-cdn-caching.md
Redis, in-memory, LRU/LFU, eviction, two-tier cache	references/application-level-caching.md
SWR, TanStack Query, service workers, browser storage	references/frontend-data-caching.md
Prompt caching, semantic caching, LLM cost reduction, agentic caching	references/llm-and-agentic-caching.md
Materialized views, query caching, database read optimization	references/database-query-caching.md
TTL, event-driven, tag-based, versioned invalidation strategies	references/cache-invalidation.md

Core Principles

1. Cache Strategically, Not Universally

Not everything should be cached. Cache data that is read frequently, expensive to compute, tolerant of brief staleness, and unlikely to change between reads. Caching mutable, low-read data adds complexity without benefit.

Decision test: If the data is read 10x more than it is written and a few seconds of staleness is acceptable, it is a strong caching candidate.

2. Choose the Right Caching Pattern

Pattern	How it works	Best for
Cache-aside	App checks cache, on miss reads DB, writes to cache	General purpose; simple, widely understood
Read-through	Cache itself loads from DB on miss	Frameworks with cache-provider abstraction
Write-through	Writes go to cache and DB synchronously	Strong consistency requirements
Write-behind	Writes go to cache, async flush to DB	Write-heavy workloads; eventual consistency OK
Refresh-ahead	Cache proactively refreshes before expiry	Predictable access patterns; low-latency reads

Default recommendation: Start with cache-aside. It is the simplest, most portable, and gives the application full control over caching behavior.

3. Choose the Right Eviction Policy

Policy	Evicts	Best when
LRU (Least Recently Used)	Oldest-accessed entries	Recency predicts future access
LFU (Least Frequently Used)	Least-accessed entries	Popularity predicts future access (skewed workloads)
TTL (Time-To-Live)	Expired entries	Data has a known freshness window
Random	Random entries	Uniform access distribution

Default recommendation: Use allkeys-lru for Redis. It handles the common case (Pareto distribution) well. Add TTL as a safety net on all entries.

4. Layer Your Caches

Apply caching at multiple layers, each serving a different purpose:

Browser Cache -> CDN Edge -> API Gateway -> App In-Memory -> Redis -> Database

• Browser cache -- Eliminates network requests entirely for returning users
• CDN edge -- Serves static and semi-static content from the nearest PoP
• Application in-memory -- Sub-millisecond reads for hot data (bounded by process memory)
• Distributed cache (Redis) -- Shared across instances, survives restarts, millisecond reads
• Database -- Source of truth; materialized views and query caching reduce load

5. Invalidation is Harder Than Caching

"There are only two hard things in Computer Science: cache invalidation and naming things." Start with TTL-based expiration and layer on event-driven invalidation for critical data paths. Never rely solely on manual invalidation.

6. Agentic Workflow Considerations

When an AI agent is implementing or reasoning about caching:

• Identify the consistency requirement first -- Ask whether the consumer tolerates stale data and for how long before choosing a strategy.
• Prefer explicit TTLs over unbounded caching -- Every cache entry should have an expiration. Zombie entries from missing invalidation logic are a common production incident.
• Consider cache warming on deploy -- Cold caches after deployment cause latency spikes. Pre-populate critical entries.
• Log cache hit/miss ratios -- If hit ratio drops below 80%, the cache may be poorly sized or the key space too large.
• Test invalidation paths -- Cache bugs are often invisible until stale data causes user-facing issues. Include invalidation in integration tests.
• When caching LLM responses -- Evaluate whether prompt caching (provider-level, exact prefix match) or semantic caching (application-level, similarity match) is appropriate. Prompt caching is cheaper and simpler; semantic caching handles paraphrased queries.

Workflow

1. Identify -- Determine what data is a caching candidate (high read-to-write ratio, expensive to compute, tolerant of staleness).
2. Select layer -- Choose the appropriate cache layer(s): HTTP, CDN, in-memory, distributed, or database.
3. Select pattern -- Pick a caching pattern (cache-aside, read-through, write-through, write-behind) based on consistency needs.
4. Configure eviction -- Set eviction policy (LRU/LFU), max memory, and TTL values.
5. Implement invalidation -- Define how and when cached data is invalidated (TTL, event-driven, tag-based, versioned keys).
6. Instrument -- Add metrics for cache hit/miss ratio, latency, eviction rate, and memory usage.
7. Test -- Verify cache behavior under load, test invalidation correctness, and simulate cache failures.
8. Monitor -- Track hit ratios in production; tune TTLs and cache sizes based on observed patterns.

Quick Reference

Cache-Aside Pattern (Most Common)

async function getUser(userId: string): Promise<User> {
  // 1. Check cache
  const cached = await redis.get(`user:${userId}`);
  if (cached) return JSON.parse(cached);

  // 2. Cache miss -- read from database
  const user = await db.users.findById(userId);
  if (!user) throw new NotFoundError('User not found');

  // 3. Populate cache with TTL
  await redis.set(`user:${userId}`, JSON.stringify(user), 'EX', 300); // 5 min TTL

  return user;
}

async function updateUser(userId: string, data: Partial<User>): Promise<User> {
  // 1. Update database (source of truth)
  const user = await db.users.update(userId, data);

  // 2. Invalidate cache
  await redis.del(`user:${userId}`);

  return user;
}

HTTP Cache-Control Cheat Sheet

# Static assets (fingerprinted filenames)
Cache-Control: public, max-age=31536000, immutable

# HTML pages
Cache-Control: no-cache
ETag: "abc123"

# API responses (short-lived, revalidatable)
Cache-Control: private, max-age=0, must-revalidate
ETag: "v2-abc123"

# API responses (CDN-friendly)
Cache-Control: public, max-age=60, s-maxage=300, stale-while-revalidate=60

# Sensitive data (never cache)
Cache-Control: no-store

TTL Guidelines

Data type	Suggested TTL	Rationale
Static assets (versioned)	1 year (immutable)	Filename changes on content change
User profile	5-15 minutes	Moderate change frequency
Product catalog	1-5 minutes	Balances freshness and performance
Session data	Match session timeout	Must not outlive session
API rate limit counters	Window duration	Must be exact
LLM prompt cache	5 min (Anthropic default)	Cost vs. freshness tradeoff
Search results	30-60 seconds	High change frequency
Configuration/feature flags	30-60 seconds	Must propagate quickly

Cache Key Design

# Pattern: {entity}:{identifier}:{optional-variant}
user:usr_123
user:usr_123:profile
product:prod_456:en-US
search:sha256(query_params)
api:v2:users:list:page=1&limit=20

# Rules:
# - Use prefixed IDs for entity type clarity
# - Include locale/version when responses vary
# - Hash complex query parameters
# - Keep keys under 512 bytes
# - Use colons as delimiters (Redis convention)

Common Anti-Patterns

Avoid	Instead
Caching without TTL	Always set a TTL, even a long one, as a safety net
Caching mutable data with no invalidation	Implement event-driven invalidation for writes
Using object serialization for cache keys	Use deterministic string keys with consistent ordering
Storing entire objects when only fields are needed	Cache only the fields that are read
Unbounded in-memory cache	Set max size with LRU/LFU eviction
Cache stampede on expiration	Use locks, jitter, or stale-while-revalidate
Ignoring cache failures	Degrade gracefully; fall through to source of truth
Same TTL for all data types	Tune TTL per data type based on change frequency

Cache Stampede Prevention

// Mutex/lock pattern to prevent thundering herd
async function getWithLock(key: string, fetchFn: () => Promise<string>): Promise<string> {
  const cached = await redis.get(key);
  if (cached) return cached;

  const lockKey = `lock:${key}`;
  const acquired = await redis.set(lockKey, '1', 'EX', 10, 'NX');

  if (acquired) {
    try {
      const value = await fetchFn();
      await redis.set(key, value, 'EX', 300);
      return value;
    } finally {
      await redis.del(lockKey);
    }
  }

  // Another process holds the lock; wait and retry
  await sleep(50);
  return getWithLock(key, fetchFn);
}

Checklist

Before shipping a caching implementation, verify:

• Caching candidates identified (high read:write ratio, expensive, staleness-tolerant)
• Caching pattern selected and documented (cache-aside, read-through, write-through, write-behind)
• TTL set on every cache entry (no unbounded entries)
• Eviction policy configured (LRU/LFU with max memory limit)
• Cache invalidation strategy implemented and tested
• Cache key design is deterministic, namespaced, and under 512 bytes
• Cache stampede prevention in place (locks, jitter, or stale-while-revalidate)
• Graceful degradation on cache failure (falls through to source of truth)
• Cache hit/miss ratio instrumented and alerted (target > 80% hit ratio)
• HTTP Cache-Control headers set appropriately for each response type
• Static assets served with immutable + long max-age + fingerprinted filenames
• Sensitive data excluded from caching (no-store on PII, auth tokens)
• Cache warming strategy for cold starts / deployments considered
• Serialization format chosen (JSON for debuggability, MessagePack/Protobuf for performance)
• Load tested with cache cold, warm, and under invalidation pressure

When to Escalate

• Distributed consistency requirements -- If the system requires strong consistency across cache nodes (not just eventual), consult with the architecture team. Distributed cache coherence protocols add significant complexity.
• Cache infrastructure sizing -- When Redis memory exceeds available RAM, data partitioning (clustering) and eviction tuning require infrastructure expertise. Estimate working set size before provisioning.
• Multi-region cache replication -- Cross-region cache replication introduces latency tradeoffs and conflict resolution challenges. Requires infrastructure and networking review.
• Cache-related data incidents -- If stale cached data causes user-facing data integrity issues (e.g., showing another user's data, stale financial data), treat as a severity-1 incident and involve the platform team.
• LLM caching cost optimization -- When prompt caching savings plateau and semantic caching is under consideration, evaluate embedding model costs, vector DB infrastructure, and cache hit quality thresholds with the ML platform team.
• Framework-level caching abstractions -- If the framework provides built-in caching (e.g., Next.js ISR, Rails fragment caching), prefer framework conventions over custom implementations and consult framework documentation.