| name | llm-cache-designer |
| description | Design caching layers in front of LLM inference — exact-match caches, semantic caches, provider prefix caching, and negative caching — so repeated questions never re-burn inference. Use this skill whenever the user mentions repeated/similar LLM queries, wants to cut inference cost or carbon for FAQ/support/RAG workloads, asks about semantic caching, or has any high-volume LLM endpoint. Part of Lean Agentic AI Skills; emits lean-findings.json plus a cache design. |
LLM Cache Designer
Advisor skill. Input: workload description (query distribution, tolerance for staleness, personalization degree), sample queries. Output: lean-findings.json (cache-opportunity findings) + a concrete cache design.
The greenest inference is the one that never runs. Cache layers convert repeated demand into a hash lookup: E drops to storage-read levels for every hit.
Subject type: emit subject.type: "ai-inference" in findings.
Design ladder (cheapest first)
- Provider prefix caching — restructure prompts static-first (with prompt-carbon-optimizer) so the provider caches the shared prefix. Zero infrastructure. Always check this first.
- Exact-match cache — hash(normalized prompt + model + params) → response. Right when inputs repeat verbatim (canned UI actions, programmatic calls). Key normalization matters: strip whitespace, stable JSON key order.
- Semantic cache — embed the query, serve cached response above a similarity threshold. Right for FAQ-shaped human input. Design decisions to specify: embedding model (small), threshold (start conservative ~0.95, tune with logged near-misses), scope keys (never share across users when responses contain personal data — say this loudly), TTL.
- Negative caching — cache "I can't help with X" style refusals/empty results too; repeated junk queries are also inference.
- Partial-pipeline caching in RAG/agents — cache retrieval results and tool outputs separately from generations; often higher hit rates than end-to-end.
What to specify in the design
Hit-rate expectation source (measure a week of logged queries for duplicate/near-duplicate rate — report the measured rate, don't guess), invalidation triggers (model version bump, KB update), staleness budget per content type, privacy scoping, and a hit/miss metric so the cache proves itself.
Trade-offs to always name
Staleness (wrong-but-fast answers), semantic false-positives (similar question, different intent — the support-bot horror story), cache storage is not free (usually trivial vs inference, but say it), personalization vs shareability tension.
Cost signal (countable)
Cost driver: billed inference tokens avoided by cache hits. Where a hit-rate can be measured from a week of query logs, cost_signal.observed names the measured near-duplicate rate; direction is qualitative — the cache converts that fraction of the driver to a storage lookup, but exact savings depend on the user's rate and cache running cost.
Not this skill's job
General app/HTTP caching (caching-strategy-designer), prompt structure (prompt-carbon-optimizer).
