optimize-llm-cost-latency

name	optimize-llm-cost-latency
description	Cuts LLM token cost and tail latency via context trimming, provider prompt caching on stable prefixes, model tiering/routing, semantic answer caching, batch APIs, and streaming, proving a measured before/after on cost-per-request and p50/p95 at equal output quality.
when_to_use	An LLM feature is too slow or too expensive, or usage is scaling and the bill/latency matters. Distinct from prompt-engineering (output quality), rag-pipeline (retrieval quality), and harden-llm-app-reliability (timeouts/retries/fallback).

When to Use

Reach for this skill when the problem is spend or speed, not what the model says:

• "Our LLM bill is $X/month and climbing — cut it"
• "This call takes 8s; make it feel fast"
• "Token usage per request is huge — trim it"
• "We send the same 6k-token system prompt on every call"
• "We re-answer near-identical questions all day"
• "We have a nightly batch of 50k classifications — too slow/expensive"

NOT this skill:

• Making the output better/more correct/structured → prompt-engineering
• Improving what gets retrieved into context (chunking/embeddings/reranking) → rag-pipeline
• Timeouts, retries, fallback models, circuit breakers when a call fails → harden-llm-app-reliability
• Blocking malicious instructions in inputs/tools → defend-llm-prompt-injection
• Generic HTTP/data response caching unrelated to LLM token economics → caching-strategy
• Cutting compute/storage/egress spend on the infra bill → cloud-cost-optimize

Steps

1. Measure first — no optimization without a baseline. Log per request: input_tokens, output_tokens, model id, end-to-end latency, and time-to-first-token (TTFT). Get exact counts from the provider's usage object or a count-tokens endpoint — never estimate from len(text)/4 for billing decisions. Compute $/req from current per-token prices and aggregate p50/p95 (means hide the tail that users actually feel). You cannot claim a win without these numbers before and after.

   # cost per request from the provider usage object (Anthropic-style)
   u = resp.usage  # input_tokens / output_tokens / cache_creation_input_tokens / cache_read_input_tokens
   cost = (u.input_tokens*IN + u.output_tokens*OUT
           + u.cache_creation_input_tokens*CACHE_WRITE   # ~1.25x input
           + u.cache_read_input_tokens*CACHE_READ) / 1e6 # ~0.1x input
   

2. Apply levers in ROI order — cheapest, highest-impact first. Do not jump to a fancy router before you've capped output and cached the prefix.

   Lever	Typical win	Effort	Use when
   Cap max_tokens + stop sequences	10–40% output cost	trivial	output runs longer than needed
   Context diet (trim history, drop dead few-shot)	20–60% input cost	low	long/growing prompts
   Prompt caching (cache stable prefix)	up to 90% on the cached portion, lower TTFT	low	long fixed system prompt / RAG docs reused across calls
   Streaming	TTFT 5–10x better (perceived)	low	user-facing chat/UI
   Model tiering/routing	50–95% on easy traffic	medium	mixed easy/hard requests
   Semantic cache	~100% on a cache hit	medium	repeated/near-duplicate queries
   Batch API	~50% on $	medium	offline, non-urgent jobs

3. Context diet — shrink input before you optimize how you send it. Every input token is paid on every call. (a) Cap history: keep the last N turns; once over a token budget, summarize older turns into a compact running summary instead of carrying raw transcript. (b) Cut dead few-shot examples: drop each example and re-run the eval — keep only those that move accuracy. Most prompts carry 3–4 examples that earn nothing. (c) Strip boilerplate, redundant instructions, and verbose tool schemas. Re-measure tokens after each cut.

4. Prompt caching — usually the single biggest win. Put the stable, large content first (system prompt, tool definitions, RAG documents, long instructions) and the variable part (the user's actual turn) last, then mark the boundary with the provider's cache control so the prefix is reused across requests. Order matters: the cache keys on an exact prefix match, so one moving token near the top busts the whole cache.

   # Anthropic: cache_control breakpoint on the stable prefix; variable user msg stays uncached
   system=[{"type":"text","text":BIG_STABLE_PROMPT,"cache_control":{"type":"ephemeral"}}]
   # OpenAI: automatic prefix caching — just keep the prefix byte-identical and put it first
   

   Cache reads are ~10% of input price; writes ~25% more than input — so caching pays off once a prefix is reused even a handful of times within its TTL (~5 min ephemeral). Verify hits via cache_read_input_tokens > 0.

5. Model tiering + routing — send easy work to the cheap model. Default the bulk of traffic to a small/fast model; escalate only when needed. Pick a deterministic router over an LLM-classifier router when you can (no extra call, no extra latency): route on cheap signals — input length, required output schema, presence of code/math, retrieval confidence, or an explicit difficulty flag. Use a tiny classifier model only when rules can't separate easy from hard. Always escalate on low-confidence or a validation failure rather than returning a bad cheap answer.

   def route(req):
       if req.tokens < 800 and not req.needs_reasoning: return SMALL   # ~1/10 the cost
       if req.needs_long_reasoning or req.high_stakes:  return LARGE
       return MID
   # escalate: if small-model output fails schema/confidence check -> retry once on LARGE
   

6. Semantic cache for repeated/near-duplicate queries. Embed the normalized query, look it up in a vector store; on cosine similarity ≥ ~0.95 return the cached answer (0 LLM tokens). Set a conservative threshold (too low → wrong cached answers), scope the key by anything that changes the answer (user/tenant/locale/tool-version), and TTL it so stale facts expire. Bypass the cache for personalized or time-sensitive responses. This is exact/near-exact answer reuse, layered on top of prompt caching (which reuses the prefix, not the answer).

7. Batch the offline work. Anything not blocking a user — nightly classification, backfills, evals, bulk summarization — goes through the provider Batch API (Anthropic Message Batches / OpenAI Batch) for ~50% off, accepting up to ~24h turnaround. Never batch interactive requests.

8. Stream to cut perceived latency. For any user-facing surface, stream tokens (stream=True / SSE) so first text shows in a few hundred ms instead of after the full generation. This doesn't reduce cost or total wall-clock, but it collapses TTFT — often the only latency users perceive. Combine with caching: a cached prefix also lowers real TTFT.

9. Re-measure and prove equal quality. Re-run the same metrics (step 1) after changes, and run your eval set to confirm output quality didn't regress (see Verify). A cost win that drops accuracy is not a win — report cost/latency and the quality delta together.

Common Errors

• Optimizing without a baseline. "Feels faster/cheaper" is not a number. Capture p50/p95 and $/req before touching anything, or you can't prove (or trust) the result.
• Estimating tokens with len/4. Fine for a rough guess, wrong for billing and for max_tokens budgeting. Use the provider usage object / count-tokens endpoint.
• Cache-busting the prefix. Putting a timestamp, request id, randomized example order, or the user message before the stable block invalidates the cache every call. Stable content first, byte-identical; variable content last.
• Caching the wrong span. Marking a tiny or rarely-reused prefix as cached pays the ~25% write premium for no reads. Cache large prefixes reused within the TTL; confirm with cache_read_input_tokens.
• Semantic-cache threshold too loose. A 0.85 similarity match returns a confidently wrong answer for a different question. Start ≥0.95, log near-miss hits, and never cache personalized/time-sensitive answers.
• Unscoped cache key. Caching by query text alone leaks one tenant/user/locale's answer to another. Include every dimension that changes the correct answer in the key.
• Router that always escalates. A misconfigured or overcautious router sends everything to the big model — you pay more and added a routing hop. Verify the cheap-model hit rate; if <50% on easy traffic, the rules are wrong.
• Unbounded max_tokens. Leaving it at the model max lets a runaway generation bill thousands of output tokens. Set it to the real ceiling and add stop sequences.
• Batching interactive traffic. Batch APIs trade latency for price; a user waiting on a 24h-window response is a broken product. Batch only offline work.
• Streaming to a non-interactive consumer. Streaming into code that just .join()s the whole thing adds complexity with zero benefit — and can hide errors that arrive mid-stream. Stream only where a human sees partial output.
• Trimming context until quality silently drops. Aggressive history truncation or cutting a load-bearing few-shot example tanks accuracy. Gate every cut behind the eval (step 9).

Verify

1. Baseline captured: before/after table exists with input_tokens, output_tokens, $/req, p50, p95, and TTFT — real measured numbers, not estimates.
2. Cost dropped: post-change $/req is measurably lower (state the %), computed from the provider usage object at current prices.
3. Latency dropped: p95 (and TTFT for user-facing paths) improved; report the actual ms, not "feels faster."
4. Cache is hot: cache_read_input_tokens > 0 on repeat requests, and the measured hit rate is reported. A prompt cache that never reads is dead config.
5. Router behaves: the cheap model handles the majority of easy traffic (hit-rate stated), and escalation fires on low-confidence/validation-fail (one logged example each).
6. Semantic cache is safe: a deliberately different query below threshold does not return a cached answer; key scoping (tenant/locale) prevents cross-leak.
7. Quality held: the eval set scores within tolerance of baseline (state the delta) — the cost/latency win did not regress accuracy. If it did, the change is rejected.

Done = a before/after table shows lower $/req and lower p50/p95 (with cache hit rate and cheap-model share stated), and the eval confirms output quality is unchanged at the new, cheaper, faster configuration.