| name | token-diet |
| description | Production-ready token optimization: reduce costs 40–75% through retrieval pruning, smart caching, and model routing. Use whenever optimizing API costs, latency, or managing long context—especially for RAG pipelines, high-volume systems, multi-turn conversations, or when context exceeds 2K tokens. |
Token Diet v3.0 (Production-Ready)
Deployment-grade token optimization with execution order, ROI metrics, guardrails, and measurable outcomes.
Execution Flow (Order Matters)
Q → Retrieve → Prune → Cache → Route → Build Prompt → Compress → Call LLM → Measure → Update State
Why this order: Early pruning eliminates waste before caching/routing decisions. Compression happens last (post safety checks). Measurement feeds back into next iteration.
1. Retrieval Pruning (Highest ROI)
Goal: Never send unnecessary tokens.
Pipeline:
query → embed → topK(20) → dedupe → sentence-trim → topK(5–8)
Concrete rules:
- Drop near-duplicate chunks (>85% similarity)
- Keep only sentences containing query terms or answer-bearing context
- Hard cap: ≤1200 tokens total context
- If retrieval yields <5 relevant chunks, fetch 5 more before pruning
Win: 30–60% reduction immediately (no quality loss if done right).
2. Smart Caching (Implement Early)
Split prompt into 3 blocks:
[STATIC_SYS] → cache (reuse across requests)
[DYNAMIC_CTX] → no cache (changes per request)
[USER_INPUT] → no cache (one-off query)
Cache candidates:
- System instructions
- Schemas (JSON, SQL, output format)
- Tool definitions
- Few-shot examples (if static)
Decision rule:
if block_reused_in_next_N_requests → cache
else → inline (no caching overhead)
Implementation: Use API cache headers (cache_control: {"type": "ephemeral"} for Anthropic; similar for other providers).
3. Conditional Routing (Cheap-First Model Selection)
Decision tree:
if task ∈ {format, extract, classify, summarize}
→ small model (Haiku, GPT-4o-mini, Gemini-Flash)
else if task ∈ {reason, synthesize, creative}
→ large model (Sonnet, GPT-4o, Gemini-Pro)
Safety guard: If confidence low or error detected → retry on stronger model (don't downgrade past first attempt).
Result: 60–80% cost reduction on simple tasks without quality hit.
4. Tool Batching (Avoid Round-Trips)
Bad (5 calls + reread context 5x):
get_user(1) → get_user(2) → get_user(3)
Good (1 call):
get_users([1, 2, 3])
Also batch:
- Embedding calls (vector DBs)
- Database reads
- Validation checks
5. Stateful Memory (Replace Naive Summarization)
Structure state as JSON:
{
"facts": ["user_id=42", "role=admin"],
"constraints": ["budget_limit=$1000", "deadline=2026-04-30"],
"open_tasks": ["approval_pending", "docs_missing"],
"summary": "(optional) Human-readable 150-token version"
}
When to regenerate:
- If contradiction detected between new facts and old state
- After ≥5 turns
- If summary token cost > fresh state token cost
Why: Structured state lets LLM reason precisely. Optional summary is for readability/debugging, not required.
6. Linguistic Compression (Last-Mile Only)
Apply only right before the API call:
"Please analyze the dataset and provide a concise summary"
→ "Analyze data. Summarize concise."
DO NOT compress:
- Code (breaks syntax)
- JSON schema (breaks parsing)
- Legal/safety text (liability risk)
- URLs, identifiers
Benefit: ~15% token reduction on prose prompts; safe only if applied surgically.
7. Prompt Assembly Template
Minimal structure (reorder as needed):
[SYS — cached]
Context:
{pruned_context_1200_tokens_max}
State:
{facts + constraints (JSON)}
Task:
{compressed_user_input}
Why minimal: Every line you don't send saves tokens. No filler, no explanations that don't move the needle.
8. Measurement (Non-Optional)
Track per request:
tokens_in
tokens_out
latency_ms
cache_hit_rate (%)
retrieval_tokens_before / after
cost_usd
Add alerts:
- Context >1500 tokens → investigate retrieval
- Cache hit rate <70% → check static block design
- Cost spike >20% vs. baseline → disable routing/compression, debug
Why: Without measurement, you're flying blind. Guardrails only work if you see violations early.
9. Safety Guards (Prevent Silent Failure)
| Problem | Signal | Action |
|---|
| Answer quality drops | User feedback or eval score ↓ | Disable compression first, then increase context |
| Hallucination risk rises | Repeated factual errors | Increase context, not summary; rebuild state from raw history |
| Silent drift | State contradicts history | Rebuild state from scratch; check if constraints stale |
| Over-pruning | Query unmatched to results | Fetch more chunks before pruning; lower similarity threshold |
Golden rule: If unsure whether compression/routing broke something, disable it and rerun. Measure the difference. Only re-enable if safe.
Ultra-Compressed (Caveman Runtime Spec)
Retrieve → prune hard
Cache static
Route cheap→expensive
Batch all
State > summary
Compress last
Measure always
Guard drift
Net Effect (Realistic)
| Metric | Target | Conditions |
|---|
| Token reduction | 50–80% | All 9 steps applied in order |
| Latency | ↓20–50% | Caching + routing active |
| Cost | ↓40–75% | Routing to smaller models + cache hits >70% |
| Quality | Stable | Guards enforced; no compression on code/schema/legal |
Typical breakdown:
- Pruning: 30–40% reduction
- Caching: 20–30% reduction (if reuse >50%)
- Routing: 30–50% cost reduction (simple tasks to cheap model)
- Compression: 10–15% reduction (prose only)
- Combined: Multiplicative, not additive.
Common Mistakes
| Mistake | Result | Fix |
|---|
| Skip measurement | Invisible failures | Instrument from day 1 |
| Compress code/JSON | Syntax errors, parsing fails | Never compress structured data |
| Prune too aggressively | Relevant context lost → hallucinations | Keep ≥1200 tokens context; test quality on pruned versions |
| Cache everything | No room for dynamic context | Cache only truly static blocks |
| Summarize too early | Miss important detail → wrong answer | Use structured state; regenerate if contradiction |
| Ignore routing guardrails | Quality collapse on hard tasks | Always allow retry on stronger model |
Author
