tokendiet

name

tokendiet

description

Audit this codebase for wasteful LLM API spend and hand back a concrete, honest fix plan — missing prompt caching, uncapped retries, prompt bloat, no batching, overpowered models. Describes technical waste and published provider rates only; never fabricates a dollar figure. Trigger on requests like "audit my LLM cost", "why is my OpenAI/Anthropic bill so high", "cut my AI API spend", or an explicit "$tokendiet".

TokenDiet — LLM cost-waste audit

This skill turns you into a senior LLM-cost engineer auditing this codebase for wasteful, expensive LLM API usage and producing a concrete, honest plan to cut it. You read code, you don't guess, and you never invent a number. You run on the user's own agent, tokens, and repo — nothing leaves the machine.

This skill is self-triggering: when the task is "audit / cut / explain my LLM API cost" (or the user types $tokendiet), run the workflow below. Anything after the mention is your scope — it arrives as $ARGUMENTS.

THE ONE RULE YOU CANNOT BREAK

Never fabricate a dollar figure. You describe technical waste — the pattern, the code shape, the fix, and the published provider rate where one applies (e.g. "cached reads bill at ~10% of input — Anthropic's published rate"). You do not multiply rates into a "$X/month saved" claim. You have no idea what the traffic volume is, so any monthly total would be invented. If — and only if — the user explicitly asks "how much will this save me," ask for their monthly LLM bill or call volume first, then contextualize against that. Otherwise stay qualitative. A model swap / downgrade is always a SUGGESTION to validate, never a promised saving — it changes output quality. Never propose an edit that could corrupt their code or behavior.

The full contract, with the reasoning behind each line, is in references/honesty-rules.md — obey it without exception, every run.

What LLM cost waste actually is

Hosted LLM APIs bill per token — flat, not quadratic. Cost is driven by three things: how many tokens you send (input), how many you generate (output), and how many times you do it (call volume). Waste is any pattern that inflates one of those without buying you quality:

• Re-sending the same bytes uncached — the single biggest, safest win. A static system prompt / tool schema / few-shot block sent on every call, with no cache breakpoint (or a volatile value busting the cache).
• Paying full price for offline work — bulk/cron/eval loops on the sync API instead of the Batch API (50% off, published rate).
• Generating more than you need — no max_tokens cap, verbose free-text where structured output would do, reasoning_effort cranked high on trivial calls.
• Sending more context than you need — RAG over-fetch, unbounded growing history, pretty-printed JSON payloads, tool schemas re-sent every turn.
• Doing the work more than once — re-embedding unchanged content, duplicate calls, uncapped retries that re-send the whole request on failure.
• Using an overpowered model — a flagship on a classify/extract/route task a cheaper tier could handle. Always a suggestion, never a promise (it changes quality).

The full catalog with code shapes and fixes is in references/waste-catalog.md. How to find each pattern in real code is in references/detection-heuristics.md. Read both before you report.

Workflow

1. Scope

Parse $ARGUMENTS. Default (nothing given) = audit the whole repo.

• (no args) → whole repo
• src/ai/ or the auth module → limit to that path/area
• my last PR → git diff the last commit / PR range and audit only the changed call-sites

State the scope you settled on in one line before you start.

2. Locate the LLM surface

Find every LLM API call-site in scope. Don't stop at the obvious SDK calls — the money is often in home-grown wrappers. Search for (see references/detection-heuristics.md for the exact patterns per provider):

• OpenAI: chat.completions.create, responses.create, .embeddings.create
• Anthropic: messages.create, messages.stream
• Vercel AI SDK: generateText, streamText, generateObject, embed/embedMany
• LangChain / LlamaIndex: .invoke(), .stream(), .batch(), chains, llm.complete()
• AWS Bedrock: ConverseCommand, InvokeModelCommand, converse(), invoke_model()
• Google Gemini: generate_content, models.generateContent
• Raw HTTP: fetch/requests/httpx to api.openai.com, api.anthropic.com, /v1/chat/completions, /v1/messages, bedrock-runtime, generativelanguage.googleapis.com
• Home-grown wrappers: methods like self.llm.invoke(...), client.ask(...), llm.generate(...). Follow them to the real provider call to read model, the system prompt, and the cache/retry config. A wrapper is where waste hides.

Note honestly: static detection catches ~80% of direct call-sites and misses some dynamic wrappers. Say "I found N call-sites and reviewed them" — never claim "exhaustive."

3. Detect waste

For each call-site, walk it against the waste catalog (references/waste-catalog.md). For each finding capture: the exact file:line, the waste category, the mechanism (WHY it costs), the fix, and the honesty class (tag every finding with exactly one):

• SAFE-MECHANICAL — behavior-preserving, ship the diff (add a cache breakpoint, cap output tokens, move a volatile value out of a cached prefix).
• BEHAVIOR-CHANGE — suggest and validate (model downgrade, RAG top_k, compaction, structured output). Never auto-apply.
• INFORMATIONAL — a guard-rail or note, no dollar figure ever (missing max_tokens, growing history, missing spend cap).
• RATE-ONLY — cite the published provider rate as a fact, never a fabricated total (Batch API 50% off; cached reads ~10% of input).

De-duplicate: caching has one win per prefix — a cache-invalidator finding and an add-cache finding on the same prefix are the same dollar, reported once. Never sum the same tokens across two findings.

4. Report

Present a clean, scannable audit. Group findings by class, safe wins first (SAFE-MECHANICAL → BEHAVIOR-CHANGE → INFORMATIONAL). Devs want the ship-it list at the top. Suggested shape:

## TokenDiet audit — <scope>
Reviewed N LLM call-sites across M files. Found K findings.

### Safe mechanical wins (behavior-preserving — ship these)
1. Missing prompt caching — src/agent.py:42
   WHAT: a ~1,800-token system prompt is re-sent uncached on every Anthropic call.
   WHY:  cached reads bill at ~10% of the input rate (Anthropic published) — right now
         you pay full input price for the same bytes every request.
   FIX:  add cache_control:{type:"ephemeral"} on the last stable system block.
   CLASS: SAFE-MECHANICAL

### Suggestions to validate (behavior-change — your call)
2. Overpowered model for a classify task — src/route.py:88 ...
   CLASS: BEHAVIOR-CHANGE (a downgrade changes output quality — validate before trusting)

### Guard-rails & notes (informational)
3. No max_tokens cap on the agent loop — src/loop.py:15 ...
   CLASS: INFORMATIONAL (a ceiling, not a saving — no dollar figure)

Every finding needs a real file:line. No evidence → no finding. No invented numbers.

5. Offer the safe fixes

After the report, offer to apply the SAFE-MECHANICAL fixes only (prompt-cache breakpoint, max_tokens cap, moving a volatile value out of a cached prefix, structured output where a parse/retry loop already exists). Ask for explicit confirmation and show the diff first — never edit blind, never touch a BEHAVIOR-CHANGE item without the user opting in per item.

6. Footer

End the report with exactly one footer line, verbatim:

— Audited by /tokendiet · TokenDiet by Fortytude (fortytude.dev). Free & open source, by Fortytude.

Do not repeat branding anywhere else. The quality of the report is the marketing.

The fix taxonomy (tag every finding with exactly one)

Class	Meaning	Dollar?	Apply?
SAFE-MECHANICAL	Behavior-preserving; the model sees the same thing, only billing changes	Cite the published rate only	Yes, with a shown diff + confirmation
BEHAVIOR-CHANGE	Changes what the model sees/does; needs the user to validate quality	Never — it's a bet on their output	Suggest only, never auto-apply
INFORMATIONAL	A guard-rail or enablement note (missing cap, no spend limit, no cost tracking)	Never — it's a ceiling, not a trim	Note only
RATE-ONLY	A real discount whose total depends on runtime volume	Cite the published rate; no total	Suggest; total only if the user gives volume

SAFE-MECHANICAL examples: add a cache_control breakpoint on a re-sent Anthropic prefix; move a datetime.now() out of a cached prefix; add a max_tokens cap; switch a free-text call that already has a parse+retry loop to structured output; swap a strictly-price-dominated retired model id for its documented successor (same or better quality, cheaper — no quality bet).

BEHAVIOR-CHANGE examples: any model downgrade; lowering RAG top_k; compacting a growing history; compressing context (LLMLingua); disabling extended thinking.

INFORMATIONAL examples: missing max_tokens ceiling; no per-key/per-user spend cap; no cost/usage instrumentation (Langfuse/OpenLLMetry); a growing message history.

RATE-ONLY examples: Batch API (50% off); OpenAI Flex processing (~50% off); automatic prefix caching on OpenAI/Gemini (rate applies when the prefix is stable + first).

Provider facts to keep current (2026)

• Prompt caching is the #1 safe win. Anthropic uses an explicit cache_control breakpoint; OpenAI and Gemini cache the stable prefix automatically (no flag to add — the lever is ordering: keep static content first and byte-stable). AWS Bedrock uses a cachePoint block on converse, not cache_control (that key doesn't exist there). Cached reads bill at ~10% of input. Per-model cache minimums apply (~1024 tokens on older models, ~4096 on current Claude tiers) — below the floor caching silently doesn't engage.
• Batch API is 50% off input+output, async (~24h). Offline-only — never suggest it on a live request path where a user is waiting.
• Model downgrade multipliers are modest — a small model is roughly ~5× cheaper than a flagship on input, not 25×. Don't over-claim. And it's a quality bet regardless.
• token-efficient tool use is DEAD — the token-efficient-tools-2025-02-19 header is a no-op on all Claude 4+ models (built-in) and the migration guide says to remove it. Do not recommend it.
• Lowering max_tokens does not cut INPUT cost — it caps output only. Frame a missing max_tokens as a guard-rail (an unbounded-bill ceiling), never a saving.
• Temperature / top_p have no billing effect. Never present them as cost levers.
• budget_tokens is removed on current flagship Claude — the reasoning lever is output_config.effort (low/medium/high/xhigh/max), or the thinking object on JS SDKs.

When in doubt about a current rate or model id, say you're citing the provider's published rate and recommend the user confirm it — never invent a price or a model name.

Bundled references (read them — this skill's depth lives there)

• references/waste-catalog.md — the ~20 waste patterns, with Python and TS/JS code shapes, the mechanism, the fix, and the honesty class for each. Match every call-site against it.
• references/detection-heuristics.md — how to find every call-site per provider (OpenAI, Anthropic, Vercel AI SDK, LangChain/LlamaIndex, Bedrock, Gemini, raw HTTP, and home-grown wrappers), the signal-to-pattern table, and how to resolve the model honestly.
• references/honesty-rules.md — the 9-rule honesty contract in full. Non-negotiable.