| name | aiase-token-economics |
| description | Token economics and LLM API cost optimization — the 6-layer cost stack, 12 concrete reduction techniques, 7 API pricing mechanisms, model routing, semantic caching, and the AgentOpt model-pairing framework. Load when the user asks about LLM costs, token optimization, API pricing, prompt caching, routing, or batching. |
Token Economics & Cost Optimization
From AIASE 2026 (NCKU), Week 8. A systematic model for reducing LLM costs without sacrificing quality.
The 6-Layer Cost Stack
| Layer | Focus | Who Controls |
|---|
| L5 | Product/User (pricing tiers, caps) | PM/Business |
| L4 | Application (generation, context, architecture) | You — primary optimization |
| L3 | API Pricing (7 contractual mechanisms) | Provider negotiation |
| L2 | Inference Compute (vLLM, quantization) | Infra team |
| L1 | Training Compute | Provider ($10M–$1B+ capex) |
| L0 | Training Data (HTTP 402, Cloudflare Pay Per Crawl) | Emerging 2026 |
Your lever is L4. Optimize there first.
L0 Deep Dive: HTTP 402 & the Agent-Driven Web
The web was built around human traffic monetized by advertising. AI agents changed the equation:
Agent-driven web crawling has grown 73,000× relative to human browsing. Agents don't see ads.
This breaks the ad-revenue model for content providers. HTTP 402 ("Payment Required") — defined in 1991 and never widely used — is now being revived as the response code for charge-by-request content access.
How it works:
Agent → GET /article HTTP/1.1
Server → 402 Payment Required (content is available, but costs per request)
Agent → GET /article + payment token
Server → 200 OK + content
Cloudflare's "Pay Per Crawl" is the first at-scale implementation: content providers opt in to charge per agent request. Cloudflare handles micropayment routing between the agent's host (e.g., OpenAI) and the content provider.
Why this matters for AI engineers:
- Datasets your RAG pipeline pulls from free web sources today may cost per-request in 12–24 months
- Budget L0 data access costs into agentic system design now
- Systems that crawl broadly (research agents, competitive intelligence) face the highest L0 exposure
- Mitigation: cache aggressively, use licensed datasets, prefer sources that publish structured feeds
Infrastructure implications: Agent-readable network infrastructure (authenticated request channels, micropayment settlement, per-request billing) is the next major build-out after GPU infrastructure. Watch this space for engineering opportunities.
12 Concrete Cost Reduction Techniques
Generation Layer
| Technique | Compression | Trade |
|---|
| Caveman Pattern (SKILL.md soft constraints: "why use many token when few do trick") | ~3× | Output terseness |
| Structured CoT (Hard Grammar) (GBNF grammar, logit masking) | 22–43× on thinking tokens | Grammar specification burden; reasoning shifts to code comments |
| Reasoning Effort Knob (minimal/low/medium/high) | 40–80% on simple tasks | −15pp accuracy on complex reasoning |
Context Layer
| Technique | Compression | Trade |
|---|
| Semantic Caching (vector similarity; 0.8 threshold) | 68.8% fewer API calls | Embedding DB overhead; similarity threshold tuning |
Context Compaction (context_management={"edits":[{"type":"compact_20260112"}]}) | 58–80% on conversation history | Risk of compacting critical rules; preserve system prompt separately |
| RAG vs. Long Context | 25,000× cheaper at scale | Higher latency |
| Code Execution MCP (instead of passing data through context) | 166× compression for bulk ops | MCP setup overhead |
Architecture Layer
| Technique | Cost Reduction | Trade |
|---|
| LLM Routing (difficulty-based model selection) | 10–60× | Routing logic overhead (only 1.3% of savings) |
| Batch API (non-real-time) | 50% discount | 24-hour latency tolerance |
| Priority API | +150% cost | Required for latency-critical paths |
| Multi-Agent Systems | 7–15× cost multiplier | Only justified when task value far exceeds cost |
| Self-Hosting (Ollama dev, vLLM prod) | Break-even at ~600M–1.2B tokens/month | Engineer time ($750–3K/month) often exceeds hardware |
7 API Pricing Mechanisms (L3)
| Mechanism | You Provide | You Get |
|---|
| Time | Accept 24hr delay | −50% (Batch) |
| Memory | Stable system prompt prefix | Cache reads at −90% |
| Capability | Accept weaker model | 5–51× price gap (Haiku vs. Opus) |
| Commitment | Prepay/flat-rate | Predictable costs |
| Prediction | Provide likely output (Predicted Outputs) | Free tokens if prediction accepted |
| Format | Structured output (JSON schema) | Avoid retry loops |
| Customization | Fine-tune investment | Long-term unit cost reduction |
Predicted Outputs Deep Dive
- Client provides expected output (e.g., unmodified code during refactor)
- Inference validates in parallel; unchanged regions are free
- Wrong predictions still charge output tokens
- Best for: code editing where 90% of file is unchanged
Prompt Caching Priority
Always implement first — lowest engineering cost, immediate ROI.
Rules for maximizing cache hit rate:
- Keep the system prompt identical across requests (any change invalidates the cache)
- Tool definition changes invalidate the entire prefix cache (tool schemas sit at hierarchy start)
- Stable prefix → cache reads cost ~90% less than cache writes
ROI Priority Sequence
- Prompt Caching — break-even on first cache read
- Batch + Routing — non-real-time workloads + heterogeneous tasks
- Caveman + Semantic Cache — output compression + query deduplication
- Context Compaction / RAG — require mature eval infrastructure
- Structured CoT — maximum compression, maximum specification burden
- Self-Hosting — appropriate only for 600M+ tokens/month
AgentOpt: Model Pairing (Counterintuitive Findings)
Strongest model ≠ best placement. From Columbia DAPLab research:
| Task | Best Combo | Finding |
|---|
| HotpotQA | Weak Planner + Strong Executor | Weak planner delegates instead of doing everything |
| MathQA | Strong Agent + Weak Critic | Critic choice barely matters; weaker critics save 118× |
| BFCL v3 | 3 models at 70% quality | Same accuracy as Opus solo, 32× cheaper |
Why strong Planner fails: Opus is too competent — it refuses to delegate, doing everything itself and burning tokens.
Anthropic Advisor Tool pattern:
- Haiku runs main loop (cheap)
- Opus serves as advisor, called only on critical moments
- Result: Better than Haiku solo, cheaper than Sonnet baseline
Stack Amplification (Multiplicative)
Batch (×0.5) → Prompt Cache (×0.28) → Routing (×0.6) → Caveman (×0.7) → Structured CoT (×0.4)
= ~40× total cost reduction
Cost: engineering complexity + eval burden per layer.
Anti-Patterns
| Pitfall | Impact |
|---|
| Tokenizer drift | Opus 4.7 new tokenizer adds 35% tokens vs. 4.6 — effective cost rises even at same per-token rate |
| Tool definition overhead | 5 MCP servers = 58 tools = 55K tokens overhead before conversation starts |
| Reasoning displacement | Structured CoT doesn't reduce total tokens — shifts overhead to code comments |
| Silent multi-agent failure | Token usage explains 80% of performance variance; no built-in failure detection |
| Cache invalidation | Tool definition changes blow up the entire prefix cache |
Observability Tools
- Langfuse — MIT license, self-hostable, open-source
- Helicone — proxy-based, minimal integration
- LangSmith — best for LangChain/LangGraph
- CloudZero/Finout — enterprise cross-provider cost allocation
Key Metrics to Track
- Cost per feature / user / session (cost attribution)
- P50/P95 latency (detect silent retries)
- Cache hit rate by type (prompt vs. semantic)
- Tool definition overhead (per-request schema tokens)
- Reasoning token consumption (detect over-thinking)
- Error/retry rate (agent loop control signals)
"Every optimization mechanism is a trade; no compression is free — only honest exchanges."
See also: [[aiase-multi-agent]] for multi-agent cost architecture, [[aiase-rag]] for RAG vs. long-context cost tradeoffs.
