aiase-token-economics

name	aiase-token-economics
description	Token economics and LLM API cost optimization — the 6-layer cost stack, 12 concrete reduction techniques, 7 API pricing mechanisms, model routing, semantic caching, and the AgentOpt model-pairing framework. Load when the user asks about LLM costs, token optimization, API pricing, prompt caching, routing, or batching.

Token Economics & Cost Optimization

From AIASE 2026 (NCKU), Week 8. A systematic model for reducing LLM costs without sacrificing quality.

The 6-Layer Cost Stack

Layer	Focus	Who Controls
L5	Product/User (pricing tiers, caps)	PM/Business
L4	Application (generation, context, architecture)	You — primary optimization
L3	API Pricing (7 contractual mechanisms)	Provider negotiation
L2	Inference Compute (vLLM, quantization)	Infra team
L1	Training Compute	Provider ($10M–$1B+ capex)
L0	Training Data (HTTP 402, Cloudflare Pay Per Crawl)	Emerging 2026

Your lever is L4. Optimize there first.

L0 Deep Dive: HTTP 402 & the Agent-Driven Web

The web was built around human traffic monetized by advertising. AI agents changed the equation:

Agent-driven web crawling has grown 73,000× relative to human browsing. Agents don't see ads.

This breaks the ad-revenue model for content providers. HTTP 402 ("Payment Required") — defined in 1991 and never widely used — is now being revived as the response code for charge-by-request content access.

How it works:

Agent → GET /article HTTP/1.1
Server → 402 Payment Required  (content is available, but costs per request)
Agent → GET /article + payment token
Server → 200 OK + content

Cloudflare's "Pay Per Crawl" is the first at-scale implementation: content providers opt in to charge per agent request. Cloudflare handles micropayment routing between the agent's host (e.g., OpenAI) and the content provider.

Why this matters for AI engineers:

• Datasets your RAG pipeline pulls from free web sources today may cost per-request in 12–24 months
• Budget L0 data access costs into agentic system design now
• Systems that crawl broadly (research agents, competitive intelligence) face the highest L0 exposure
• Mitigation: cache aggressively, use licensed datasets, prefer sources that publish structured feeds

Infrastructure implications: Agent-readable network infrastructure (authenticated request channels, micropayment settlement, per-request billing) is the next major build-out after GPU infrastructure. Watch this space for engineering opportunities.

12 Concrete Cost Reduction Techniques

Generation Layer

Technique	Compression	Trade
Caveman Pattern (SKILL.md soft constraints: "why use many token when few do trick")	~3×	Output terseness
Structured CoT (Hard Grammar) (GBNF grammar, logit masking)	22–43× on thinking tokens	Grammar specification burden; reasoning shifts to code comments
Reasoning Effort Knob (minimal/low/medium/high)	40–80% on simple tasks	−15pp accuracy on complex reasoning

Context Layer

Technique	Compression	Trade
Semantic Caching (vector similarity; 0.8 threshold)	68.8% fewer API calls	Embedding DB overhead; similarity threshold tuning
Context Compaction (context_management={"edits":[{"type":"compact_20260112"}]})	58–80% on conversation history	Risk of compacting critical rules; preserve system prompt separately
RAG vs. Long Context	25,000× cheaper at scale	Higher latency
Code Execution MCP (instead of passing data through context)	166× compression for bulk ops	MCP setup overhead

Architecture Layer

Technique	Cost Reduction	Trade
LLM Routing (difficulty-based model selection)	10–60×	Routing logic overhead (only 1.3% of savings)
Batch API (non-real-time)	50% discount	24-hour latency tolerance
Priority API	+150% cost	Required for latency-critical paths
Multi-Agent Systems	7–15× cost multiplier	Only justified when task value far exceeds cost
Self-Hosting (Ollama dev, vLLM prod)	Break-even at ~600M–1.2B tokens/month	Engineer time ($750–3K/month) often exceeds hardware

7 API Pricing Mechanisms (L3)

Mechanism	You Provide	You Get
Time	Accept 24hr delay	−50% (Batch)
Memory	Stable system prompt prefix	Cache reads at −90%
Capability	Accept weaker model	5–51× price gap (Haiku vs. Opus)
Commitment	Prepay/flat-rate	Predictable costs
Prediction	Provide likely output (Predicted Outputs)	Free tokens if prediction accepted
Format	Structured output (JSON schema)	Avoid retry loops
Customization	Fine-tune investment	Long-term unit cost reduction

Predicted Outputs Deep Dive

• Client provides expected output (e.g., unmodified code during refactor)
• Inference validates in parallel; unchanged regions are free
• Wrong predictions still charge output tokens
• Best for: code editing where 90% of file is unchanged

Prompt Caching Priority

Always implement first — lowest engineering cost, immediate ROI.

Rules for maximizing cache hit rate:

1. Keep the system prompt identical across requests (any change invalidates the cache)
2. Tool definition changes invalidate the entire prefix cache (tool schemas sit at hierarchy start)
3. Stable prefix → cache reads cost ~90% less than cache writes

ROI Priority Sequence

1. Prompt Caching — break-even on first cache read
2. Batch + Routing — non-real-time workloads + heterogeneous tasks
3. Caveman + Semantic Cache — output compression + query deduplication
4. Context Compaction / RAG — require mature eval infrastructure
5. Structured CoT — maximum compression, maximum specification burden
6. Self-Hosting — appropriate only for 600M+ tokens/month

AgentOpt: Model Pairing (Counterintuitive Findings)

Strongest model ≠ best placement. From Columbia DAPLab research:

Task	Best Combo	Finding
HotpotQA	Weak Planner + Strong Executor	Weak planner delegates instead of doing everything
MathQA	Strong Agent + Weak Critic	Critic choice barely matters; weaker critics save 118×
BFCL v3	3 models at 70% quality	Same accuracy as Opus solo, 32× cheaper

Why strong Planner fails: Opus is too competent — it refuses to delegate, doing everything itself and burning tokens.

Anthropic Advisor Tool pattern:

• Haiku runs main loop (cheap)
• Opus serves as advisor, called only on critical moments
• Result: Better than Haiku solo, cheaper than Sonnet baseline

Stack Amplification (Multiplicative)

Batch (×0.5) → Prompt Cache (×0.28) → Routing (×0.6) → Caveman (×0.7) → Structured CoT (×0.4)
= ~40× total cost reduction

Cost: engineering complexity + eval burden per layer.

Anti-Patterns

Pitfall	Impact
Tokenizer drift	Opus 4.7 new tokenizer adds 35% tokens vs. 4.6 — effective cost rises even at same per-token rate
Tool definition overhead	5 MCP servers = 58 tools = 55K tokens overhead before conversation starts
Reasoning displacement	Structured CoT doesn't reduce total tokens — shifts overhead to code comments
Silent multi-agent failure	Token usage explains 80% of performance variance; no built-in failure detection
Cache invalidation	Tool definition changes blow up the entire prefix cache

Observability Tools

• Langfuse — MIT license, self-hostable, open-source
• Helicone — proxy-based, minimal integration
• LangSmith — best for LangChain/LangGraph
• CloudZero/Finout — enterprise cross-provider cost allocation

Key Metrics to Track

1. Cost per feature / user / session (cost attribution)
2. P50/P95 latency (detect silent retries)
3. Cache hit rate by type (prompt vs. semantic)
4. Tool definition overhead (per-request schema tokens)
5. Reasoning token consumption (detect over-thinking)
6. Error/retry rate (agent loop control signals)

---

"Every optimization mechanism is a trade; no compression is free — only honest exchanges."

See also: [[aiase-multi-agent]] for multi-agent cost architecture, [[aiase-rag]] for RAG vs. long-context cost tradeoffs.