llm-performance

name	llm-performance
description	Enforces latency/throughput/cost rules for LLM-calling code. Use when code calls Anthropic/OpenAI/Gemini/Groq, defines system prompts, tool schemas, structured output schemas, streaming handlers, or agent loops — or when user mentions latency, TTFT, TPS, token budgets, cache hit rate, thinking tokens, or "make AI faster". Provider-impartial. Framework-agnostic.

LLM Performance

Metrics

TTFT	time to first token
TPS	tokens/sec generation throughput
NIT	input tokens/request
NOT	output tokens/response
NTT	thinking tokens before first output
ACL	avg context length across requests
Cache hit rate	% input tokens from cache

Diagnostic Map

Symptom	Fix	Reference
High TTFT, no thinking	restructure prompt for cache	prompt-caching.md
High TTFT, thinking on	disable/cap thinking	thinking-budgets.md
Long total, lots of output	compact schema; brevity prompt	output-token-reduction.md
Long total, many tool calls	reduce tool surface	tool-surface-design.md
Slow on every request	wrong model — route to faster	model-selection-and-routing.md
Blank screen	stream response + nested calls	streaming-patterns.md
Context grows monotonically	compact history every turn	memory-compaction.md
System prompt > 2000 tokens	caveman + verify with embeddings	caveman-compression.md
Sync side-effects in tools	move to background queue	background-queues.md
Bad output regenerated end-to-end	per-chunk validate, abort+retry	fail-fast-validation.md
Rapid-fire input (IDE)	buffer + cancel in-flight	request-buffering.md

Rules (priority order)

1. Fastest model that's good enough

Discover what's in use (read code, ask user, WebFetch provider docs — verify model names are current; treat training-data names as stale). Categorise workload: single task class, multiple distinct classes, or variable-complexity within one. Match strategy: downgrade / classification / self-upgrade. Propose, don't impose. Conservative on existing apps; optimistic on greenfield. → model-selection-and-routing.md

2. Minimise output tokens

Output = 2–4× input cost, dominant latency contributor.

// ❌ { "productName": "X", "productCategory": "Y", "isAvailable": true }
// ✅ { "n": "X", "c": "Y", "a": true }

Pair compact schema with code expansion step. Text → instruct brevity:

Answer in 1–2 sentences. No preamble. No "Sure! Here's...". Lead with answer.

max_tokens = safety net only (truncates mid-token). → output-token-reduction.md

3. Stream everything

Text, structured output, nested LLM calls. Arrays → JSONL, not JSON array.

// ❌ [{...}, {...}, {...}]   nothing renders until ]
// ✅ {...}\n{...}\n{...}     each line renders on emit

Sub-LLM calls (summariser, sub-agent) must stream too. → streaming-patterns.md

4. Disable/cap thinking

On by default in modern frontier models. Disable for simple tasks; cap budget for reasoning.

Provider	Param	Disable	Cap
Anthropic	thinking	{type: "disabled"}	{type: "enabled", budget_tokens: 1024}
OpenAI	reasoning_effort	"minimal"	"low"/"medium"/"high"
Gemini	thinking_config.thinking_budget	0	int cap

Some models (Gemini 2.5 Pro) cap-only. → thinking-budgets.md

5. Static → dynamic prompt order

Cache matches left-to-right. Order: system → tools → few-shot → RAG → history → user message.

Forbidden: dynamic content in system prompt.

// ❌ "system": "Time: 2026-04-25T19:45Z. Session: 7f2b. You are assistant."
// ✅ "system": "You are assistant.",
//    "messages": [{ "role": "user", "content": "Time: ...\nSession: ...\n\n<req>" }]

Audit system prompt for: timestamps, request/trace/session IDs, user identifiers, locales, A/B variants, nonces. All → user message. → prompt-caching.md

6. Use-case tools, not CRUD

Each tool call interrupts generation, risks cache miss. Each tool:

• Use-case shape, not data-shape
• Return only fields model needs
• Paginate (5 not 500)
• Pre-format (currency, dates, translations)

// ❌ flights.search + flights.get + flights.book → 3 round-trips
// ✅ bookFlight → 1 call

MCP = dev convenience. Audit tokens before production. → tool-surface-design.md

7. Compact history every turn

After each turn, summarise async with fast cheap model, replace history.

Must survive compaction: user goals, active constraints, referenced entities (IDs/names), unresolved tasks, preferences. Tool results = worst offenders, truncate hard. Sacrifices 1 cache turn, saves NIT every subsequent turn. → memory-compaction.md

8. Caveman system prompts

❌ "You are a helpful assistant that should always provide detailed answers"
✅ "Provide detailed answers."

Verify: cosine sim of embeddings > 0.95 → no meaning lost. Never compress safety rules, edge-case handling, wording-precise instructions. → caveman-compression.md

9. Background queues for side-effects

Loop contains only must-block-user work. Queue: side-effectful tools, compaction, embeddings, analytics, logging. Independent ops → parallel.

// ❌ const a = await t1(); const b = await t2(); const c = await t3();
// ✅ const [a,b,c] = await Promise.all([t1(), t2(), t3()]);

→ background-queues.md

10. Fail fast on bad structured output

Validate every streamed chunk. On failure: abort stream → append error to prompt → retry. Never buffer-then-validate-then-regenerate. → fail-fast-validation.md

11. Buffer rapid requests (IDE-like only)

Not for chat. Buffer + batch. Supersede → cancel in-flight. Tool-call boundaries = checkpoints. → request-buffering.md

DO / DON'T

Area	DO	DON'T
Model	fastest good-enough; route up	frontier "to be safe"
Schema	short keys, expand in code	human-readable keys per call
Stream	text + JSONL + nested	buffer until done
Tools	use-case, paginated, pre-formatted	CRUD passthrough, raw dumps
Prompt order	static → dynamic	timestamps/IDs in system prompt
Thinking	disable simple; cap reasoning	provider default
History	async compact every turn	append forever
Side-effects	queue + parallel	sequential await
Validation	per-chunk, abort+retry	buffer, parse, regenerate
MCP	dev only; audit before prod	ship as-is

Forbidden

// dynamic content top of prompt — kills cache
"system": "Request ID: 7f2b... You are assistant."

// verbose schema in streamed output
{ "userFullName": "...", "userEmailAddress": "...", "isUserActive": true }

// sequential await for independent calls
const a = await t1(); const b = await t2();

// sync side-effect in tool execute
await smtp.send(...); return "sent";  // queue it

// buffer-then-validate streamed structured output
const full = await response.text(); schema.parse(JSON.parse(full));

// thinking left at default for trivial tasks

// CRUD tools 1:1
[{name:"users.list"},{name:"users.get"},{name:"users.update"}]

// unbounded max_tokens on streaming UI

References (load on demand per Diagnostic Map)

• model-selection-and-routing.md — providers, OSS, routing strategies
• output-token-reduction.md — schema compaction, brevity, max_tokens
• streaming-patterns.md — SSE, JSONL, nested streaming
• tool-surface-design.md — use-case tools, MCP audit, sandboxes
• prompt-caching.md — provider matrix, structure, preheating
• thinking-budgets.md — per-provider config, dynamic budget
• memory-compaction.md — compaction prompts, cache tradeoff
• caveman-compression.md — rules, embedding verification
• background-queues.md — async tools, parallel fan-out
• fail-fast-validation.md — streaming validation, retry-with-feedback
• request-buffering.md — debounce, cancel, tool-call checkpoints