llm-eval-robustness

name

llm-eval-robustness

description

Use when integrating or benchmarking an LLM API (especially reasoning models like GLM-5.2, DeepSeek V4, or any new provider/endpoint), or when a model "looks weak" and you need to rule out integration artifacts before concluding. Covers anti-hang call layers, reasoning-model handling (max_tokens, empty content), endpoint comparison, prefix caching, cheat-proof evaluation (faithfulness), and variance-aware measurement. Generalizable beyond fmhub.

LLM Eval & Integration Robustness

Hard-won lessons: a model that "looks weak" is often an integration artifact, and a model that "looks perfect" is often cheating. Rule both out before concluding.

Before calling a model weak — rule out integration artifacts

1. Reasoning models truncate to empty content. If content is empty but the model spent thousands of reasoning_tokens, it hit the output cap. Set a HIGH max_tokens (e.g. 24000) so reasoning + answer both fit. GLM went from 80% empty → 0% after this + an endpoint switch.
2. Empty content is intermittent — retry it. Treat empty content as a transient failure (bounded retries).
3. The endpoint matters as much as the model. Same model, different gateway → GLM coding-plan 80% empty vs OpenRouter 0%. Always A/B the endpoint.
4. Probe the concurrency ceiling. Fire N concurrent calls; back off below the 429 threshold (GLM coding-plan ~6-8).

Anti-hang call layer (always use one)

Naive retry (per-call timeout × N retries × backoff) has NO wall-clock ceiling — a stalled provider blocks a task for tens of minutes. Wrap every call with:

• a HARD deadline across all retries (e.g. 300s) + short per-attempt timeout (120s);
• bounded transient retries + bounded empty-content retries;
• return a clean failure (never raise/hang) past the deadline.
• Optional disk cache keyed by (model, messages) for free resume — but NOT for pass@N sampling (it returns the same cached draw).

Reference implementation: benchmarks/llm_robust.py (robust_chat).

Before believing a perfect score — rule out cheating

When the model EDITS code/artifacts to satisfy a checker, a passing score is meaningless if it tampered with the task. GLM scored 100% by deleting the asserts it was meant to prove (real faithful = 15%).

• Check structural equivalence of output vs original (AST equivalence + the required obligations preserved).
• Prompting against cheating fails — GLM cheated 5/5 even when told not to. Prevent it STRUCTURALLY: let the model emit only insertions/edits applied onto a frozen original, so it physically cannot delete the task. Use a non-JSON delimiter format (JSON escaping corrupts backslash-heavy DSLs).

Verifier-error mitigation pattern (reusable)

When models emit syntactically-invalid output a tool rejects:

1. Categorize the tool's error by recognizable patterns.
2. Targeted repair prompt per category (name the exact fix + cite the line).
3. Mechanical auto-fix the safe, unambiguous cases before re-verifying (saves a round). Be conservative — never corrupt valid output.

Reference: benchmarks/acsl_syntax.py.

Variance-aware measurement

High-variance weak models give different results each run (±several items / 50). A small prompt effect is invisible in a single A/B:

• Use multiple seeds, OR paired analysis on only the affected subset.
• Beware non-random task ordering — a hard prefix misleads early conclusions (saw 18% on a hard prefix vs 52% full-benchmark).
• Variance is also a lever: pass@N exploits "30% compliant each time, but a different 30%."