| name | ai-eval-efficiency-audit |
| description | Audit AI evaluation infrastructure for wasted compute โ full benchmark suites re-run on unchanged cases, LLM-as-judge grading without caching, oversized judge models, redundant eval passes per commit, and missing result reuse. Use this skill whenever the user shares eval harness configs or CI eval steps, complains that evals are slow or expensive, mentions LLM-as-judge costs, or runs benchmark suites on every change. Part of Lean Agentic AI Skills; emits lean-findings.json. |
AI Eval Efficiency Audit
Subject type: emit subject.type: "ai-inference" in findings.
Producer skill. Input: eval harness configs, CI eval steps, judge prompts, run logs with case counts and durations. Output: lean-findings.json.
Evaluation is inference that scales with development velocity, not user value โ every commit can trigger thousands of model calls to conclude "still fine." The measurement layer deserves the same efficiency discipline as the thing it measures, and almost nobody audits it.
Signatures
- Unchanged cases re-graded โ the full eval set runs on every commit with no content-hash gating; cases whose inputs, model, and prompt are identical re-burn generation and judging. High. Fix: cache results keyed on (case hash, model version, prompt hash); re-run only invalidated cases.
- LLM-as-judge without caching โ judge calls repeated on identical (output, rubric) pairs across runs. High: judge tokens often exceed candidate tokens. Fix: judge-result cache; deterministic judges (temperature 0) so the cache is sound.
- Oversized judge โ frontier-tier judge grading tasks a mid-tier judge scores identically. Medium-high. Fix: judge-agreement check on a sample (measure agreement, don't assume), then downshift where agreement holds.
- Everything on every commit โ full suite in CI where a smoke subset + nightly full run serves. High. Fix: stratified subset on PR (cover each category), full suite on merge/nightly โ same safety-net pattern as lean-ci-audit's test selection.
- Redundant multi-sample runs โ N samples per case for pass@k on tasks reported as pass@1; self-consistency sampling where single-shot is reported. Medium.
- Verbose eval outputs โ unbounded generation length in eval configs; candidates ramble, judges read the ramble. Medium. Fix: same output caps as production (pairs with prompt-carbon-optimizer).
- No eval telemetry โ nothing records tokens per eval run, so growth is invisible. Medium. Fix: per-run token/cost logging; pairs with sci-receipt for the eval pipeline itself.
Cost signal (countable)
Cost driver: eval tokens per run ร runs per week (candidate + judge, separately โ judges often dominate). Case counts, samples-per-case, and run frequency are all countable from configs and CI history; quote them in cost_signal.observed. Never claim a quality-safety margin without the agreement measurement.
Honesty rules
Subset selection trades regression coverage โ every subset recommendation ships with the full-run safety net named. Judge downshifts require a measured agreement rate on the user's own rubric, not vibes. Token counts real; per-eval energy directional.
Not this skill's job
Production inference tuning (llm-serving-optimizer, model-right-sizer), the CI pipeline around the evals (lean-ci-audit), designing the evals' correctness (the team's job โ this skill only makes them cheaper to run).
