reliability-benchmark

name

reliability-benchmark

description

Run a first-party reliability benchmark comparing two or more LLMs on hard, trap-laden tasks through one identical harness, graded by reading and scored pass^k. Use when the user wants to benchmark, compare, or A/B models for reliability (not raw capability), test a new model release against a baseline, or reproduce the Kimi K3 vs Opus study. Triggers on "benchmark these models", "compare model X vs Y", "run the reliability benchmark", "test <model> against <model>", "is <model> actually as good as its scores".

Reliability Benchmark

You (the agent) run an interactive, honest model comparison for the user. The whole premise is that a vendor score and a first-party score are different measurements, so the harness must be identical across models and the grading must be done by reading, not by regex. Walk the user through it step by step. Do not skip the guards; every one exists because something produced confident, well-formed, completely fake data.

The runnable kit lives in harness/ next to this skill's repo: pi_sdk_runner.mjs (the runner), grade.py (blind grader + MDE), finish_matrix.sh (auto-filler), and tasks/ (easy.json, hard/, advanced/). Prompts and ground truth are in PROMPTS.md.

---

Step 1 - Which models?

Ask the user which models they want to compare, if they have not already said. You need at least two (a candidate and a baseline). Record each as a display name plus the id its provider expects. Recommend always including a strong frontier baseline (e.g. anthropic/claude-opus-4.8) so "36% failure" has something to mean.

Step 2 - How should each model connect?

For each model, ask how the user wants to reach it, and set up that route. Present these options:

Option	Best for	What it needs	Model id form
OpenRouter (recommended)	Any model, one key, cross-vendor	OPENROUTER_API_KEY	moonshotai/kimi-k3, anthropic/claude-opus-4.8
Claude Code	Anthropic models on a Claude subscription (no API key)	logged-in Claude Code / ANTHROPIC creds	claude-opus-4-8, claude-sonnet-5
Kimi For Coding	Kimi models on a Kimi subscription	KIMI_API_KEY (a sk-kimi-... subscription key)	k3, kimi-for-coding
Custom endpoint	Any self-hosted / other vendor	base URL + key	provider-specific

Route details to apply once the user picks:

• OpenRouter - set OPENROUTER_API_KEY. Pass --provider openrouter and the full vendor/model ids to the runner. This is the recommended default because one key reaches every model and the runner can verify the served model per request.
• Claude Code - Anthropic models answer through the logged-in Claude Code session or ANTHROPIC_API_KEY / ANTHROPIC_BASE_URL. Cost is billed to the subscription, so the CLI's reported dollar cost is Anthropic pricing for whatever model answered; recompute cost from tokens for any non-Anthropic model reached this way.
• Kimi For Coding - set KIMI_API_KEY and point an Anthropic-shaped client at https://api.kimi.com/coding (it appends /v1/messages); OpenAI-shaped at .../coding/v1. Models: k3 (1M ctx), kimi-for-coding (K2.7). Reasoning is always on and cannot be disabled, so "thinking vs non-thinking" comparisons against other models are not apples-to-apples; say so in the writeup. Do not benchmark through a subscription UI/plan - providers sometimes serve quantised weights on subscription tiers. Route through the API.
• Custom endpoint - collect the base URL and key. If it is Anthropic-compatible use ANTHROPIC_BASE_URL; if OpenAI-compatible use OPENAI_BASE_URL. Confirm the model id the endpoint expects.

Keys live in a git-ignored .env. Never write a key into the repo, the vault, or a committed file. Confirm each key is set before running.

Step 3 - Configure the run

Ask three things:

1. Which tasks? Options: the easy control (10 tasks, expect ties - run this first so the user sees the tie baseline), the hard traps (H01-H12, where models separate), the advanced four (A01-A04), or a specific subset. Some hard/advanced tasks are agentic and need a real repo (see below).
2. Chat or agentic? Chat tasks (code inlined) run anywhere with -n 5. Agentic tasks need --tools and --cwd <repo> pointing at a real codebase. The agentic H-tasks were written against a Postgres chat app with a governance layer; to reproduce them exactly the user needs a comparable repo, otherwise stick to the chat tasks and the easy control.
3. How many runs? Default n=5. Fewer than 5 cannot expose a pass^k gap; more than ~5 saturates (spend extra budget on more tasks, not more runs).

Step 4 - Run

Chat tasks:

node harness/pi_sdk_runner.mjs \
  --glob 'tasks/hard/H04*.md,tasks/hard/H10*.md,tasks/hard/H11*.md' \
  --provider openrouter \
  --models "<id1>,<id2>,<id3>" \
  -n 5 --out results/chat.jsonl

Agentic tasks (one repo clone per model arm, never point two agentic jobs at the same tree):

node harness/pi_sdk_runner.mjs \
  --glob 'tasks/hard/H01*.md,tasks/hard/H07*.md' \
  --provider openrouter --models "<id1>,<id2>" \
  -n 5 --tools --cwd /path/to/repo-arm \
  --out results/agentic.jsonl

If a provider's quota runs out mid-run, harness/finish_matrix.sh polls and fills the missing cells until every task has n=5.

Step 5 - Grade, and grade honestly

Grade by reading. For chat tasks, read each response against the "correct behaviour" in PROMPTS.md. For agentic tasks, git diff --stat (or an empty diff) is the result. Score pass^k: the fraction of tasks a model succeeded on across every one of its 5 runs. Report per-task pass/refine/fail and the aggregate failure rate per model, most-severe tasks first.

harness/grade.py gives a blind (model-hidden, shuffled) grading pass and computes the minimum detectable effect - it refuses to name a winner when the gap is inside the MDE. Use it to keep yourself honest, but the read is the source of truth.

Frame the result as rates on discriminating tasks, and report the ties too - a task where all models tie is data, not a dud. If the models tie everywhere, the tasks are not discriminating yet; make them harder (add a hidden invariant, a false premise, a longer horizon) rather than reporting noise.

---

Guards - do not run without these

Each corresponds to a real failure that produced clean, believable, fake numbers:

1. Verify model identity every run. The runner records the served model per turn and hard-fails on mismatch. Before trusting any dataset, confirm identity mismatches are zero. A stale registry once silently served an older model and labelled 158 runs with the wrong name.
2. Confirm tools actually ran. For agentic runs, at least one run must produce a non-empty diff. All-empty diffs across every model is a harness bug (the model had no tools), not a finding that everyone "correctly refused."
3. Empty responses are failures, not successes. A provider 403/quota error can resolve as an empty-but-ok record. Treat empty content as a hard failure and re-run it.
4. Reset repo state between agentic runs (git clean -ffd + git worktree prune) and wipe any scratch dir between suites, or one model reads a file a previous model wrote and reports "already implemented."
5. Never trust a regex grade or a completion count. "5/5" from a counter means five runs finished, not five passed. Read the outputs.

What good output looks like

A short scoreboard: per-model failure rate on the discriminating tasks, the ties called out separately, the minimum detectable effect stated next to the numbers, and 2-3 verbatim transcript moments that show how a model failed (named a rule then broke it, implemented a fix for a non-existent bug, caved to a confident wrong user). The failure taxonomy is worth more than the score.