| name | quality-eval-core |
| description | Shared reference for the quality-eval cluster: the four quality gates (write → commit → CI → ship), the evidence ladder that says what proves a pass at each, pass@k vs a single green run, and the AI write-and-review blind spot. USE WHEN deciding what counts as done, choosing which gate catches a failure class, or wiring a measurement loop — the conventions every quality-eval spoke shares. |
| cluster | quality-eval |
| version | 1.0.0 |
Quality-Eval Core
Shared model for the quality-eval cluster. The TDD, E2E, verification, benchmark,
linting, and audit spokes all depend on these conventions — keep them consistent here so
no spoke contradicts another about what "done" means.
1. The four gates (the cluster's defining model)
Quality is enforced at four stages, cheapest-and-earliest first. Each gate catches a
different failure class; a failure that escapes one is meant to be caught by the next.
Edit ──write-time──> Commit ──commit-time──> CI ──build-time──> Release ──ship-time──> Prod
plankton- verification-loop e2e / windows- production-audit
code-quality tdd-workflow desktop-e2e
ai-regression-testing browser-qa · benchmark
| Gate | Stage | Catches | Spokes |
|---|
| Write | every file edit | style / lint / type drift | plankton-code-quality |
| Commit | before a PR | "not actually finished", missing tests, regressions | verification-loop, tdd-workflow, ai-regression-testing, error-handling |
| CI | build / pipeline | broken behavior, slow paths, broken UI | e2e-testing, windows-desktop-e2e, browser-qa, benchmark |
| Ship | pre-launch | "what breaks in prod?" | production-audit |
Rule: push each check to the earliest gate that can catch it. A write-time linter
should not be doing a production audit's job, and a production audit should not be the
first place a type error is noticed.
2. The evidence ladder
A claim is only as strong as the artifact behind it. Climb the ladder; never report a rung
you didn't run.
- Asserted — "it works." (no value)
- Ran — a command exited 0; paste the output.
- Tested — a named test covers the behavior and is green.
- Measured — a number against a baseline (coverage %, p95 latency, pass rate).
- Reproduced — green k times (
pass@k), or across agents/runs.
Everything below "Ran" is a guess. verification-loop and eval-harness exist to force a
claim up this ladder before a PR.
3. pass@k — one green run is not a pass
Anything non-deterministic (agent output, flaky E2E, perf, LLM calls) must be judged by
how often it passes, not whether it can:
pass@1 — succeeded once. Says almost nothing for a stochastic task.pass@k — succeeded in ≥1 of k runs (capability ceiling).- pass rate — fraction of k runs that pass (reliability floor — the number that matters for shipping).
eval-harness defines criteria + computes this for Claude Code tasks; agent-eval uses it
to rank agents; e2e-testing/browser-qa use it to separate a real failure from a flake.
4. The AI write-and-review blind spot
When the same model writes code and reviews it, its blind spots are correlated — it
approves the bugs it was prone to make. Defenses, in order:
- An independent check the model can't reason its way around: a deterministic test (
tdd-workflow, ai-regression-testing), a real browser (browser-qa), a real OS UIA tree (windows-desktop-e2e).
- A gate the model cannot satisfy by editing its own config — treat changes to lint/test/CI config in the same iteration as suppressing a violation as a red flag (
plankton-code-quality).
- A second judge for subjective output (a different model/agent —
agent-eval).
5. Tooling matrix
| Concern | Default tool | Spoke |
|---|
| Web E2E | Playwright (Page Object Model) | e2e-testing |
| Desktop E2E | pywinauto + Windows UIA | windows-desktop-e2e |
| Visual / interaction QA | headless browser automation | browser-qa |
| Write-time lint/format | Plankton (biome / ruff / gofmt …) via PostToolUse hooks | plankton-code-quality |
| Perf baseline | JSON baselines in .benchmarks/, git-tracked | benchmark |
| Eval / pass@k | EDD harness | eval-harness |
| Agent comparison | reproducible-task CLI | agent-eval |
| Errors | typed errors / retries / circuit breakers (TS · Python · Go) | error-handling |
6. Shared guardrails
- Evidence over assertion — paste the command output; "Ran/Tested/Measured", never "Asserted".
- pass@k for anything stochastic — report the pass rate, not a lucky single green run.
- Earliest gate wins — catch each failure class at the cheapest stage that can.
- Independent judgment — don't let the thing under test grade itself; break the write-and-review loop.
- No config laundering — a gate satisfied by loosening its own config is not satisfied; flag it.
- State coverage honestly — a passing suite over thin tests is a weaker claim than its green color implies; name the gap.
