| name | strata:eval |
| disable-model-invocation | true |
| description | Measure whether a retrieval/search change actually improved results, using a committed golden set (recall@k + MRR, fully local, no LLM judge). Use when the user says "did search get better or worse", "prove the ranking improved", "check for retrieval regressions", or "compare with and without rerank". Command-only — run it explicitly as /strata:eval. |
strata:eval
A regression guard + measuring stick for vault recall. It runs a small,
committed set of query → expected notes cases through the same retrieval the
recall tool uses, and reports recall@k and MRR.
The golden set
Lives at <vault>/<repo>/.eval/golden.json (commit it — it's versioned with
the vault):
{
"cases": [
{"query": "rate limiting policy",
"expected": ["decisions/2026-05-21-token-bucket.md"],
"scope": "decisions"}
]
}
scope is optional (null/omitted = all scopes). 20–50 hand-picked cases is
plenty. Seed them from queries you actually run, or from the usage ledger's
top-recalled notes.
Run it
"${CLAUDE_PLUGIN_ROOT}/bin/strata" eval -k 5
Measure the rerank lift
Compare the pipeline with and without the cross-encoder rerank:
"${CLAUDE_PLUGIN_ROOT}/bin/strata" eval -k 5 --sweep
--sweep runs the golden set rerank-OFF then rerank-ON and prints both rows +
the lift, so the decision is a number. If the lift is zero (or negative) on your
set, leave rerank off (it's off by default) — don't pay the per-call model load
for no gain.
When to run
- Before/after any retrieval change (rerank, RRF weighting, a new scope).
- Periodically, as a regression guard — a drop means recall quality slipped.
