recall-gate

name	recall-gate
description	Run the retrieval recall gate with dead-gold triage and the binary A/B regression test. Required before release tags; use after any retrieval-adjacent merge.

Recall Gate

Answers one question with evidence: did retrieval change? Numbers moving is NOT the answer — fixture drift and corpus drift both move numbers without any retrieval change. Classify before believing.

1. Run both splits

CQS_NO_DAEMON=1 cqs eval evals/queries/v3_test.v2.json 2>/dev/null | grep OVERALL
CQS_NO_DAEMON=1 cqs eval evals/queries/v3_dev.v2.json  2>/dev/null | grep OVERALL

Aggregate = mean of the two (equal n=109). Compare against the latest recorded gate in ~/training-data/research/eval.md — NEVER against numbers from memory. Run-to-run noise is ±1pp with an active daemon.

Pin the substrate. Eval numbers depend on WHICH index computed them — a bare cqs eval relays to the daemon's RESIDENT index (index_substrate: "resident" in --json), while CQS_NO_DAEMON=1 forces a fresh CLI-side build ("fresh-built"). The two differ within the ~1pp HNSW-reconstruction noise, so comparing a resident run against a fresh-built baseline manufactures a phantom delta. Either pin CQS_NO_DAEMON=1 on every run (as above — fresh-built always) OR assert the --json index_substrate marker is uniform across every run you compare (including the recorded baseline). The recorded gate must note which substrate it was taken on.

2. If any metric dropped: check dead golds FIRST

The matcher is (origin, name) — file moves, splits, renames, and deletions kill golds and cap R@K, looking exactly like a recall regression:

# per split: for each q['gold_chunk'], does the index hold a chunk with that (origin, name)?
sqlite3 .cqs/slots/gemma/index.db "SELECT 1 FROM chunks WHERE origin=? AND name=?"

Re-pin dead golds: moved function → same name, new origin. Deleted function → re-point to the nearest current equivalent that genuinely answers the query text (read the query!), and record a repinned_<date> note in the query's metadata. Re-run step 1.

3. If a delta survives re-pinning: same-corpus binary A/B (definitive)

git worktree add /tmp/cqs-ab <baseline-tag>
cd /tmp/cqs-ab && CARGO_TARGET_DIR=$HOME/.cargo-target/cqs-ab cargo build --release --features cuda-index
CQS_NO_DAEMON=1 $HOME/.cargo-target/cqs-ab/release/cqs eval <fixture>   # both splits
CQS_NO_DAEMON=1 cqs eval <fixture>                                       # both splits, new binary

Pin BOTH sides to CQS_NO_DAEMON=1 (daemon vs CLI path differences pollute the comparison). Bit-identical output = zero retrieval change; the inter-capture delta is corpus drift — say so and pass the gate. A real difference = regression; bisect the suspect PRs ("no ranking change" claims with pins are where to start). Clean up the worktree + target dir after.

4. Record and sync

• Append the gate block to ~/training-data/research/eval.md (table, verdict, re-pins).
• On release gates, sync the three public surfaces to the new snapshot: README TL;DR line, Cargo.toml description, gh repo edit --description.
• Fixture re-pins commit with the gate (release branch or a docs PR).