start

name	start
description	Fix wrong-chunk-index selection in cross-doc diversity
user_invocable	true

/start -- Fix BM25 Chunk Selection

TASK

Fix ONE bug: the cross-doc diversity selects the wrong chunk from the correct document family.

THE BUG

When cross-doc diversity adds a result from another document family, it picks the chunk with the highest fusion score. But fusion score doesn't correlate with query relevance within a family. BM25 score does.

Examples of the bug:

• q-015: picks data-caching-0 (non-gold) instead of data-caching-1 (gold)
• q-017: picks deploy-containers-1 (non-gold) instead of deploy-containers-0 (gold)
• q-019: picks auth-api-keys-1 (non-gold) instead of auth-api-keys-2 (gold)

THE FIX

In ${SAR_TARGET_REPO:?not set}/src/rag/retriever.py, find the cross-doc diversity logic that selects the best chunk from a target document family. Change the selection criterion from fusion score to BM25 score. The BM25 scorer's per-chunk scores are available in the retriever — use them to pick the chunk with the highest BM25 match to the query.

EXECUTION

1. Dispatch evaluator to verify current baseline metrics
2. Dispatch improver to implement the BM25 chunk selection fix in retriever.py
3. Dispatch evaluator to measure the result
4. KEEP if recall improved AND precision >= 0.95. DISCARD otherwise.
5. If discarded, try ONE variation. Then STOP and report final metrics.

ORCHESTRATOR RULES

• Dispatch evaluator and improver agents ONLY
• Do NOT read source files, run eval, edit code, or commit directly
• Forward eval reports VERBATIM to the improver

AGENT TEMPLATES

Evaluator

Agent(subagent_type="evaluator", prompt="Run evaluation on target at ${SAR_TARGET_REPO:?not set}. Use claude -p /run to invoke the target's /run skill. Report the FULL JSON output. Also: cd ${SAR_TARGET_REPO:?not set} && git log --oneline -3")

Improver

Agent(subagent_type="improver", prompt="Target: ${SAR_TARGET_REPO:?not set}. Eval report: [FULL JSON]. Task: In ${SAR_TARGET_REPO:?not set}/src/rag/retriever.py, find the cross-doc diversity best-chunk selection. Change it to use BM25 scores instead of fusion scores for picking the best chunk within a document family. The BM25 scorer has per-chunk scores available. Read retriever.py first. Make ONE change. Commit: cd ${SAR_TARGET_REPO:?not set} && git add -A && git commit -m 'change: bm25 chunk selection in cross-doc diversity'")

KEEP/DISCARD

• KEEP: recall improved AND precision >= 0.95
• DISCARD: cd ${SAR_TARGET_REPO:?not set} && git reset --hard HEAD~1

EXIT

Max 3 iterations. Then STOP and print final metrics.