bench-study

name	bench-study
description	Run one of the project's comparative benchmark studies (realistic, ablation, appraisal, comparative). Use when executing Pre-reg S2/S3 or running appraisal quality. Validates env, runs the make target, summarizes results, and prompts for evidence update.
disable-model-invocation	true
arguments	["study"]
allowed-tools	Bash(make bench-*) Bash(make llm-config*) Read

Benchmark Study Runner

Argument: $study — one of: realistic, ablation, appraisal, comparative, comparative-sbert, appraisal-confound.

Stop immediately if the argument is not in the list above and ask the user to pick.

Study → Make Target Mapping

Argument	Make target	Pre-reg	LLM required
realistic	make bench-realistic	S2	No
ablation	make bench-ablation	S3	No
appraisal	make bench-appraisal	M1.3	Yes (EMOTIONAL_MEMORY_LLM_API_KEY)
comparative	make bench-comparative	—	No (hash embedder, quick)
comparative-sbert	make bench-comparative-sbert	—	No (SBERT embedder, paper-canonical)
appraisal-confound	make bench-appraisal-confound	Addendum A	No

Steps

1. Validate config (LLM-requiring studies only)

For appraisal:

make llm-config-strict

2. Run the study

make bench-$study

For realistic, ablation, and comparative-sbert the Makefile already uses the SBERT embedder.

3. Interpret results

Read benchmarks/$study/results.md (or console output for appraisal).

Report:

• Aggregate metric (top1_acc / F1 / pass rate)
• Per-category / per-challenge breakdown
• Bootstrap 95% CI if available
• Any result that survived Holm correction
• Comparison to prior numbers in docs/research/09_current_evidence.md

4. Determine DISCOVERY vs CONFIRMATORY framing

git log --oneline benchmarks/preregistration.md | head -1
git log --oneline benchmarks/$study/results.json 2>/dev/null | head -1

• Pre-reg commit predates results commit and the study is declared in pre-reg → CONFIRMATORY
• Otherwise → DISCOVERY (note that confirmatory evidence is still pending)

5. Prompt for evidence update

Remind the user to run /evidence-update $study to update docs/research/09_current_evidence.md with the new numbers and framing.