メニュー
Run the LoCoMo external benchmark (Pre-reg Study S1) end-to-end. Use when executing or re-running the AFT vs naive_rag comparison on LoCoMo10. Enforces dry-run-first, env-var check, background execution, and post-run evidence update.
Codex または Claude でインストール この Prompt をコピーして Codex、Claude、または他のアシスタントに貼り付けると、Skill ページを確認してインストールできます。
SOC 職業分類に基づく
| name | bench-locomo |
| description | Run the LoCoMo external benchmark (Pre-reg Study S1) end-to-end. Use when executing or re-running the AFT vs naive_rag comparison on LoCoMo10. Enforces dry-run-first, env-var check, background execution, and post-run evidence update. |
| disable-model-invocation | true |
| allowed-tools | Bash(make bench-locomo*) Bash(make llm-config*) Read Edit |
Pre-registration: benchmarks/preregistration.md §Study S1.
Canonical execution: make bench-locomo (exports .env, sets PYTHONUNBUFFERED=1).
CRITICAL: benchmarks/preregistration.md is frozen. Never edit it.
make llm-config-strict
Fail fast if EMOTIONAL_MEMORY_LLM_API_KEY is missing or config is unsupported.
make bench-locomo-dry
Confirm exit code 0 and non-empty benchmarks/locomo/results.md before
committing to the full run.
Start the full run as a background task. Stream stdout monitor. Flag and stop on:
prediction fields consistently empty → retrieval issue in adapterAfter the run completes:
benchmarks/locomo/results.json — non-empty, n_conversations=10, n_qa_total=1986benchmarks/locomo/results.md — Aggregate Scores table populated with F1, BLEU-1, Judge AccRun /evidence-update locomo to update docs/research/09_current_evidence.md.
Framing: this run is CONFIRMATORY (pre-reg S1 was committed before any LoCoMo
data — verify with git log --oneline benchmarks/preregistration.md | head -1).
temperature=0 is rejected by gpt-5-mini; call_llm strips it automatically on the
first HTTP 400 and retries — no manual intervention needed.EMOTIONAL_MEMORY_LLM_REASONING_EFFORT=minimal in .env to reduce per-call
latency from ~4.7 s → ~1.4 s (validated in local testing).| Mode | Cost | Wall-clock |
|---|---|---|
| Dry run (2 conv, 5 QA, no judge) | <$0.01 | ~2 min |
| Full run (10 conv, 1986 QA, with judge) | ~$5–8 | ~3 h |
Run the pre-publish validation gate for a new release. Use before `make publish` or Zenodo deposit. Takes a version argument, runs make release-check, validates CHANGELOG [Unreleased] is non-empty, verifies version strings align, and previews the Zenodo metadata sync.
Run one of the project's comparative benchmark studies (realistic, ablation, appraisal, comparative). Use when executing Pre-reg S2/S3 or running appraisal quality. Validates env, runs the make target, summarizes results, and prompts for evidence update.
Update docs/research/09_current_evidence.md after a benchmark study completes. Use after any bench-locomo, bench-realistic, bench-ablation, or bench-appraisal run. Pulls numbers from results.md, enforces DISCOVERY vs CONFIRMATORY framing, cross-references the pre-reg.
Build the paper + arXiv submission bundle. Use before arXiv upload or when preparing a workshop submission. Runs latexmk, produces paper/arxiv-submission.tar.gz, then validates all result tables against committed benchmarks/*/results.md.
Verify pre-registration integrity — no uncommitted edits to benchmarks/preregistration.md, and declared studies match produced result files. Use before paper building, release gates, or when resuming scientific work after a break.