원클릭으로 Manus에서 모든 스킬 실행

bench-study

스타4

포크0

업데이트2026년 4월 24일 10:58

Run one of the project's comparative benchmark studies (realistic, ablation, appraisal, comparative). Use when executing Pre-reg S2/S3 or running appraisal quality. Validates env, runs the make target, summarizes results, and prompts for evidence update.

설치

Codex 또는 Claude로 설치 이 Prompt를 복사해 Codex, Claude 또는 다른 어시스턴트에 붙여 넣으면 Skill 페이지를 검토하고 설치를 진행할 수 있습니다.

Manus에서 실행

출처

gianlucamazza

gianlucamazza/emotional-memory

GitHub 저장소 열기 Creator 저장소 보기

다운로드

Manus에서 실행

Benchmark Study Runner

Argument: $study — one of: realistic, ablation, appraisal, comparative, comparative-sbert, appraisal-confound.

Stop immediately if the argument is not in the list above and ask the user to pick.

Study → Make Target Mapping

Argument	Make target	Pre-reg	LLM required
`realistic`	`make bench-realistic`	S2	No
`ablation`	`make bench-ablation`	S3	No
`appraisal`	`make bench-appraisal`	M1.3	Yes (`EMOTIONAL_MEMORY_LLM_API_KEY`)
`comparative`	`make bench-comparative`	—	No (hash embedder, quick)
`comparative-sbert`	`make bench-comparative-sbert`	—	No (SBERT embedder, paper-canonical)
`appraisal-confound`	`make bench-appraisal-confound`	Addendum A	No

Steps

1. Validate config (LLM-requiring studies only)

For appraisal:

make llm-config-strict

2. Run the study

make bench-$study

For realistic, ablation, and comparative-sbert the Makefile already uses the SBERT embedder.

3. Interpret results

Read benchmarks/$study/results.md (or console output for appraisal).

Report:

Aggregate metric (top1_acc / F1 / pass rate)
Per-category / per-challenge breakdown
Bootstrap 95% CI if available
Any result that survived Holm correction
Comparison to prior numbers in docs/research/09_current_evidence.md

4. Determine DISCOVERY vs CONFIRMATORY framing

git log --oneline benchmarks/preregistration.md | head -1
git log --oneline benchmarks/$study/results.json 2>/dev/null | head -1

Pre-reg commit predates results commit and the study is declared in pre-reg → CONFIRMATORY
Otherwise → DISCOVERY (note that confirmatory evidence is still pending)

5. Prompt for evidence update

Remind the user to run /evidence-update $study to update docs/research/09_current_evidence.md with the new numbers and framing.

이 저장소의 다른 Skills

같은 저장소

release-gate

gianlucamazza/emotional-memory

Run the pre-publish validation gate for a new release. Use before `make publish` or Zenodo deposit. Takes a version argument, runs make release-check, validates CHANGELOG [Unreleased] is non-empty, verifies version strings align, and previews the Zenodo metadata sync.

2026-05-274

bench-locomo

gianlucamazza/emotional-memory

Run the LoCoMo external benchmark (Pre-reg Study S1) end-to-end. Use when executing or re-running the AFT vs naive_rag comparison on LoCoMo10. Enforces dry-run-first, env-var check, background execution, and post-run evidence update.

2026-04-244

evidence-update

gianlucamazza/emotional-memory

Update docs/research/09_current_evidence.md after a benchmark study completes. Use after any bench-locomo, bench-realistic, bench-ablation, or bench-appraisal run. Pulls numbers from results.md, enforces DISCOVERY vs CONFIRMATORY framing, cross-references the pre-reg.

2026-04-244

paper-bundle

gianlucamazza/emotional-memory

Build the paper + arXiv submission bundle. Use before arXiv upload or when preparing a workshop submission. Runs latexmk, produces paper/arxiv-submission.tar.gz, then validates all result tables against committed benchmarks/*/results.md.

2026-04-244

prereg-guard

gianlucamazza/emotional-memory

Verify pre-registration integrity — no uncommitted edits to benchmarks/preregistration.md, and declared studies match produced result files. Use before paper building, release gates, or when resuming scientific work after a break.

2026-04-244

name	bench-study
description	Run one of the project's comparative benchmark studies (realistic, ablation, appraisal, comparative). Use when executing Pre-reg S2/S3 or running appraisal quality. Validates env, runs the make target, summarizes results, and prompts for evidence update.
disable-model-invocation	true
arguments	["study"]
allowed-tools	Bash(make bench-) Bash(make llm-config) Read

Benchmark Study Runner

Argument: $study — one of: realistic, ablation, appraisal, comparative, comparative-sbert, appraisal-confound.

Stop immediately if the argument is not in the list above and ask the user to pick.

Study → Make Target Mapping

Argument	Make target	Pre-reg	LLM required
`realistic`	`make bench-realistic`	S2	No
`ablation`	`make bench-ablation`	S3	No
`appraisal`	`make bench-appraisal`	M1.3	Yes (`EMOTIONAL_MEMORY_LLM_API_KEY`)
`comparative`	`make bench-comparative`	—	No (hash embedder, quick)
`comparative-sbert`	`make bench-comparative-sbert`	—	No (SBERT embedder, paper-canonical)
`appraisal-confound`	`make bench-appraisal-confound`	Addendum A	No

Steps

1. Validate config (LLM-requiring studies only)

For appraisal:

make llm-config-strict

2. Run the study

make bench-$study

For realistic, ablation, and comparative-sbert the Makefile already uses the SBERT embedder.

3. Interpret results

Read benchmarks/$study/results.md (or console output for appraisal).

Report:

Aggregate metric (top1_acc / F1 / pass rate)
Per-category / per-challenge breakdown
Bootstrap 95% CI if available
Any result that survived Holm correction
Comparison to prior numbers in docs/research/09_current_evidence.md

4. Determine DISCOVERY vs CONFIRMATORY framing

git log --oneline benchmarks/preregistration.md | head -1
git log --oneline benchmarks/$study/results.json 2>/dev/null | head -1

Pre-reg commit predates results commit and the study is declared in pre-reg → CONFIRMATORY
Otherwise → DISCOVERY (note that confirmatory evidence is still pending)

5. Prompt for evidence update

Remind the user to run /evidence-update $study to update docs/research/09_current_evidence.md with the new numbers and framing.