メニュー
Evaluate whether benchmark measures its claimed capability
Codex または Claude でインストール この Prompt をコピーして Codex、Claude、または他のアシスタントに貼り付けると、Skill ページを確認してインストールできます。
SOC 職業分類に基づく
SOTA Performance Baseline Campaign — 5 strategies for systematically collecting, standardizing, and analyzing performance data across methods. Produces standardized comparison tables, progress curves, and headroom analysis.
Assess systematic biases in the evidence body — publication bias, reporting bias, and selective outcome reporting. Budget: 40 studies, 40 effect sizes, 40 web searches.
Track evidence accumulation over time — cumulative meta-analysis protocol design. Budget: 40 studies, 40 effect sizes, 30 web searches.
Design structured data extraction form for systematic meta-analysis data collection
Systematically extract effect sizes and conditions from papers for meta-analytic synthesis
Determine effect size types and calculation methods for meta-analytic synthesis
| name | construct-validity-assessment |
| description | Evaluate whether benchmark measures its claimed capability |
| execution | subagent |
| prompt | ./prompt.md |
| input | benchmark_name, claimed_capability, task_examples |
| used-by | benchmark-archaeology |
Evaluate whether a benchmark actually measures the capability it claims to measure, using psychometric validity frameworks adapted for AI evaluation.
Validity verdict with evidence for each validity dimension.
