| name | benchmark-inventory |
| description | Identify and catalog all relevant benchmarks in target domain |
| execution | subagent |
| prompt | ./prompt.md |
| input | research_domain, capability_focus |
| used-by | benchmark-archaeology |
Benchmark Inventory SOP
Identify, catalog, and characterize all relevant benchmarks for a given research domain and capability focus area.
Input
- research_domain: The broad research area (e.g., "natural language understanding", "code generation", "multimodal reasoning")
- capability_focus: Specific capability of interest (e.g., "commonsense reasoning", "mathematical problem solving")
Procedure
- Search Papers With Code for benchmarks tagged with the domain
- Search Semantic Scholar for benchmark papers in the domain
- Search web for leaderboards and evaluation suites
- For each benchmark found, collect: name, year, paper, size, primary metric, current SOTA, status
- Classify by: capability tested, modality, difficulty level, maintenance status
Output
Structured catalog of benchmarks with metadata sufficient for downstream analysis selection.