coverage-mapping

name	coverage-mapping
description	Map evaluation coverage, identify untested capability dimensions — 20 benchmarks, 30 papers, 50 web searches
used-by	benchmark-archaeology

Coverage Mapping Strategy

Map the evaluation landscape for a domain to identify which capabilities are well-tested, which are undertested, and which have no evaluation coverage at all. Produces a capability taxonomy with benchmark coverage annotations.

Purpose

Build a comprehensive map of "what we can and cannot measure" for a given AI capability domain. Identify white spaces where important capabilities lack rigorous evaluation, and redundancies where multiple benchmarks test the same narrow skill.

Budget

Resource	Floor	Target
Benchmarks mapped	15	20
Papers read	20	30
Web searches	35	50

State Ledger

<HARD-GATE>
| Metric | Current | Target | Status |
|--------|---------|--------|--------|
| Benchmarks mapped | 0 | 20 | PENDING |
| Capability taxonomy nodes | 0 | 30 | PENDING |
| Papers fetched | 0 | 30 | PENDING |
| Papers read | 0 | 20 | PENDING |
| Web searches | 0 | 50 | PENDING |
| Coverage annotations complete | 0 | 20 | PENDING |
| White spaces identified | 0 | 5 | PENDING |
| Redundancy clusters found | 0 | 3 | PENDING |
</HARD-GATE>

Cannot exit until 80% of all targets met.

Available Tactics

• score-trajectory-analysis — Understand maturity level of each benchmark

Available SOPs

• benchmark-inventory — Catalog all benchmarks in domain
• capability-taxonomy-mapping — Build hierarchical capability taxonomy
• metric-decomposition — Understand what each benchmark actually measures
• benchmark-synthesis — Produce coverage map report

Execution Guidance

1. Domain Scoping: Define the capability domain and its boundaries
2. Taxonomy Construction: a. Run capability-taxonomy-mapping to build hierarchical capability tree b. Use papers and web searches to refine taxonomy with community consensus
3. Benchmark Inventory: a. Run benchmark-inventory to collect all known benchmarks in domain b. For each benchmark, run metric-decomposition to identify tested capabilities
4. Coverage Annotation: a. Map each benchmark to taxonomy nodes it covers b. Identify coverage density per node (over-tested vs under-tested) c. Mark white spaces (zero coverage nodes)
5. Redundancy Analysis: Cluster benchmarks that test identical capabilities
6. Maturity Assessment: Run score-trajectory-analysis on key benchmarks to assess evaluation maturity
7. Synthesis: Produce coverage map with gap prioritization

Output Format

coverage_map:
  domain: string
  taxonomy:
    - node: string
      level: int
      children: list
      coverage_status: well-covered|partial|minimal|none
      benchmarks: list[string]
  white_spaces:
    - capability: string
      importance: high|medium|low
      reason_untested: string
      proposed_evaluation: string
  redundancy_clusters:
    - capability: string
      benchmarks: list[string]
      differentiation: string
  coverage_statistics:
    total_capabilities: int
    well_covered: int
    partial: int
    minimal: int
    none: int
    coverage_ratio: float