Audit benchmark suites against ABC framework (Task/Outcome/Reporting validity). Checks instruction quality, verifier correctness, reproducibility. Triggers on benchmark audit, audit benchmark, abc audit, task validity.

2026-03-1729

check-infra

sourcegraph/CodeScaleBench

Verify infrastructure readiness before launching benchmark runs — tokens, Docker, disk, credentials. Triggers on check infra, infrastructure check, ready to run, pre-run check.

2026-03-1729

compare-configs

sourcegraph/CodeScaleBench

Compare benchmark results across agent configurations (baseline, SG_full). Show where configs diverge. Triggers on compare configs, config comparison, which config wins, MCP impact.

2026-03-1729

cost-report

sourcegraph/CodeScaleBench

Token and cost analysis per run, suite, and config. Shows most expensive tasks and config cost comparison. Triggers on cost report, how much did it cost, token usage, spending.

2026-03-1729

generate-report

sourcegraph/CodeScaleBench

Generate the aggregate CSB evaluation report from completed Harbor runs. Triggers on generate report, eval report, ccb report, benchmark report.

2026-03-1729

name	next
description	Plan upcoming work, analyze coverage gaps, and recommend next steps for benchmarking.

Skill: What's Next / Planning

Scope

Use this skill when the user asks to:

Identify what benchmarking work should happen next
Analyze coverage gaps and incomplete suites
Plan variance runs and gap-fill campaigns
Recommend task categories to focus on
Generate next-step recommendations based on current state

Canonical Commands

# Overall readiness assessment
python3 scripts/infra/check_infra.py

# Current run status and coverage
python3 scripts/analysis/aggregate_status.py --staging

# Gap analysis (missing suites/configs)
grep -r "missing\|incomplete" runs/staging/*/result.json | head -20

# Plan variance runs
python3 scripts/analysis/plan_variance_runs.py --target 50 --dry-run

# Coverage gap analysis
python3 scripts/analysis/analyze_run_coverage.py --runs-dir runs/staging

Planning Workflow

Assess current state — status, completion rates, failures
Identify gaps — missing suites, configurations, models
Categorize work — high-priority, medium, low-priority next steps
Estimate effort — time, resources, dependencies
Recommend — specific run commands and configurations

Gap Analysis Dimensions

Suite coverage — which suites are incomplete?
Configuration — missing baseline, MCP, or paired runs?
Model coverage — which models need testing?
Environment — Daytona vs local Docker?
Rerun strategy — which failures should be retried?

Variance Planning

Power analysis — how many runs for statistical significance?
Rebalancing — equalize difficulty across runs
DOE (Design of Experiments) — optimize run selection
Gap-fill — targeted runs to complete coverage

Related Skills

/status — check current completion state
/run — execute next steps
/audit — validate readiness for next phase