Audit benchmark suites against ABC framework (Task/Outcome/Reporting validity). Checks instruction quality, verifier correctness, reproducibility. Triggers on benchmark audit, audit benchmark, abc audit, task validity.

2026-03-1729

check-infra

sourcegraph/CodeScaleBench

Verify infrastructure readiness before launching benchmark runs — tokens, Docker, disk, credentials. Triggers on check infra, infrastructure check, ready to run, pre-run check.

2026-03-1729

compare-configs

sourcegraph/CodeScaleBench

Compare benchmark results across agent configurations (baseline, SG_full). Show where configs diverge. Triggers on compare configs, config comparison, which config wins, MCP impact.

2026-03-1729

cost-report

sourcegraph/CodeScaleBench

Token and cost analysis per run, suite, and config. Shows most expensive tasks and config cost comparison. Triggers on cost report, how much did it cost, token usage, spending.

2026-03-1729

generate-report

sourcegraph/CodeScaleBench

Generate the aggregate CSB evaluation report from completed Harbor runs. Triggers on generate report, eval report, ccb report, benchmark report.

2026-03-1729

name	triage
description	Investigate and triage failed benchmark tasks, analyze root causes, and plan reruns.

Skill: Triage Failures

Scope

Use this skill when the user asks to:

Investigate why specific tasks failed
Categorize failures by type (timeout, OOM, infra, logic)
Recommend fixes or reruns
Analyze failure patterns across suites
Prepare rerun configurations for failed tasks

Canonical Commands

# Triage a specific task result
python3 scripts/analysis/status_fingerprints.py runs/staging/run_dir/task_id/result.json

# List all failures in a run with categorization
python3 scripts/analysis/aggregate_status.py --staging --suite csb_sdlc_debug --failures-only

# Deep analysis of error patterns
grep -r "error\|fail\|timeout" runs/staging/run_dir/*/trajectory.json

# Check task logs and stdout
cat runs/staging/run_dir/task_id/agent/solution.md

Failure Categories

Timeout — task exceeded time limit (check time_limit_sec in task.toml)
OOM — out of memory (memory_mb insufficient in task.toml)
Infrastructure — Docker, network, permissions, image pull failures
Auth — OAuth token expired or insufficient scopes
Logic — agent produced wrong output or reference fix incorrect
Verification — test script broken or changed contract

Rerun Strategy

Identify root cause using fingerprints and logs
If infrastructure issue: increase timeout or memory, switch environment (Daytona → local)
If auth: refresh credentials before rerun
If task contract broken: update test.sh or oracle_checks.py
Use /run skill with --rerun flag or custom selection file

Related Skills

/run — execute reruns after diagnosis
/status — see current failure overview
/evaluate — deep analysis of task outputs