Audit benchmark suites against ABC framework (Task/Outcome/Reporting validity). Checks instruction quality, verifier correctness, reproducibility. Triggers on benchmark audit, audit benchmark, abc audit, task validity.

2026-03-1729

check-infra

sourcegraph/CodeScaleBench

Verify infrastructure readiness before launching benchmark runs — tokens, Docker, disk, credentials. Triggers on check infra, infrastructure check, ready to run, pre-run check.

2026-03-1729

compare-configs

sourcegraph/CodeScaleBench

Compare benchmark results across agent configurations (baseline, SG_full). Show where configs diverge. Triggers on compare configs, config comparison, which config wins, MCP impact.

2026-03-1729

cost-report

sourcegraph/CodeScaleBench

Token and cost analysis per run, suite, and config. Shows most expensive tasks and config cost comparison. Triggers on cost report, how much did it cost, token usage, spending.

2026-03-1729

generate-report

sourcegraph/CodeScaleBench

Generate the aggregate CSB evaluation report from completed Harbor runs. Triggers on generate report, eval report, ccb report, benchmark report.

2026-03-1729

name	status
description	Monitor active runs, check task completion status, and watch benchmark execution progress.

Skill: Status & Monitoring

Scope

Use this skill when the user asks to:

Check the status of active or completed runs
Monitor benchmark execution progress in real-time
List task completion rates by suite and configuration
Watch for failures or bottlenecks during runs
Analyze run fingerprints and anomalies

Canonical Commands

# Overall staging status
python3 scripts/analysis/aggregate_status.py --staging

# Watch active runs (refresh every 10s)
python3 scripts/analysis/aggregate_status.py --watch

# Status by suite
python3 scripts/analysis/aggregate_status.py --staging --suite csb_sdlc_debug

# Status fingerprints (failure patterns)
python3 scripts/analysis/status_fingerprints.py runs/staging/run_dir/result.json

Key Metrics

Completion rate — tasks passed / total tasks
Model success rate — tasks where agent succeeded
Verification success — tasks that passed post-run validation
Timeouts & OOM — infrastructure failure rates
Anomalies — tasks with unexpected patterns (fingerprint match)

Monitoring Patterns

Real-time watch: --watch mode updates every 10 seconds
Batch status: Full scan across runs/staging with counts by suite
Fingerprint triage: Use status_fingerprints to bucket failures (timeout, OOM, auth, etc.)

Related Skills

/run — launch and manage runs
/audit — deeper validation and health checks
/triage — investigate specific task failures