Menu
Configure and launch CodeScaleBench runs with current paired-run and curation guardrails.
Instalar com Codex ou Claude Copie este prompt, cole no Codex, Claude ou outro assistente e deixe que ele revise a página da skill e instale para você.
Baseado na classificação ocupacional SOC
| name | run-benchmark |
| description | Configure and launch CodeScaleBench runs with current paired-run and curation guardrails. |
Use this skill when the user asks to run benchmark suites, rerun failures, or launch official/gap-fill batches in CodeScaleBench.
Before executing any benchmark run, ask the user to confirm:
anthropic/claude-haiku-4-5-20251001 for test runs)--selection-file?--baseline-only, or --full-only? Which --full-config?staging (default) or official?Do NOT launch a run until the user has confirmed these five parameters.
./configs/<suite>_2config.sh./configs/run_selected_tasks.shconfigs/eval_matrix.json*_3config.sh runners exist.baseline + sourcegraph_full.--baseline-only requires valid existing sourcegraph_full counterpart runs.--full-only requires valid existing baseline counterpart runs.ALLOW_UNPAIRED_SINGLE_CONFIG=true.# Paired per-suite run
./configs/pytorch_2config.sh --parallel 4
# Paired selected-task run
./configs/run_selected_tasks.sh --benchmark csb_sdlc_pytorch
# Gap-fill baseline only (guarded)
./configs/run_selected_tasks.sh --benchmark csb_sdlc_pytorch --baseline-only
# Gap-fill full only (guarded)
./configs/run_selected_tasks.sh --benchmark csb_sdlc_pytorch --full-only
Before launching:
python3 scripts/check_infra.pySOURCEGRAPH_ACCESS_TOKEN is set for MCP-enabled lanes.~/.claude/.credentials.json (or configured multi-account homes).python3 scripts/quarantine_invalid_tasks.py --execute
python3 scripts/generate_manifest.py --require-triage --fail-on-unknown-prefix
python3 scripts/validate_official_integrity.py --runs-dir runs/official --check-mcp-trace-health
python3 scripts/audit_traces.py --json
python3 scripts/cost_report.py
python3 scripts/generate_eval_report.py --runs-dir runs/official --output-dir eval_reports
Runs land under runs/<category>/<run_dir>/ with per-config task directories and flagged_tasks.json.
Use runs/official/MANIFEST.json as the authoritative curated inventory.
Archive old completed benchmark runs to save disk space and speed up scans. Triggers on archive runs, clean up runs, disk space, old runs.
Audit benchmark suites against ABC framework (Task/Outcome/Reporting validity). Checks instruction quality, verifier correctness, reproducibility. Triggers on benchmark audit, audit benchmark, abc audit, task validity.
Verify infrastructure readiness before launching benchmark runs — tokens, Docker, disk, credentials. Triggers on check infra, infrastructure check, ready to run, pre-run check.
Compare benchmark results across agent configurations (baseline, SG_full). Show where configs diverge. Triggers on compare configs, config comparison, which config wins, MCP impact.
Token and cost analysis per run, suite, and config. Shows most expensive tasks and config cost comparison. Triggers on cost report, how much did it cost, token usage, spending.
Generate the aggregate CSB evaluation report from completed Harbor runs. Triggers on generate report, eval report, ccb report, benchmark report.