メニュー
Score individual task quality on instruction clarity, verifier quality, and reproducibility. Triggers on score task, task quality, rate tasks, task score.
Codex または Claude でインストール この Prompt をコピーして Codex、Claude、または他のアシスタントに貼り付けると、Skill ページを確認してインストールできます。
SOC 職業分類に基づく
Archive old completed benchmark runs to save disk space and speed up scans. Triggers on archive runs, clean up runs, disk space, old runs.
Audit benchmark suites against ABC framework (Task/Outcome/Reporting validity). Checks instruction quality, verifier correctness, reproducibility. Triggers on benchmark audit, audit benchmark, abc audit, task validity.
Verify infrastructure readiness before launching benchmark runs — tokens, Docker, disk, credentials. Triggers on check infra, infrastructure check, ready to run, pre-run check.
Compare benchmark results across agent configurations (baseline, SG_full). Show where configs diverge. Triggers on compare configs, config comparison, which config wins, MCP impact.
Token and cost analysis per run, suite, and config. Shows most expensive tasks and config cost comparison. Triggers on cost report, how much did it cost, token usage, spending.
Generate the aggregate CSB evaluation report from completed Harbor runs. Triggers on generate report, eval report, ccb report, benchmark report.
| name | score-tasks |
| description | Score individual task quality on instruction clarity, verifier quality, and reproducibility. Triggers on score task, task quality, rate tasks, task score. |
| user-invocable | true |
Score individual benchmark tasks on three weighted quality dimensions.
Runs scripts/abc_score_task.py which scores tasks on:
cd ~/CodeScaleBench && python3 scripts/abc_score_task.py --task benchmarks/csb_sdlc_pytorch/sgt-005
python3 scripts/abc_score_task.py --suite csb_sdlc_pytorch --format table
Flag tasks scoring below the threshold as needing review:
python3 scripts/abc_score_task.py --all --threshold 0.7 --format table
python3 scripts/abc_score_task.py --suite csb_sdlc_swebenchpro --format json
Suite summary:
| Task | Overall | Instruction | Verifier | Reproducibility | Needs Review |
|---|---|---|---|---|---|
| sgt-005 | 0.85 | 0.90 | 0.80 | 0.85 | No |
| sgt-008 | 0.65 | 0.70 | 0.55 | 0.70 | Yes |
Highlight tasks that need review and explain which sub-checks failed.
/benchmark-audit — Suite-level ABC audit (broader, checks outcome and reporting too)/validate-tasks — Pre-flight validation (operational, not quality scoring)