菜单
Extract metrics, score traces, and evaluate benchmark task results.
用 Codex 或 Claude 帮你安装 复制这段 Prompt，粘贴到 Codex、Claude 或其他助手里，让它检查 Skill 页面并帮你完成安装。
基于 SOC 职业分类
Archive old completed benchmark runs to save disk space and speed up scans. Triggers on archive runs, clean up runs, disk space, old runs.
Audit benchmark suites against ABC framework (Task/Outcome/Reporting validity). Checks instruction quality, verifier correctness, reproducibility. Triggers on benchmark audit, audit benchmark, abc audit, task validity.
Verify infrastructure readiness before launching benchmark runs — tokens, Docker, disk, credentials. Triggers on check infra, infrastructure check, ready to run, pre-run check.
Compare benchmark results across agent configurations (baseline, SG_full). Show where configs diverge. Triggers on compare configs, config comparison, which config wins, MCP impact.
Token and cost analysis per run, suite, and config. Shows most expensive tasks and config cost comparison. Triggers on cost report, how much did it cost, token usage, spending.
Generate the aggregate CSB evaluation report from completed Harbor runs. Triggers on generate report, eval report, ccb report, benchmark report.
| name | evaluate |
| description | Extract metrics, score traces, and evaluate benchmark task results. |
Use this skill when the user asks to:
# Extract task metrics from a run
python3 scripts/evaluation/extract_task_metrics.py --run-dir runs/staging/run_dir
# Score all traces in a run
python3 scripts/evaluation/trace_quality_pipeline.py --run-dir runs/staging/run_dir
# Reextract metrics for specific suite
python3 scripts/evaluation/reextract_all_metrics.py --filter csb_sdlc_debug
# Compute retrieval metrics (IR tasks)
python3 scripts/evaluation/compute_retrieval_metrics.py --run-dir runs/staging/run_dir
# Dual verification check
python3 scripts/evaluation/cross_validate_oracles.py --run-dir runs/staging/run_dir
/run — launch runs to generate traces/report — compile evaluation reports/audit — validate trace quality and consistency