Menu
Generate evaluation reports, analyze run costs, and compare configurations.
Instalar com Codex ou Claude Copie este prompt, cole no Codex, Claude ou outro assistente e deixe que ele revise a página da skill e instale para você.
Baseado na classificação ocupacional SOC
Archive old completed benchmark runs to save disk space and speed up scans. Triggers on archive runs, clean up runs, disk space, old runs.
Audit benchmark suites against ABC framework (Task/Outcome/Reporting validity). Checks instruction quality, verifier correctness, reproducibility. Triggers on benchmark audit, audit benchmark, abc audit, task validity.
Verify infrastructure readiness before launching benchmark runs — tokens, Docker, disk, credentials. Triggers on check infra, infrastructure check, ready to run, pre-run check.
Compare benchmark results across agent configurations (baseline, SG_full). Show where configs diverge. Triggers on compare configs, config comparison, which config wins, MCP impact.
Token and cost analysis per run, suite, and config. Shows most expensive tasks and config cost comparison. Triggers on cost report, how much did it cost, token usage, spending.
Generate the aggregate CSB evaluation report from completed Harbor runs. Triggers on generate report, eval report, ccb report, benchmark report.
| name | report |
| description | Generate evaluation reports, analyze run costs, and compare configurations. |
Use this skill when the user asks to:
# Generate full evaluation report
python3 scripts/maintenance/generate_eval_report.py
# Cost breakdown analysis
python3 scripts/evaluation/cost_report.py --run-dir runs/staging/run_dir
# Cost analysis by model/suite
python3 scripts/evaluation/cost_breakdown_analysis.py --staging
# Compare configurations
python3 scripts/evaluation/compare_configs.py --config1 baseline --config2 sourcegraph_full
# IR analysis (information retrieval)
python3 scripts/evaluation/ir_analysis.py --run-dir runs/staging/run_dir
# Oracle impact analysis
python3 scripts/evaluation/oracle_ir_analysis.py --runs-dir runs/staging
/evaluate — extract and score individual task results/run — generate results to analyze/status — check run status before generating reports