// Compute information retrieval quality metrics (precision, recall, MRR, nDCG, MAP) comparing file retrieval across baseline and MCP configs against ground truth. Triggers on ir analysis, retrieval metrics, file recall, ground truth, search quality.
Archive old completed benchmark runs to save disk space and speed up scans. Triggers on archive runs, clean up runs, disk space, old runs.
Audit benchmark suites against ABC framework (Task/Outcome/Reporting validity). Checks instruction quality, verifier correctness, reproducibility. Triggers on benchmark audit, audit benchmark, abc audit, task validity.
Verify infrastructure readiness before launching benchmark runs — tokens, Docker, disk, credentials. Triggers on check infra, infrastructure check, ready to run, pre-run check.
Compare benchmark results across agent configurations (baseline, SG_full). Show where configs diverge. Triggers on compare configs, config comparison, which config wins, MCP impact.
Token and cost analysis per run, suite, and config. Shows most expensive tasks and config cost comparison. Triggers on cost report, how much did it cost, token usage, spending.
Generate the aggregate CSB evaluation report from completed Harbor runs. Triggers on generate report, eval report, ccb report, benchmark report.
| name | ir-analysis |
| description | Compute information retrieval quality metrics (precision, recall, MRR, nDCG, MAP) comparing file retrieval across baseline and MCP configs against ground truth. Triggers on ir analysis, retrieval metrics, file recall, ground truth, search quality. |
| user-invocable | true |
Measure how well agents find the right files, comparing baseline (local tools) vs MCP (Sourcegraph) retrieval against per-task ground truth.
Runs scripts/ir_analysis.py which:
configs/ground_truth_files.json (or builds it from benchmark task dirs)agent/claude-code.txt) to extract which files were accessed via tool callsIf configs/ground_truth_files.json doesn't exist or needs refreshing:
cd ~/CodeScaleBench && python3 scripts/ir_analysis.py --build-ground-truth
This extracts ground truth files from each benchmark's task structure (patches, diffs, ground_truth dirs, test scripts, instructions). Reports per-benchmark counts and confidence levels.
cd ~/CodeScaleBench && python3 scripts/ir_analysis.py --json 2>/dev/null
Or for human-readable table output:
cd ~/CodeScaleBench && python3 scripts/ir_analysis.py 2>/dev/null
Per-benchmark IR scores:
| Benchmark | Config | N | File Recall | MRR | P@5 | R@5 | nDCG@5 | MAP | Ctx Eff |
|---|
Overall aggregates:
| Config | File Recall | MRR | MAP | Context Efficiency |
|---|---|---|---|---|
| baseline | X | X | X | X |
| sourcegraph_full | X | X | X | X |
Statistical tests (baseline vs SG_full):
| Metric | Welch's t | p-value | Cohen's d | Bootstrap 95% CI |
|---|
Key metrics to focus on:
For detailed per-task scores:
python3 scripts/ir_analysis.py --per-task --json 2>/dev/null
Filter to a specific benchmark:
python3 scripts/ir_analysis.py --suite csb_sdlc_swebenchpro 2>/dev/null
python3 scripts/ir_analysis.py --build-ground-truth
python3 scripts/ir_analysis.py --json > /tmp/ir_results.json
python3 scripts/ir_analysis.py --suite csb_sdlc_pytorch
python3 scripts/ir_analysis.py --per-task
agent/claude-code.txt. Extracts file paths from Read, Grep, Glob, Write, Edit tool inputs and MCP tool results./workspace/ prefix, a//b/ diff notation, lowercases for comparison.started_at timestamp) wins.csb_metrics/statistics.py — Welch's t-test, Cohen's d, bootstrap CI. No scipy dependency.| Benchmark | Strategy | Source File | Confidence |
|---|---|---|---|
| SWE-bench Pro | Patch headers | solve.sh / solution/solve.sh | high |
| PyTorch | Diff headers | tests/expected.diff / instruction.md | high |
| K8s Docs | Directory listing | ground_truth/ | high |
| Governance | Test script paths | tests/test.sh | medium |
| Enterprise | Test script paths | tests/test.sh | medium |
| Others | Instruction regex | instruction.md | low |
/mcp-audit — MCP usage patterns and adoption rates (complements IR quality metrics)/compare-configs — Binary pass/fail divergence with optional statistical tests/evaluate-traces — Comprehensive trace audit (broader scope, data integrity focus)