compare-configs

name	compare-configs
description	Compare benchmark results across agent configurations (baseline, SG_full). Show where configs diverge. Triggers on compare configs, config comparison, which config wins, MCP impact.
user-invocable	true

Compare Configs

Compare results between agent configurations to find signal about MCP tool impact.

Steps

1. Run the comparison script

cd ~/CodeScaleBench && python3 scripts/compare_configs.py --format json

2. Parse and present the results

Present the JSON output as markdown tables covering:

Overall pass rates:

Config	Pass	Total	Rate
baseline	X	Y	Z%
SG_full	X	Y	Z%

Divergence analysis:

• Tasks where all configs pass (stable)
• Tasks where all configs fail (likely task/adapter issue)
• Divergent tasks (some pass, some fail) — this is the interesting signal
• Tasks where baseline fails but MCP passes — MCP tools are helping
• Tasks where MCP fails but baseline passes — MCP tools are hurting

Divergent task detail table:

Suite	Task	baseline	SG_full	Signal
csb_sdlc_pytorch	sgt-005	PASS	PASS	MCP-Full matches

3. Highlight key findings

Focus the narrative on:

• Biggest winner: Which config has the highest pass rate
• MCP helps: Tasks where only MCP configs pass
• MCP hurts: Tasks where only baseline passes (investigate why)
• All-fail: Tasks failing everywhere (fix these first, helps all configs)

4. MCP-conditioned analysis (optional, when user asks about MCP specifically)

The basic divergence analysis treats all SG tasks equally. For deeper MCP insight, run the MCP audit which conditions on actual usage:

python3 scripts/mcp_audit.py --paired-only --json --verbose 2>/dev/null

This separates:

• Used-MCP tasks: Agent actually called MCP tools. Reward delta reflects true MCP value.
• Zero-MCP tasks: MCP available but unused. Reward delta reflects preamble overhead only.
• Intensity buckets: Light (1-5 calls), Moderate (6-20), Heavy (20+) — shows dose-response.

Present the MCP-conditioned reward delta table:

Group	N	BL Reward	SF Reward	Delta
Used-MCP	N	X	Y	+Z%
Zero-MCP	N	X	Y	-Z%
Light	N	X	Y	Z%
Moderate	N	X	Y	Z%
Heavy	N	X	Y	Z%

Key insight: Overall SG vs BL delta is diluted by zero-MCP tasks. Conditioning on usage reveals the true MCP signal. For full analysis, suggest /mcp-audit.

Variants

Filter to one suite

python3 scripts/compare_configs.py --suite csb_sdlc_pytorch --format json

Show only divergent tasks

python3 scripts/compare_configs.py --divergent-only --format json

Table format (compact)

python3 scripts/compare_configs.py --format table