메뉴
Batch re-extract task_metrics.json for all runs after extraction bug fixes or schema changes. Triggers on reextract metrics, refresh metrics, update task metrics, fix extraction.
Codex 또는 Claude로 설치 이 Prompt를 복사해 Codex, Claude 또는 다른 어시스턴트에 붙여 넣으면 Skill 페이지를 검토하고 설치를 진행할 수 있습니다.
SOC 직업 분류 기준
| name | reextract-metrics |
| description | Batch re-extract task_metrics.json for all runs after extraction bug fixes or schema changes. Triggers on reextract metrics, refresh metrics, update task metrics, fix extraction. |
| user-invocable | true |
Batch re-extract task_metrics.json for all active task directories after fixing extraction bugs or adding new metric fields.
Runs scripts/reextract_all_metrics.py which:
runs/official/ finding all task directories with result.jsonextract_task_metrics.py) on eachselected_benchmark_tasks.jsonAlways show what will be re-extracted first:
cd ~/CodeScaleBench && python3 scripts/reextract_all_metrics.py --dry-run
If filtering to a specific suite:
python3 scripts/reextract_all_metrics.py --dry-run --filter csb_sdlc_pytorch
python3 scripts/reextract_all_metrics.py
Or filtered:
python3 scripts/reextract_all_metrics.py --filter csb_sdlc_swebenchpro
The script reports:
Pay attention to CORRECTED lines — these indicate tasks where the old extraction had inflated or deflated metrics.
After re-extraction, always regenerate the MANIFEST to pick up corrected metrics:
python3 scripts/generate_manifest.py
Spot-check a few task_metrics.json files to confirm the fix applied correctly:
# Example: check a specific task
python3 -c "
import json, pathlib
p = pathlib.Path('runs/official')
# Find a recent task_metrics.json
for f in sorted(p.rglob('task_metrics.json'))[:3]:
d = json.loads(f.read_text())
print(f'{f.parent.name}: cost=\${d.get(\"cost_usd\", \"n/a\")}, mcp={d.get(\"tool_calls_mcp\", \"n/a\")}, total={d.get(\"tool_calls_total\", \"n/a\")}')
"
When extract_task_metrics.py logic changes (e.g., transcript-first tool counting):
python3 scripts/reextract_all_metrics.py
python3 scripts/generate_manifest.py
When new extractors are added to csb_metrics/extractors.py:
python3 scripts/reextract_all_metrics.py
To extract metrics for newly completed tasks:
python3 scripts/reextract_all_metrics.py --filter csb_sdlc_pytorch
claude-code.txt over trajectory.json to capture Task subagent MCP calls.n_input_tokens from result.json which inflated costs 50-100x for MCP runs.__broken_verifier, validation_test, archive, __archived.config/batch_ts/task__hash/ and config/task__hash/ layouts.task_metrics.json with corrected data. Original result.json is never modified./mcp-audit — Analyzes MCP usage patterns from task_metrics.json (run AFTER re-extraction)/generate-report — Generates evaluation report (uses task_metrics.json data)/evaluate-traces — Comprehensive trace evaluationArchive old completed benchmark runs to save disk space and speed up scans. Triggers on archive runs, clean up runs, disk space, old runs.
Audit benchmark suites against ABC framework (Task/Outcome/Reporting validity). Checks instruction quality, verifier correctness, reproducibility. Triggers on benchmark audit, audit benchmark, abc audit, task validity.
Verify infrastructure readiness before launching benchmark runs — tokens, Docker, disk, credentials. Triggers on check infra, infrastructure check, ready to run, pre-run check.
Compare benchmark results across agent configurations (baseline, SG_full). Show where configs diverge. Triggers on compare configs, config comparison, which config wins, MCP impact.
Token and cost analysis per run, suite, and config. Shows most expensive tasks and config cost comparison. Triggers on cost report, how much did it cost, token usage, spending.
Generate the aggregate CSB evaluation report from completed Harbor runs. Triggers on generate report, eval report, ccb report, benchmark report.