Menü
Create, mine, and validate new benchmark tasks and task suites.
Mit Codex oder Claude installieren Kopieren Sie diesen Prompt, fügen Sie ihn in Codex, Claude oder einen anderen Assistant ein und lassen Sie die Skill-Seite prüfen und installieren.
Basierend auf der SOC-Berufsklassifikation
Archive old completed benchmark runs to save disk space and speed up scans. Triggers on archive runs, clean up runs, disk space, old runs.
Audit benchmark suites against ABC framework (Task/Outcome/Reporting validity). Checks instruction quality, verifier correctness, reproducibility. Triggers on benchmark audit, audit benchmark, abc audit, task validity.
Verify infrastructure readiness before launching benchmark runs — tokens, Docker, disk, credentials. Triggers on check infra, infrastructure check, ready to run, pre-run check.
Compare benchmark results across agent configurations (baseline, SG_full). Show where configs diverge. Triggers on compare configs, config comparison, which config wins, MCP impact.
Token and cost analysis per run, suite, and config. Shows most expensive tasks and config cost comparison. Triggers on cost report, how much did it cost, token usage, spending.
Generate the aggregate CSB evaluation report from completed Harbor runs. Triggers on generate report, eval report, ccb report, benchmark report.
| name | scaffold |
| description | Create, mine, and validate new benchmark tasks and task suites. |
Use this skill when the user asks to:
# Scaffold new task from template
python3 scripts/authoring/scaffold_task.py --suite csb_sdlc_debug --task-name my-task-001
# Mine tasks from repo
python3 scripts/authoring/mine_bug_tasks.py --source-repo https://github.com/org/repo
# Validate task structure
python3 scripts/authoring/validate_tasks_preflight.py --task benchmarks/csb/debug/my-task-001
# Create task specifications from ground truth
python3 scripts/infra/build_daytona_registry.py --task benchmarks/csb/debug/my-task-001
# Backfill task metadata
python3 scripts/evaluation/backfill_instruction_artifacts.py --task-id my-task-001
benchmarks/csb/CATEGORY/TASK_ID/
├── task.toml # Metadata: language, difficulty, time_limit_sec
├── instruction.md # Task description and requirements
├── instruction_mcp.md # MCP-specific retrieval hints
├── environment/
│ ├── Dockerfile
│ └── Dockerfile.sg_only # Sourcegraph-only variant
├── tests/
│ ├── test.sh # Main verifier
│ ├── oracle_checks.py # Ground truth validation
│ ├── task_spec.json # Task contract
│ └── ground_truth.json # Reference solution
└── reviewers.json # Metadata: author, reviewer
validate_tasks_preflight.py/audit — validate task definitions/run — execute new tasks/evaluate — score task results