Audit benchmark suites against ABC framework (Task/Outcome/Reporting validity). Checks instruction quality, verifier correctness, reproducibility. Triggers on benchmark audit, audit benchmark, abc audit, task validity.

2026-03-1729

check-infra

sourcegraph/CodeScaleBench

Verify infrastructure readiness before launching benchmark runs — tokens, Docker, disk, credentials. Triggers on check infra, infrastructure check, ready to run, pre-run check.

2026-03-1729

compare-configs

sourcegraph/CodeScaleBench

Compare benchmark results across agent configurations (baseline, SG_full). Show where configs diverge. Triggers on compare configs, config comparison, which config wins, MCP impact.

2026-03-1729

cost-report

sourcegraph/CodeScaleBench

Token and cost analysis per run, suite, and config. Shows most expensive tasks and config cost comparison. Triggers on cost report, how much did it cost, token usage, spending.

2026-03-1729

generate-report

sourcegraph/CodeScaleBench

Generate the aggregate CSB evaluation report from completed Harbor runs. Triggers on generate report, eval report, ccb report, benchmark report.

2026-03-1729

name	scaffold
description	Create, mine, and validate new benchmark tasks and task suites.

Skill: Task Scaffolding & Authoring

Scope

Use this skill when the user asks to:

Create new benchmark tasks or suites
Mine tasks from source repositories
Scaffold task definitions with metadata and structure
Validate task authoring and compliance
Update task metadata and verifiers

Canonical Commands

# Scaffold new task from template
python3 scripts/authoring/scaffold_task.py --suite csb_sdlc_debug --task-name my-task-001

# Mine tasks from repo
python3 scripts/authoring/mine_bug_tasks.py --source-repo https://github.com/org/repo

# Validate task structure
python3 scripts/authoring/validate_tasks_preflight.py --task benchmarks/csb/debug/my-task-001

# Create task specifications from ground truth
python3 scripts/infra/build_daytona_registry.py --task benchmarks/csb/debug/my-task-001

# Backfill task metadata
python3 scripts/evaluation/backfill_instruction_artifacts.py --task-id my-task-001

Task Definition Structure

benchmarks/csb/CATEGORY/TASK_ID/
├── task.toml                 # Metadata: language, difficulty, time_limit_sec
├── instruction.md            # Task description and requirements
├── instruction_mcp.md        # MCP-specific retrieval hints
├── environment/
│   ├── Dockerfile
│   └── Dockerfile.sg_only    # Sourcegraph-only variant
├── tests/
│   ├── test.sh               # Main verifier
│   ├── oracle_checks.py      # Ground truth validation
│   ├── task_spec.json        # Task contract
│   └── ground_truth.json     # Reference solution
└── reviewers.json            # Metadata: author, reviewer

Task Authoring Steps

Create — scaffold template with metadata
Write — instruction.md, environment/Dockerfile
Implement — test.sh and oracle_checks.py
Validate — validate_tasks_preflight.py
Review — ensure compliance before inclusion

Related Skills

/audit — validate task definitions
/run — execute new tasks
/evaluate — score task results