validate-tasks

name	validate-tasks
description	Pre-flight validation of benchmark tasks before launching runs. Catches truncated instructions, metadata mismatches, missing test scripts. Triggers on validate tasks, preflight, pre-flight check, check tasks.
user-invocable	true

Validate Tasks

Run pre-flight checks on benchmark task definitions to catch problems before committing to multi-hour runs.

What It Catches

• Truncated or missing instruction.md (< 200 chars)
• Template placeholders left in instructions (#ISSUE_NUMBER, {{...}})
• Missing or non-executable tests/test.sh
• Language/difficulty mismatches between task.toml and selected_benchmark_tasks.json
• Tasks not registered in the selection registry
• expected_changes.json referencing repos not mentioned in instruction.md (crossrepo)
• Known bad flags in test.sh (--output_path vs --result_path)

Steps

1. Run validation

For a specific suite (most common before a run):

cd ~/CodeScaleBench && python3 scripts/validate_tasks_preflight.py --suite csb_sdlc_pytorch

For all selected tasks:

cd ~/CodeScaleBench && python3 scripts/validate_tasks_preflight.py --all

For a single task:

cd ~/CodeScaleBench && python3 scripts/validate_tasks_preflight.py --task benchmarks/csb_sdlc_pytorch/sgt-005

2. Present results

Show issues grouped by severity:

• CRITICAL: Will definitely cause run failures — must fix before launching
• WARNING: May affect results quality — should fix
• INFO: Informational (e.g., task not in selection registry)

3. Offer to fix

If issues are found, offer to fix them:

• Truncated instruction → investigate and regenerate
• Language mismatch → update task.toml to match selection registry
• test.sh not executable → chmod +x
• Template placeholders → need manual replacement

Variants

JSON output (for piping)

python3 scripts/validate_tasks_preflight.py --all --format json

Critical only

python3 scripts/validate_tasks_preflight.py --all --critical-only