Use when a benchmark task involves compile.sh, executable generation, build failures, missing artifacts, toolchain issues, package managers, or making the final submission reproducible from a clean build.
Use when a terminal benchmark replacement mostly works but edge cases are uncertain, and the agent should generate bounded randomized stdin, file, or argv cases to compare original and candidate stdout, stderr, and exit code.
Use near the start of a benchmark task to choose whether to repair the existing project, write a small replacement executable/library, wrap existing functionality, or implement only the tested subset, while avoiding overbuilding and unavailable dependencies.
Use immediately before finishing a benchmark task to verify the final solution is reproducible, clean, committed if required, and does not rely on temporary files, original executables, hidden artifacts, or accidental environment state.
Use for ProgramBench or terminal-benchmark tasks where an agent must solve an unknown software task through terminal interaction, infer the required behavior from artifacts and black-box observations, implement the smallest correct replacement or fix,…
Use when tests, eval, build, or smoke checks fail and the agent needs to classify the failure, extract the smallest reproducer, identify likely cause, and choose the next command or edit without getting lost in long logs.