terminal-bench

name	terminal-bench
description	Use for ProgramBench or terminal-benchmark tasks where an agent must solve an unknown software task through terminal interaction, infer the required behavior from artifacts and black-box observations, implement the smallest correct replacement or fix, validate it, and prepare a compliant submission. Applies to CLI tools, libraries, test-driven tasks, build failures, file converters, parsers, and executable behavior replication.
tags	programbench, terminal, black-box, testing, implementation, submission

Terminal Bench

Use this skill when solving a benchmark task primarily through terminal interaction. The task may be a CLI behavior clone, a library/API fix, a parser/formatter/converter, a build/test failure, or a small application with hidden tests.

The priority is a tight loop:

understand the contract -> establish an oracle -> implement minimally -> validate -> repair -> submit cleanly

Do not default to broad repository exploration. Gather enough evidence to choose the next action.

Core Rule

Preserve observable behavior. Depending on the task, the observable contract may include:

stdout
stderr
exit_code
files created or modified
return values
exceptions and error messages
serialization format
ordering and whitespace
performance on small cases
build artifacts

For executable/CLI tasks, always record stdout, stderr, and exit_code together.

Phase 1: Identify the Task Surface

First classify what kind of benchmark this is:

cli/executable clone: reproduce ./executable behavior
library/API task: tests import functions/classes and check behavior
parser/formatter/converter: transform input files or text
build repair: make compile/test/install succeed
algorithmic implementation: produce correct outputs for known input classes
integration wrapper: connect existing project pieces into expected executable

Use focused inspection:

pwd
find . -maxdepth 2 -type f | sort | sed -n '1,120p'
find . -maxdepth 2 -type f \( -name 'README*' -o -name '*test*' -o -name 'compile.sh' -o -name 'Makefile' -o -name 'pyproject.toml' -o -name 'package.json' -o -name 'Cargo.toml' \) -print
git status --short

Then read only the files likely to define the contract: task statement, tests, README snippets, build files, or examples.

Phase 2: Establish an Oracle

Choose the strongest available oracle:

existing tests: run the smallest relevant test first
original executable: compare black-box behavior
examples/docs: convert examples into checks
type/signature expectations: inspect imports and public API
build contract: compile.sh must create expected artifact

If ./executable is the original oracle for a CLI/executable clone, preserve it immediately before editing, rebuilding, or running compile.sh:

test -x ./executable && cp ./executable /tmp/original-executable

After implementation starts, do not recreate /tmp/original-executable from ./executable; it may already be the candidate.

For CLI/executable tasks, use a small behavior harness:

run_case() {
  name="$1"
  shift
  echo "=== $name ==="
  "$@" >"/tmp/${name}.out" 2>"/tmp/${name}.err"
  code=$?
  printf 'exit=%s\n' "$code"
  printf -- '--- stdout ---\n'
  cat "/tmp/${name}.out"
  printf -- '--- stderr ---\n'
  cat "/tmp/${name}.err"
}

Start with a small matrix:

run_case no_args ./executable
run_case help_long ./executable --help
run_case help_short ./executable -h
run_case bad_flag ./executable --definitely-invalid
printf '' | ./executable
printf 'abc\n' | ./executable

For non-CLI tasks, create tiny examples that isolate the expected behavior. Prefer small one-command checks over running the whole suite repeatedly.

Phase 3: Narrow the Behavior

Convert observations into a compact contract:

inputs accepted:
outputs produced:
errors:
side effects:
ordering/formatting:
edge cases:
unknowns:

Use task-specific probes:

text filters: empty input, multiline input, invalid bytes, trailing newline
file tools: missing file, empty file, multiple files, directory path
parsers: malformed input, comments, whitespace, minimal valid case
formatters: idempotence, blank lines, indentation, invalid input
searchers: no match, invalid pattern, multiple files, stdin vs file
libraries: public API signatures, exception types, boundary values
build tasks: missing dependency, generated binary path, executable bit

Stop probing once the next implementation step is clear.

Phase 4: Implement the Smallest Correct Change

If the implementation path is still unclear, load minimal-implementation-strategy before editing. If the task is primarily a build/artifact problem, load build-system-diagnostics.

Prefer the simplest implementation that satisfies the inferred contract:

- Python is often best for text processing and benchmark wrappers.
- Use the repository's existing language when tests import project code.
- Shell is fine for tiny glue, but avoid brittle pipelines for complex parsing.
- Do not depend on the original executable or hidden benchmark artifacts.
- Do not rewrite unrelated project structure.

Preserve exact details when known:

trailing newlines
spaces and separators
line ordering
stdout vs stderr
exit status
exception type and message
created file names
compile output location

Phase 5: Validate Differentially

After each meaningful change, validate against the oracle.

For CLI/executable clones, compare only if the original oracle was already preserved:

test -x /tmp/original-executable || { echo "missing original oracle; skip differential comparison"; exit 1; }

Compare important cases:

case_one() {
  label="$1"
  shift
  /tmp/original-executable "$@" >/tmp/orig.out 2>/tmp/orig.err
  orig_code=$?
  ./executable "$@" >/tmp/new.out 2>/tmp/new.err
  new_code=$?
  echo "=== $label ==="
  echo "orig_exit=$orig_code new_exit=$new_code"
  diff -u /tmp/orig.out /tmp/new.out || true
  diff -u /tmp/orig.err /tmp/new.err || true
}

If stdin/file/argv edge behavior remains uncertain after obvious cases pass, load cli-differential-fuzzing for bounded Python-only differential fuzzing.

For test-driven tasks:

run the smallest failing test
fix the direct cause
rerun that test
then run the closest broader test group

Classify failures before editing:

argument parsing
input selection
formatting/whitespace
stdout/stderr/exit code
exception type/message
algorithmic logic
state mutation
build path or permissions
unsupported edge case

If a validation command fails with a noisy log, load test-failure-triage and reduce it to one minimal failing command before further edits.

Phase 6: Submission Readiness

Before finishing, load submission-review for the final reproducibility and cleanliness check. At minimum, verify the benchmark contract:

test -f compile.sh && echo compile.sh-ok
test -f compile.sh && sh ./compile.sh
test -e ./executable && echo executable-exists
test -x ./executable && echo executable-is-executable
git status --short

Check these explicitly:

- compile.sh exists when required and creates the expected executable/artifact.
- The final executable or library entry point exists after a clean build.
- The solution does not call the original executable.
- Temporary probes, fuzz files, and logs are not part of the submission.
- Relevant smoke tests still pass.
- A final commit exists if the benchmark expects one.

Command Discipline

Use focused commands:

rg, find, file, strings, head, tail, xxd, diff, cmp, wc, sed -n

Avoid:

- long recursive dumps
- broad test suites before a minimal candidate exists
- destructive cleanup before preserving useful observations
- assuming --help exists
- assuming errors go to stderr
- assuming the task is CLI-only

Bound risky commands:

timeout 5 ./executable ...

Report Format

After using this skill, summarize findings compactly:

Terminal-bench summary:
- task surface:
- oracle used:
- inferred contract:
- implementation choice:
- validations run:
- remaining uncertainty:
- next step: