cli-differential-fuzzing

name	cli-differential-fuzzing
description	Use when a terminal benchmark replacement mostly works but edge cases are uncertain, and the agent should generate bounded randomized stdin, file, or argv cases to compare original and candidate stdout, stderr, and exit code.
tags	programbench, cli, fuzzing, differential-testing, edge-cases

CLI Differential Fuzzing

Use this skill after a first replacement implementation exists and there is a saved original executable to compare against.

This is bounded fuzzing for terminal benchmarks. The goal is not to fuzz forever or only find crashes. The goal is to quickly find behavior mismatches between:

/tmp/original-executable
./executable

Always compare:

stdout
stderr
exit_code
timeout behavior

Assume the benchmark Docker image has no fuzzing tools installed. Do not install packages. Use Python standard library fuzzing.

When to Use

Use this skill when:

- the implementation passes obvious examples but hidden edge cases are likely
- the CLI is parser/filter/formatter/search/converter-like
- stdin, files, or argv behavior is under-specified
- a small mismatch could be caused by whitespace, stderr, or exit code

Do not use it when:

- no candidate implementation exists yet
- the original program is interactive or very slow
- every execution has heavy side effects
- time is so limited that a focused hand-written case is better

Require the Original

This skill requires a preserved original oracle. Do not create it from the current ./executable after implementation has started because that may copy the candidate and make comparisons meaningless.

test -x /tmp/original-executable || { echo "missing original oracle; stop fuzzing"; exit 1; }

If /tmp/original-executable was not saved before overwriting ./executable, first determine whether the benchmark keeps another true original path. Do not assume. Do not compare a candidate against itself.

Seed Corpus

Create a tiny seed corpus from observed behavior and likely edge cases:

mkdir -p /tmp/cli-fuzz-seeds
printf '' > /tmp/cli-fuzz-seeds/empty.txt
printf '\n' > /tmp/cli-fuzz-seeds/newline.txt
printf 'abc\n' > /tmp/cli-fuzz-seeds/simple.txt
printf 'a b c\n' > /tmp/cli-fuzz-seeds/spaces.txt
printf 'hello\nworld\n' > /tmp/cli-fuzz-seeds/multiline.txt
printf '0\n1\n-1\n' > /tmp/cli-fuzz-seeds/numbers.txt

Add domain-specific seeds from earlier probes. Keep seeds small.

Generate Fuzz Inputs

Use a simple deterministic Python mutator by default. It works in normal ProgramBench Docker images and needs only Python standard library:

python3 - <<'PY'
from pathlib import Path
import random

random.seed(1)
seed_dir = Path('/tmp/cli-fuzz-seeds')
seeds = [p.read_bytes() for p in seed_dir.glob('*') if p.is_file()]
atoms = [
    b'', b'\n', b'\r\n', b' ', b'\t',
    b'-', b'--', b'---', b'_', b'.', b',', b':',
    b'0', b'1', b'-1', b'999999999',
    b'abc', b'ABC', b'a b', b'hello\nworld\n',
    b'\\x00', b'\\\\', b'/', b'//', b'../',
    b'"', b"'", b'[]', b'{}', b'()',
]
out = Path('/tmp')

def mutate(seed: bytes) -> bytes:
    data = bytearray(seed)
    for _ in range(random.randint(1, 6)):
        op = random.choice(['insert', 'delete', 'flip', 'repeat', 'truncate'])
        if op == 'insert':
            pos = random.randrange(len(data) + 1)
            data[pos:pos] = random.choice(atoms)
        elif op == 'delete' and data:
            del data[random.randrange(len(data))]
        elif op == 'flip' and data:
            data[random.randrange(len(data))] = random.randrange(256)
        elif op == 'repeat' and 0 < len(data) < 4096:
            data *= random.randint(1, 3)
        elif op == 'truncate' and data:
            del data[random.randrange(len(data)):]
    return bytes(data[:8192])

for i in range(100):
    seed = random.choice(seeds or [b''])
    (out / f'cli-fuzz-{i}.in').write_bytes(mutate(seed))
PY

If Python is unavailable, skip randomized fuzzing and use hand-written edge cases from the seed corpus.

Stdin Fuzz

Compare original and candidate on the generated stdin files:

for f in /tmp/cli-fuzz-*.in; do
  timeout 3 /tmp/original-executable < "$f" >/tmp/orig.out 2>/tmp/orig.err
  orig_code=$?
  timeout 3 ./executable < "$f" >/tmp/new.out 2>/tmp/new.err
  new_code=$?

  if ! cmp -s /tmp/orig.out /tmp/new.out || \
     ! cmp -s /tmp/orig.err /tmp/new.err || \
     [ "$orig_code" != "$new_code" ]; then
    echo "MISMATCH stdin file=$f"
    echo "orig_exit=$orig_code new_exit=$new_code"
    printf -- '--- input ---\n'
    sed -n '1,80p' "$f"
    printf -- '--- stdout diff ---\n'
    diff -u /tmp/orig.out /tmp/new.out || true
    printf -- '--- stderr diff ---\n'
    diff -u /tmp/orig.err /tmp/new.err || true
    break
  fi
done

Use 20 to 200 generated cases. More than that is usually wasteful inside benchmark runs.

Single Python Differential Runner

When shell loops become noisy, use this single Python runner. It generates cases and compares stdin behavior in one command:

python3 - <<'PY'
from pathlib import Path
import random
import subprocess

random.seed(1)
orig = '/tmp/original-executable'
new = './executable'
seeds = [p.read_bytes() for p in Path('/tmp/cli-fuzz-seeds').glob('*') if p.is_file()]
atoms = [b'', b'\n', b'\r\n', b' ', b'\t', b'-', b'--', b'0', b'1', b'-1', b'abc', b'\\x00', b'\\\\', b'"', b"'"]

def mutate(seed):
    data = bytearray(seed)
    for _ in range(random.randint(1, 6)):
        op = random.choice(['insert', 'delete', 'flip', 'repeat', 'truncate'])
        if op == 'insert':
            pos = random.randrange(len(data) + 1)
            data[pos:pos] = random.choice(atoms)
        elif op == 'delete' and data:
            del data[random.randrange(len(data))]
        elif op == 'flip' and data:
            data[random.randrange(len(data))] = random.randrange(256)
        elif op == 'repeat' and 0 < len(data) < 4096:
            data *= random.randint(1, 3)
        elif op == 'truncate' and data:
            del data[random.randrange(len(data)):]
    return bytes(data[:8192])

for i in range(100):
    inp = mutate(random.choice(seeds or [b'']))
    try:
        o = subprocess.run([orig], input=inp, stdout=subprocess.PIPE, stderr=subprocess.PIPE, timeout=3)
        n = subprocess.run([new], input=inp, stdout=subprocess.PIPE, stderr=subprocess.PIPE, timeout=3)
    except subprocess.TimeoutExpired as exc:
        print('TIMEOUT case', i, 'cmd=', exc.cmd)
        break
    if (o.returncode, o.stdout, o.stderr) != (n.returncode, n.stdout, n.stderr):
        p = Path(f'/tmp/cli-fuzz-mismatch-{i}.in')
        p.write_bytes(inp)
        print('MISMATCH stdin case', i, 'saved=', p)
        print('orig_exit=', o.returncode, 'new_exit=', n.returncode)
        print('input=', repr(inp[:500]))
        print('orig_stdout=', repr(o.stdout[:500]))
        print('new_stdout=', repr(n.stdout[:500]))
        print('orig_stderr=', repr(o.stderr[:500]))
        print('new_stderr=', repr(n.stderr[:500]))
        break
else:
    print('NO_MISMATCH cases=100')
PY

Argv Fuzz

Use Python subprocess.run([...]) for argv fuzz. Avoid shell string interpolation because quoting bugs create false mismatches.

python3 - <<'PY'
import subprocess

orig = '/tmp/original-executable'
new = './executable'
cases = [
    [],
    ['-h'],
    ['--help'],
    ['--version'],
    ['--bad'],
    ['-'],
    ['--'],
    ['foo'],
    ['foo', 'bar'],
    ['/no/such/file'],
    ['', ''],
    ['a b'],
]

for args in cases:
    o = subprocess.run([orig, *args], input=b'', stdout=subprocess.PIPE, stderr=subprocess.PIPE, timeout=3)
    n = subprocess.run([new, *args], input=b'', stdout=subprocess.PIPE, stderr=subprocess.PIPE, timeout=3)
    if (o.returncode, o.stdout, o.stderr) != (n.returncode, n.stdout, n.stderr):
        print('MISMATCH argv:', repr(args))
        print('orig_exit=', o.returncode, 'new_exit=', n.returncode)
        print('orig_stdout=', o.stdout[:500])
        print('new_stdout=', n.stdout[:500])
        print('orig_stderr=', o.stderr[:500])
        print('new_stderr=', n.stderr[:500])
        break
PY

Add domain-specific argv cases from help output and earlier observations.

File Input Fuzz

For file-oriented tools, write mutated content to a temp file and pass that filename:

for f in /tmp/cli-fuzz-*.in; do
  timeout 3 /tmp/original-executable "$f" >/tmp/orig.out 2>/tmp/orig.err
  orig_code=$?
  timeout 3 ./executable "$f" >/tmp/new.out 2>/tmp/new.err
  new_code=$?

  if ! cmp -s /tmp/orig.out /tmp/new.out || \
     ! cmp -s /tmp/orig.err /tmp/new.err || \
     [ "$orig_code" != "$new_code" ]; then
    echo "MISMATCH file input=$f"
    echo "orig_exit=$orig_code new_exit=$new_code"
    diff -u /tmp/orig.out /tmp/new.out || true
    diff -u /tmp/orig.err /tmp/new.err || true
    break
  fi
done

Also test missing files, empty files, directories, and multiple files if relevant.

Triage Mismatches

Before editing, classify the first mismatch:

argv parsing
stdin/file selection
binary/unicode handling
empty input
newline/trailing whitespace
stdout formatting
stderr wording
exit code
timeout/performance
side effect

Fix only the smallest likely cause. Then rerun the failing case plus a small fuzz batch.

Stop Conditions

Stop fuzzing when:

- the first high-value mismatch is found
- 100-200 cases pass for the relevant input mode
- fuzzing reveals only irrelevant unsupported behavior
- timeouts or side effects make results noisy

Do not spend the whole benchmark budget fuzzing. Fuzzing is a tool to find the next useful fix.

Report Format

Return a compact report:

Fuzz report:
- original path:
- candidate path:
- mode tested: stdin | argv | file
- cases run:
- mismatch found: yes/no
- failing input/args:
- stdout diff summary:
- stderr diff summary:
- exit-code diff:
- likely cause:
- recommended fix: