test-failure-triage

name	test-failure-triage
description	Use when tests, eval, build, or smoke checks fail and the agent needs to classify the failure, extract the smallest reproducer, identify likely cause, and choose the next command or edit without getting lost in long logs.
tags	testing, debugging, triage, programbench, failures

Test Failure Triage

Use this skill whenever a test, build, eval, or smoke command fails.

The goal is to turn noisy output into one precise next action.

Triage Shape

Always produce this internal summary before editing:

failure type:
smallest failing command:
observed output:
expected output or contract:
likely cause:
next command:
next edit:

Failure Types

Classify first:

missing executable/artifact
permission error
compile error
missing dependency
wrong stdout
wrong stderr
wrong exit code
exception type/message mismatch
wrong API signature/import path
file side-effect mismatch
timeout/performance
test harness/environment issue
empty or invalid eval result

Do not patch until the category is clear.

Log Reduction

Keep logs small and targeted:

cmd > /tmp/fail.out 2> /tmp/fail.err; code=$?
echo "exit=$code"
tail -80 /tmp/fail.out
tail -120 /tmp/fail.err

Search for high-signal lines:

rg -n "error|failed|failure|panic|traceback|exception|expected|actual|assert|not found|permission|timeout" /tmp/fail.out /tmp/fail.err

If a full test suite fails, rerun the smallest test or single command that reproduces the failure.

Compare Contract

For CLI/executable behavior, compare the triple:

stdout
stderr
exit_code

For library/API behavior, compare:

import path
function/class name
argument handling
return value
exception type
exception message
side effects

For build behavior, compare:

build command
generated artifact path
executable bit
runtime dynamic library errors
missing tool or dependency

Next Action Rules

Use these defaults:

missing executable -> inspect compile.sh and expected output path
permission error -> chmod executable or fix compile.sh
compile error -> fix first compiler error, not all later cascades
missing dependency -> avoid new dependency; use standard library or vendored code
wrong stdout/stderr -> create one golden case and diff
wrong exit code -> inspect argparse/error path or exception handling
API mismatch -> inspect tests/imports before changing behavior
timeout -> reduce algorithmic complexity or add fast path
empty eval -> inspect eval file/container/log, not implementation first

Stop Condition

Stop triage once there is one minimal failing command and one likely cause. Then fix that cause and rerun the same command.