| name | test-first-evidence |
| description | Record failing-test evidence, waivers, and final validation through the nils-cli test-first-evidence command. |
Test-First Evidence
Use this skill when an agent workflow needs a deterministic record of before-fix evidence or an explicit waiver plus final validation.
Contract
Prereqs:
- Run inside the target project or from a directory where evidence paths make sense.
- Choose an explicit output directory, preferably under a project-scoped
agent-out run directory.
- Released usage:
test-first-evidence available on PATH from nils-cli 0.8.4 or newer.
- Local checkout fallback usage: Rust/Cargo plus a validated local
nils-cli checkout that builds the
nils-test-first-evidence package when PATH is absent or too old.
Inputs:
init: required --out DIR and --classification TEXT; optional repeatable --production-path PATH, repeatable --note TEXT,
optional --force, and optional --format text|json.record-failing: required --out DIR, --command TEXT, --exit-code CODE, and --summary TEXT; optional --test-name TEXT,
repeatable --artifact PATH, and optional --format text|json.record-waiver: required --out DIR and --reason TEXT; optional repeatable --substitute-validation TEXT, and optional
--format text|json.record-final: required --out DIR, --command TEXT, and --status pass|fail; optional --summary TEXT, repeatable
--artifact PATH, and optional --format text|json.verify / show: required --out DIR; optional --format text|json.
Outputs:
- Writes
test-first-evidence.json under --out DIR.
- JSON stdout uses versioned schema values such as
cli.test-first-evidence.verify.v1.
- The record schema is
test-first-evidence.record.v1.
- Secret-like tokens in recorded text are redacted by the CLI.
Exit codes:
0: command succeeded; for verify, evidence is complete.1: runtime failure or incomplete evidence.64: usage error.
Failure modes:
test-first-evidence is unavailable on PATH and no validated local checkout invocation is being used.- Evidence directory cannot be created or written.
verify finds neither failing evidence nor waiver, missing final validation, or final validation status is not pass.- Caller records unhelpful waiver text instead of a concrete reason and substitute validation; the workflow must reject weak evidence even if
the CLI can store it.
Setup
Released PATH boundary:
test-first-evidence --help
test-first-evidence --version
Use the PATH command after installing nils-cli 0.8.4 or newer with nils-test-first-evidence on PATH.
Local checkout fallback boundary:
cargo run --locked --manifest-path "$HOME/Project/sympoies/nils-cli/Cargo.toml" \
-p nils-test-first-evidence --bin test-first-evidence -- --help
Run the Cargo form from the workflow's target directory only when PATH is absent
or reports an older nils-cli. Keep the same test-first-evidence subcommands
and flags in both modes.
Commands (only entrypoints)
Released PATH command:
test-first-evidence init --out <dir> --classification <classification> [--production-path <path> ...] [--format json]
test-first-evidence record-failing --out <dir> --command <command> --exit-code <code> --summary <summary> [--format json]
test-first-evidence record-waiver --out <dir> --reason <reason> [--substitute-validation <text> ...] [--format json]
test-first-evidence record-final --out <dir> --command <command> --status pass|fail [--summary <summary>] [--format json]
test-first-evidence verify --out <dir> [--format json]
test-first-evidence show --out <dir> [--format json]
test-first-evidence completion <bash|zsh>
Local checkout fallback command:
cargo run --locked --manifest-path /path/to/nils-cli/Cargo.toml \
-p nils-test-first-evidence --bin test-first-evidence -- <subcommand> ...
Do not hand-edit test-first-evidence.json or duplicate redaction, completeness, or JSON envelope logic in skill-local scripts.
Workflow
- Create or choose a run directory for the task:
agent-out project --topic <topic> --mkdir
- Initialize the record before production edits:
test-first-evidence init --out <run-dir>/test-first --classification <classification> --production-path <path> --format json
- Record either failing evidence or a waiver before production edits:
test-first-evidence record-failing ... or test-first-evidence record-waiver ...
- After implementation, record final validation:
test-first-evidence record-final --out <run-dir>/test-first --command <command> --status pass --format json
- Before reporting completion, verify:
test-first-evidence verify --out <run-dir>/test-first --format json
- Cite the generated record path and summarize only the evidence fields needed for the user.
References
docs/runbooks/skills/TOOLING_INDEX_V2.md
