test-first-evidence

name	test-first-evidence
description	Record failing-test evidence, waivers, and final validation through the nils-cli test-first-evidence command.

Test-First Evidence

Use this skill when an agent workflow needs a deterministic record of before-fix evidence or an explicit waiver plus final validation.

Contract

Prereqs:

• Run inside the target project or from a directory where evidence paths make sense.
• Choose an explicit output directory, preferably under a project-scoped agent-out run directory.
• Released usage: test-first-evidence available on PATH from nils-cli 0.8.4 or newer.
• Local checkout fallback usage: Rust/Cargo plus a validated local nils-cli checkout that builds the nils-test-first-evidence package when PATH is absent or too old.

Inputs:

• init: required --out DIR and --classification TEXT; optional repeatable --production-path PATH, repeatable --note TEXT, optional --force, and optional --format text|json.
• record-failing: required --out DIR, --command TEXT, --exit-code CODE, and --summary TEXT; optional --test-name TEXT, repeatable --artifact PATH, and optional --format text|json.
• record-waiver: required --out DIR and --reason TEXT; optional repeatable --substitute-validation TEXT, and optional --format text|json.
• record-final: required --out DIR, --command TEXT, and --status pass|fail; optional --summary TEXT, repeatable --artifact PATH, and optional --format text|json.
• verify / show: required --out DIR; optional --format text|json.

Outputs:

• Writes test-first-evidence.json under --out DIR.
• JSON stdout uses versioned schema values such as cli.test-first-evidence.verify.v1.
• The record schema is test-first-evidence.record.v1.
• Secret-like tokens in recorded text are redacted by the CLI.

Exit codes:

• 0: command succeeded; for verify, evidence is complete.
• 1: runtime failure or incomplete evidence.
• 64: usage error.

Failure modes:

• test-first-evidence is unavailable on PATH and no validated local checkout invocation is being used.
• Evidence directory cannot be created or written.
• verify finds neither failing evidence nor waiver, missing final validation, or final validation status is not pass.
• Caller records unhelpful waiver text instead of a concrete reason and substitute validation; the workflow must reject weak evidence even if the CLI can store it.

Setup

Released PATH boundary:

test-first-evidence --help
test-first-evidence --version

Use the PATH command after installing nils-cli 0.8.4 or newer with nils-test-first-evidence on PATH.

Local checkout fallback boundary:

cargo run --locked --manifest-path "$HOME/Project/sympoies/nils-cli/Cargo.toml" \
  -p nils-test-first-evidence --bin test-first-evidence -- --help

Run the Cargo form from the workflow's target directory only when PATH is absent or reports an older nils-cli. Keep the same test-first-evidence subcommands and flags in both modes.

Commands (only entrypoints)

Released PATH command:

test-first-evidence init --out <dir> --classification <classification> [--production-path <path> ...] [--format json]
test-first-evidence record-failing --out <dir> --command <command> --exit-code <code> --summary <summary> [--format json]
test-first-evidence record-waiver --out <dir> --reason <reason> [--substitute-validation <text> ...] [--format json]
test-first-evidence record-final --out <dir> --command <command> --status pass|fail [--summary <summary>] [--format json]
test-first-evidence verify --out <dir> [--format json]
test-first-evidence show --out <dir> [--format json]
test-first-evidence completion <bash|zsh>

Local checkout fallback command:

cargo run --locked --manifest-path /path/to/nils-cli/Cargo.toml \
  -p nils-test-first-evidence --bin test-first-evidence -- <subcommand> ...

Do not hand-edit test-first-evidence.json or duplicate redaction, completeness, or JSON envelope logic in skill-local scripts.

Workflow

1. Create or choose a run directory for the task: agent-out project --topic <topic> --mkdir
2. Initialize the record before production edits: test-first-evidence init --out <run-dir>/test-first --classification <classification> --production-path <path> --format json
3. Record either failing evidence or a waiver before production edits: test-first-evidence record-failing ... or test-first-evidence record-waiver ...
4. After implementation, record final validation: test-first-evidence record-final --out <run-dir>/test-first --command <command> --status pass --format json
5. Before reporting completion, verify: test-first-evidence verify --out <run-dir>/test-first --format json
6. Cite the generated record path and summarize only the evidence fields needed for the user.

References

• docs/runbooks/skills/TOOLING_INDEX_V2.md