testing

## The NL-TDD Cycle ``` 1. Write spec (.nlpm-test/artifact-name.spec.md) — define expectations 2. /nlpm:test — RED: spec fails (artifact doesn't exist) 3. Write the artifact — create the NL artifact 4. /nlpm:test — check if it passes 5. /nlpm:score — check quality score 6. Iterate until GREEN: all specs pass + score ≥ threshold ``` ## Spec File Format Location: `.nlpm-test/` directory in the project root (or alongside the artifact). Filename convention: `<artifact-name>.spec.md` — matches the artifact filename without path. ```yaml --- artifact: agents/my-agent.md # path to the artifact being tested type: agent # agent | skill | command | rule | hook | prompt min_score: 85 # minimum /nlpm:score threshold for this artifact --- ``` Body sections (all optional — include what matters for this artifact): ### triggers_on (skills and agents) ```markdown ## Triggers On Queries that SHOULD trigger this artifact: - "review my database migrations before deploying" - "check if these schema changes are safe" - "audit the migration for breaking changes" ``` ### does_not_trigger_on (skills and agents) ```markdown ## Does Not Trigger On Queries that should NOT trigger this artifact: - "write a migration for adding a users table" - "help me with CSS styling" - "deploy to production" ``` ### output_contains (agents and commands) ```markdown ## Output Contains Expected elements in the output: - "## Migration Review" (heading present) - "| Table | Change | Risk |" (table structure) - severity classification (CRITICAL/HIGH/MEDIUM/LOW) ``` ### output_format (agents and commands) ```markdown ## Output Format The output should be a markdown report with: 1. Summary section with counts 2. Findings table with columns: File, Finding, Severity 3. Action items list ``` ### handles_input (commands) ```markdown ## Handles Input | Input | Expected Behavior | |-------|------------------| | (empty) | Score all artifacts in cwd | | directory path | Score artifacts in that directory | | nonexistent path | Error: "Directory not found: {path}" | | file path | Score that single file | ``` ### follows_rules (rules) ```markdown ## Follows Rules Code that SHOULD comply: ```python result: Result[User, AppError] = get_user(id) ``` Code that SHOULD violate: ```python user = get_user(id).unwrap() # should be flagged ``` ``` ### frontmatter_valid (all types) ```markdown ## Frontmatter Valid Required fields: - description: present and trigger-style ("Use when...") - model: sonnet - tools: [Read, Glob, Grep] - skills: [nlpm:conventions, nlpm:scoring] ``` ## Test Results Format ``` NLPM Test Report Spec Artifact Result Details ───────────────────────────────────────────────────────────────────────────────── my-agent.spec.md agents/my-agent.md PASS 5/5 checks my-skill.spec.md skills/core/SKILL.md FAIL 3/5 checks ✗ Trigger: "optimize React hooks" → predicted NO trigger (expected YES) ✗ Score: 68/100 (min: 85) Overall: 1 passed, 1 failed (50%) RED items (fix these): 1. skills/core/SKILL.md — trigger gap: "optimize React hooks" not covered by description 2. skills/core/SKILL.md — score 68 < min 85: missing <example> blocks (R06, -10) ``` ## Best Practices for Specs - Write specs BEFORE writing the artifact (TDD discipline) - 5-10 trigger queries for skills/agents (mix positive and negative) - Include edge cases in `handles_input` for commands - `min_score` should match your project's threshold (default 85 for new artifacts, 70 for legacy) - Specs are living documents — update when behavior requirements change ## Spec File Discovery The tester discovers specs by: 1. Looking in `.nlpm-test/` directory 2. Matching spec filename to artifact filename: `my-agent.spec.md` → `agents/my-agent.md` (uses the `artifact:` frontmatter field) 3. If `artifact` path doesn't exist → spec is RED by default (artifact not yet created — this is the TDD "write test first" state) ## Worked Example: TDD Cycle 1. **RED — write the spec first.** Create `.nlpm-test/my-agent.spec.md` with `artifact: agents/my-agent.md` and a `triggers_on:` block listing 5 user queries the agent should match. The artifact does not exist yet — `/nlpm:test` reports RED with one failure: `artifact not found`. 2. **GREEN — write the artifact.** Create `agents/my-agent.md` with frontmatter (name, description, model, tools) and a body. The description must be specific enough that the listed trigger queries land on it. Re-run `/nlpm:test`: the agent now exists, the tester predicts triggers, and the spec passes. 3. **REFACTOR — change the body, re-run the spec.** Edit the artifact's behavior. The same spec runs unchanged; if a refactor breaks a trigger query or violates a `handles_input` case, the test goes RED. Fix forward, re-run. This is the natural-language analogue of `pytest` or `jest` — specs are the contract; artifacts are the implementation. ## Scope Note This skill covers the spec format and runner contract for NL-TDD. For the scoring rubric the tester compares against, see `nlpm:scoring`. For the schemas of artifact frontmatter the spec checks, see `nlpm:conventions`.