tracey-verify

name	tracey-verify
description	Audit spec coverage using tracey MCP tools. Finds uncovered, untested, and stale rules, then semantically verifies that test code actually exercises the spec requirements. Use when the user asks to verify tracey coverage, audit specs, check spec fulfillment, or run a tracey verification.

Tracey Verification

Perform a two-phase audit: (1) identify coverage gaps, (2) semantically verify that tests marked as verified actually match their spec text.

Scope restriction: NEVER read implementation code. Only read test files and the spec (plan/spec.md).

All tracey calls use CallMcpTool.

---

Phase 1: Coverage Gap Report

Run these MCP calls in parallel:

Tool	Arguments	Purpose
tracey_status	{}	Overall coverage percentages
tracey_validate	{}	Structural errors (circular deps, unknown refs)
tracey_uncovered	{}	Rules with no [verify ...]
tracey_untested	{}	Rules implemented but not tested
tracey_stale	{}	References pointing at old rule versions

Present the results as a summary table:

## Coverage Summary
- Total rules: N
- Verified (tested): N
- Uncovered (no verification): N
- Untested (impl but no test): N
- Stale references: N
- Validation errors: N

List each gap with its rule ID. For uncovered rules, include the spec text (available from tracey_rule).

---

Phase 2: Semantic Audit

For every rule that IS marked as verified, confirm the test actually exercises the spec requirement. This catches superficial annotations.

Step 1: Collect verified rules

From the Phase 1 status output, identify all rules that have verification references. These are the rules NOT listed in the uncovered/untested outputs.

Step 2: Fetch rule details

For each verified rule, call tracey_rule with { "rule_id": "<id>" }. This returns:

• The spec text
• Verification reference locations (file path + line number)

Batch these calls — issue multiple tracey_rule calls in parallel (up to ~10 at a time).

Step 3: Read and audit test code

For each verification reference location:

1. Read the test file at the referenced location. Read enough context to capture the full test function (typically 50-100 lines from the annotation).
2. Compare the test against the spec text. Check:
   • Preconditions: Does the test set up the scenario the spec describes?
   • Assertions: Does the test assert the behavior the spec requires?
   • Substance: Is the test actually exercising the rule, or is the annotation placed on an unrelated test?

Use parallel subagents (Task tool with subagent_type="explore") to read and audit multiple test files concurrently when there are many rules. Each subagent should receive the rule ID, spec text, and file location, and return a verdict.

Step 4: Classify each rule

• PASS: Test clearly exercises the spec requirement.
• WEAK: Test partially covers the requirement or assertions are insufficient.
• MISMATCH: Test does not match the spec text — annotation appears wrong.

---

Output Format

Present the final report to the user:

## Tracey Verification Report

### Coverage Summary
| Metric | Count |
|--------|-------|
| Total rules | N |
| Verified | N |
| Uncovered | N |
| Untested | N |
| Stale | N |

### Gaps

#### Uncovered Rules (no verification)
| Rule ID | Spec Text |
|---------|-----------|
| ... | ... |

#### Untested Rules (impl only)
| Rule ID |
|---------|
| ... |

#### Stale References
| Rule ID | Location | Issue |
|---------|----------|-------|
| ... | ... | ... |

### Audit Findings

#### Issues Found
| Rule ID | Verdict | Test Location | Notes |
|---------|---------|---------------|-------|
| ... | WEAK | tests/tests/foo.rs:42 | Missing assertion for X |
| ... | MISMATCH | tests/tests/bar.rs:10 | Test checks Y, spec requires Z |

#### Passed (N rules verified correctly)
<collapsed list or omitted for brevity>

If all verified rules pass, say so and skip the detailed pass list. Focus the user's attention on gaps and issues.