| name | audit-specs-tests |
| description | Audit the AltTab test suite for spec/test/code consistency. For every co-located triad in `src/` (`Foo.swift` + `FooSpecs.md` + `FooTests.swift`), it cross-checks that each XCTest method has a matching scenario line in the spec and vice-versa, flags drift between a scenario's description and the test's doc-comment, and reports orphan tests, orphan scenarios, and unpaired files. Use after adding or editing any `*Tests.swift` / `*Specs.md`, before committing a test change, or to get an overall health check of the suite. |
/audit-specs-tests
Project context
AltTab co-locates tests and a living spec next to the source they cover. The convention (see the plan in .claude/plans/ and src/pro/license/ as a worked example) is a triad per concept:
src/<feature>/
├── Foo.swift # production / kernel (app target, or app+test for kernels)
├── FooSpecs.md # the living spec (not compiled)
└── FooTests.swift # XCTest (unit-tests target only)
The spec is documentation tied 1:1 to the test methods. Its ## Test scenarios section lists every test, grouped, as:
### A. <Group name>
- **testAFooDoesBar** — one-line description of what it pins.
This skill's job is to keep the three in sync: every func test… ⇔ exactly one - **test…** — spec line, descriptions roughly agree, and no triad is missing a leg.
What counts as a triad
Pair by base name in the same folder: <Base>Tests.swift ⇔ <Base>Specs.md. A <Base>.swift (or a kernel like SelectionResolver.swift, SearchTestable.swift) in the folder is the code under test, but a spec may legitimately cover several classes (e.g. LicenseManagerSpecs.md documents Clock/Keychain/LicenseAPI too) — so a missing same-name .swift is a note, not an error.
Workflow
-
Enumerate. Find all test and spec files:
find src -name '*Tests.swift' | sort
find src -name '*Specs.md' | sort
Also list any test files still under unit-tests/ and flag them as not yet migrated (the target end-state is zero test files outside src/).
-
For each *Tests.swift, extract the test methods and their doc-comments. Test names:
grep -nE '^\s*func test[A-Za-z0-9_]+\(' src/<path>/FooTests.swift
For each, capture the immediately-preceding /// or // comment lines (the test's intent) and the enclosing // MARK: - group. Note methods wrapped in #if DEBUG.
-
For each *Specs.md, parse the scenario index. In the ## Test scenarios section, collect every ### <Group> heading and every - **testName** — description bullet. Build the set of spec-listed test names and their descriptions.
-
Cross-check each triad and classify findings:
- Orphan test —
func testX exists but no - **testX** — line in the spec. (Most common drift; spec is stale.)
- Orphan scenario — spec lists
testX but no func testX in the suite. (Test deleted/renamed; spec stale.)
- Group mismatch — a test's
// MARK: group and its spec ### group disagree (readability drift, not fatal).
- Description drift — the spec bullet and the test's doc-comment describe materially different behavior. Judge semantically, don't string-match; only flag real divergence.
- Unpaired file — a
*Tests.swift with no *Specs.md (or vice-versa).
-
Lighter code check (kernels only). For decision-kernel files (*Testable.swift, *Resolver.swift), list public/static entry-point functions and enum … Decision cases, and flag any with no scenario referencing them — a hint that a branch is undocumented/untested. Skip this for AppKit-coupled production files (intentionally not unit-tested here).
-
Optionally repair. If asked, regenerate a spec's ## Test scenarios section from the live test methods + their doc-comments (preserving the existing ### Group order where it matches // MARK: sections). Never invent behavior — only restate what the test asserts. Do not edit *Tests.swift from this skill.
Reporting
- A per-triad status table:
folder/Base · #tests · #scenarios · status (✅ in sync / ⚠️ drift / ❌ unpaired).
- A punch list grouped by finding type (orphan tests, orphan scenarios, description drift, unpaired, undocumented kernel branches), each with the file + line.
- The count of test files still outside
src/ (migration progress).
- If nothing is wrong, say so plainly. Then offer to fix the drift (regenerate stale spec scenario indexes) and list exactly which files would change.
