Analyze the overall quality of the apple-fm source. Generate a comprehensive
report.
-
Run unit tests with coverage
npm test
Report total tests, pass/fail, and coverage per file. The thresholds in
vitest.config.ts (statements 80 / branches 75 / functions 80 / lines 80) are
the floor; flag any file that drags the suite under them. cli.ts and
repl.ts are intentionally excluded (thin I/O) — confirm they have no
testable logic that crept in.
Coverage is a floor, not a ceiling. A line/branch/function/statement
percentage only proves each line ran during the suite — it says nothing about
whether the right behavior was asserted, or whether sequences of operations
across internal state boundaries were exercised. A module can sit at 100% on
every metric and still ship basic bugs, because every operation was tested once
from a clean initial state while the transitions between states were never
tried. So treat high coverage as the trigger for the behavioral /
state-transition audit (step 6) — not as a stopping point. Never report a
coverage number as if it were a proof of correctness.
-
Run the linter
npm run lint
Report errors/warnings grouped by rule.
-
Run typecheck
npm run typecheck
Report any type errors.
-
Build the helper (best-effort)
npm run build:helper
On a macOS 26 machine this should emit bin/apple-fm-helper. On other
machines the script no-ops with exit 0 by design — note which happened. If it
built, run bin/apple-fm-helper --probe and report the availability line.
-
Check anti-patterns (conventions from CLAUDE.md):
- Relative imports missing
.js — grep src//tests/ for relative
imports without a .js suffix.
any leaks — grep src/ for : any\b, as any\b, <any>. The house
style is unknown + a narrowing check.- Truthy string checks —
strict-boolean-expressions is on; flag any
if (someString) style test (use === undefined / .length).
- Runtime dependencies —
package.json dependencies must stay empty
(the Node layer only spawns the helper). Flag any addition.
- Node imports the model directly — only the Swift helper (any
.swift
file under apple-fm-helper/, currently main.swift, GuidedGeneration.swift,
and Tools.swift) may reference FoundationModels. Grep src/ for
FoundationModels (should be zero hits outside comments).
- Protocol drift —
tests/fixtures/stub-helper.js must still implement the
same events as docs/4-protocol.md and the Swift helper.
- File length — flag any
src/*.ts over ~200 LOC; prefer splitting.
-
Behavioral / state-transition audit (the part coverage % can't see)
Static checks and a coverage number verify that lines ran; they cannot tell
you a behavior or a transition between internal states is missing. This step
hunts exactly those gaps.
6a. Identify the stateful modules. A module is "stateful" if its behavior
depends on something that persists or changes between calls, not just on the
current arguments. Heuristics — flag any module that has:
- multiple code paths keyed on an internal mode / flag / phase field,
- an explicit state machine or lifecycle (create → use → tear down → recreate),
- a cache with hit/miss/fallback paths, or lazy init / respawn on failure,
- accumulated history that is later mutated, summarized, evicted, or reset.
In apple-fm the prime suspects are:
src/liveSession.ts (LiveSession) — the long-lived
apple-fm-helper --session process: no-process → spawned → busy → dead →
lazy respawn, with id-correlated in-flight commands. Transitions: first
command spawns; a crash mid-command must reject that command and respawn on
the next; overlapping/out-of-order command ids; dispose while busy.src/session.ts (ChatSession) — multi-turn history with automatic
compaction: below compactAtTokens → over threshold → summarize older turns,
keep keepRecentTurns verbatim → continue appending. Transitions: cross the
threshold exactly once vs. repeatedly; a summary that itself pushes back over
threshold; empty history then refill; compaction interleaved with new turns.src/tools/permissions.ts (PermissionPolicy) and
src/tools/registry.ts (ToolRegistry) — per-call mode + remembered
allow/deny decisions accumulating across calls.
6b. For each, build a small state/transition map — list the states and the
legal (and illegal) transitions between them. This is quick prose or a table,
not a formal FSM.
6c. Check the tests against the transitions, not the operations. Read the
corresponding test file and ask: does it only ever drive each operation once,
from a clean initial state? Or does it exercise multi-step sequences that
cross state boundaries? Concretely, look for tests that try:
- out-of-order — a teardown/late operation before the setup that "should"
precede it (e.g. respond to a command after the process died),
- interleaved — two flows in flight at once (overlapping command ids, a new
turn arriving mid-compaction),
- repeated — the same transition twice (compact, then compact again; respawn
after a second crash),
- empty-then-refill — drain to empty, then add again (history cleared then
regrown; registry emptied then repopulated).
6d. Flag and recommend. For every stateful module whose tests only cover
single-operation-from-clean-state, flag it — even if its line/branch coverage
is 100% — and recommend an adversarial transition-matrix test: enumerate the
states on both axes and assert the behavior of each state→state move, including
the illegal ones. Give the concrete example sequences from 6c so the follow-up
is actionable. File these as hs-task/hs-bug tickets.
-
Check the build shape
npm run build && ls dist/
Verify dist/index.js, dist/cli.js, and their .d.ts files exist, and that
dist/cli.js keeps its #!/usr/bin/env node shebang.
