verify-dictation

name	verify-dictation
description	The canonical "prove VZT Flow actually works" checklist — the human hotkey test plus its automated equivalents. Use before claiming a dictation-pipeline change works, before a release, or when the user runs /verify-dictation.

Verify dictation

VZT Flow's core promise is "hold a key, talk, transcript lands wherever your cursor is." This checklist proves that promise end to end, with real audio and real numbers — not "the code compiles so it probably works."

0. Prerequisites

First, confirm which build you are about to verify. Verifying the wrong binary is worse than not verifying — it produces a confident green result about an app nobody is running. This has happened: a "hotkey is broken" investigation ran for an hour against a healthy process before anyone checked that /Applications/VZT Flow.app was an ad-hoc v0.3.2 dev build, two versions behind a crash fix it was still carrying (CLAUDE.md gotcha (j)).

./target/release/flow status | grep version    # the RUNNING daemon's version
grep -m1 '^version' Cargo.toml                 # what this tree is
codesign -dv --verbose=2 "/Applications/VZT Flow.app" 2>&1 | grep -E "flags|TeamIdentifier"

Release = flags=0x10000(runtime) + TeamIdentifier=LKHKU5BW73. Local build = flags=0x2(adhoc) + TeamIdentifier=not set. State which one you tested; a report that doesn't name the build under test is not a verification.

source ~/.cargo/env
cargo build --release --workspace
./target/release/flow doctor

flow doctor must show both models present (Parakeet always; Qwen3 cleanup if you're testing clean/polish). If a model is missing:

./target/release/flow models download parakeet-v3
./target/release/flow models download cleanup   # only if testing clean/polish

1. The human hotkey test (do this yourself, not via script)

1. Launch the desktop app (or relaunch a dev build — see CLAUDE.md gotcha (a): use nohup <dev-binary> &, never open, and never kill the user's running daily-driver instance without relaunching it for them).
2. Hold Right Option, speak a full sentence, release. Confirm: overlay shows Recording → Transcribing (with a mode badge) → Done, and the transcript pastes into whatever's focused.
3. Tap Right Option (press+release under 300ms). Confirm hands-free starts, then stops on its own after ~2.5s of silence (or tap again to stop manually).
4. Hold, start speaking, hit Esc mid-recording. Confirm nothing is transcribed or pasted — the overlay just clears.
5. Focus a password field, dictate. Confirm the overlay shows "Secure field — transcript on clipboard" and nothing gets typed into the field.

This step has no automated equivalent for the actual key event capture and paste — it depends on real Accessibility/Input Monitoring grants and a real frontmost app. Everything below automates what can be automated.

2. Automated equivalents

ASR accuracy + speed (real TTS audio, not silence):

say -o /tmp/flow-verify.aiff "the quick brown fox jumps over the lazy dog near the supabase database"
ffmpeg -y -i /tmp/flow-verify.aiff /tmp/flow-verify.wav
./target/release/flow transcribe /tmp/flow-verify.wav

Check: transcript is accurate, "supabase" comes back correctly cased (tests the dictionary pass), and the RTF line on stderr is in the expected ballpark (~0.1x / ~10x realtime on Apple Silicon — see docs/PRD.md for the measured baseline; flag anything meaningfully worse as a regression).

Cleanup deadline behavior:

./target/release/flow clean-test "um so like I think we should uh go with option two" --mode clean
./target/release/flow clean-test "a normal sentence" --mode clean --timeout-ms 1

The second command forces the deadline miss — confirm it reports the raw fallback path, not an LLM result, and doesn't hang.

Code mode (deterministic, no model):

./target/release/flow code-test "const camel case user profile equals await get user open paren close paren"

Expect exactly: const userProfile = await getUser()

Paste mechanics in isolation:

./target/release/flow paste-test "vzt-flow verify $(date +%s)"

Daemon socket (only if the desktop app is already running — check with flow status first, don't launch an extra instance):

./target/release/flow status
./target/release/flow history -n 5

Exercising the toggle path is fine provided you assert the state comes back — with a settle window, never a single read:

./target/release/flow toggle
./target/release/flow cancel
# poll for a few seconds; sub-second `recording` here is NORMAL
for i in $(seq 1 30); do
  s=$(./target/release/flow status | grep '^state:')
  [ "$s" = "state: idle" ] && { echo "recovered: $s"; break; }
done

The settle window is the whole point. A cancel issued right after a start arrives while CoreAudio is still opening the input device, so state legitimately reads recording for under a second. A previous pass here read that instantly, called it a wedge, and published a bogus 3-in-8 failure rate — a 1s delay before the cancel gives 0/8. An instantaneous read cannot distinguish a transient state from a stuck one.

A state that stays non-idle past the window is a real fault (CLAUDE.md gotcha (k)): through 0.3.3 a stop/cancel reaching an idle audio worker was discarded outright, leaving dictation_state on Recording with the mic live and only an app restart recovering. 0.3.4 closes that hole. If state does not return to idle, restart the app and report a failure rather than moving on.

Long-audio safety (only relevant if touching ASR/chunking — see CLAUDE.md gotcha (b)): never feed >60s of audio straight into flow transcribe without going through the chunked path first; watch ps -o rss on the flow/desktop process while it runs and compare against docs/PRD.md's memory-budget numbers.

3. Overlay visual QA

Use the tray's Test overlay item — cycles Recording→Transcribing→Done with no mic/model involved. This is the reliable way to screenshot overlay states; scripted clicks on the menu-bar extra itself are documented as unreliable on this multi-monitor dev machine.

4. Report format

For each section above: command/action → actual output/observation → pass/fail. Real numbers only (RTF, RSS, timing) — no "should be fine." Anything that can't be verified in the current environment (e.g. no Windows hardware, no physical hotkey access in a headless agent run) must be called out explicitly as unverified, not silently skipped.