How to end-to-end test the SpeakType Windows desktop app (Electron, desktop/ dir) — launch flags, Doubao login workaround, synthetic hotkeys, store locations, known traps.
Instrucciones de origen · Vista previa de solo lectura
name
testing-speaktype-desktop
description
How to end-to-end test the SpeakType Windows desktop app (Electron, desktop/ dir) — launch flags, Doubao login workaround, synthetic hotkeys, store locations, known traps.
Testing SpeakType Windows desktop (Electron)
Round 247 learnings (packaged v0.16.0)
Sherpa worker model-switch bug (may persist): switching local models mid-session (parakeet→sensevoice, whisper→sensevoice) makes the sherpa worker eval fail with SyntaxError: Identifier 'gc' has already been declared; every retry fails until the app process is fully restarted. A COLD start with the model already selected initializes fine. Preflight before any sensevoice-dependent test: check %APPDATA%\SpeakType\logs\main.log for sherpa worker started (...) with NO following sherpa worker error; if the error appears, restart the app rather than clicking Retry.
MoveWindow trap: the SpeakType main window is often maximized; MoveWindow silently no-ops on a maximized window (returns True). Click the restore button first, then resize (see C:\Users\Administrator\tts\resize.ps1, targets Get-Process SpeakType MainWindowHandle).
Theme flip helper that works non-interactively: C:\Users\Administrator\tts\theme.ps1 -v 0|1 (sets both Personalize values + broadcasts ImmersiveColorSet).
When AI polish is enabled with the mock server, ALL dictation output becomes the MOCK-REWRITE: prompt echo — disable AI polish before testing plain dictation/whisper switching, or results look garbled.
Persona app-rule verification: History items show time · <persona name> · duration · Local offline; the rule persona also appears inside the mock-rewrite prompt echo (style instructions section), which is independent corroboration.
This VM may lack the historical C:\Users\Administrator\tts assets; verify rkey.ps1/hf246.wav/sample.wav exist before recording, and note in the report if they had to be re-synthesized.
sample.wav contains ~0.64s leading and ~2.21s trailing INTERNAL silence around the speech. When concatenating it with fixed silence gaps to test pause-duration logic (e.g. hands-free paragraph breaks measured last-voice→first-voice), those internal silences inflate the intended gap by ~2.85s and can flip the verdict. Trim the PCM to first/last sample with |peak|≥1000 before concatenating (see C:\Users\Administrator\tts\r354\gen354.mjs).
Also note the fake mic LOOPS the WAV: add a long tail silence and exit hands-free during the tail, or the loop restart appears as another long-gap sentence.
NEVER enable Windows Firewall on this box (netsh advfirewall set allprofiles state on): it severs the Devin control channel instantly; recovery needed a VM reboot. The firewall is intentionally OFF, so program-scoped firewall block rules are silently ineffective too.
hosts-file blocking may silently stop working (entries present, ANSI encoding fine, flushdns/Dnscache OK, but ping still resolves real IPs). Always verify with ping <domain> that blocking is effective BEFORE relying on it — a "blocked" download that succeeds means the block never applied.
Null routes (route add <ip> mask 255.255.255.255 <bogus-gw>) do NOT block here (virtual NAT proxy-ARPs everything). Proxy env vars (HTTPS_PROXY) are ignored by Node/undici fetch in the Electron main process.
Simplest reliable block for packaged builds: patch app.asar — npx asar extract resources/app.asar tmp, string-replace the three source hosts in out/main/index.js (https://huggingface.co/, https://hf-mirror.com/, the GH models-v1 release base) with https://127.0.0.1:1/…, npx asar pack back (back up app.asar first). Fetch to 127.0.0.1:1 fails instantly (no 10s/source timeout like IPsec) and exercises the genuine network-error path; asar integrity is not enforced (app boots fine, verified 0.15.1/Electron 43). Restore the original asar afterwards.
Re-confirmed 2026-08-17: HTTPS_PROXY=http://127.0.0.1:1 is ignored by Node fetch in Electron main (status 200 through "fake proxy"); hosts blocking remains dead on this box — always ping to verify before trusting any block.
Partial-progress display: since #130 (d383cbe) any COMPLETE file on disk counts as progress, but since #261 (652f34e) percent <1% returns null — tokens.txt-only (no .part) shows plain again; a real .part with meta still shows . A resumed download's final file can be verified by sha256 == c71f0ce0… (the HF LFS oid in download.ts GH_ASSET_SHA256).
This is separate from the Chrome extension skill (testing-speaktype). Do NOT test the desktop app via the extension procedure.
Memory-soak diagnostics on the packaged build (round 276)
The packaged SpeakType.exe accepts --inspect=9229: http://127.0.0.1:9229/json exposes the main-process Node inspector ws. Over that ws, Runtime.evaluate needs includeCommandLineAPI: true or require is undefined and every eval silently returns undefined.
Reusable scripts in C:\Users\Administrator\tts\r366\: memsample.ps1 (per-role WS/PM csv every 60s, classifies main/renderer/gpu/network by CommandLine), heappoll.cjs (process.memoryUsage + v8.getHeapStatistics csv via inspector; needs node >=22 for global WebSocket — use C:\hostedtoolcache\node\24.0.1\x64\node.exe), snap.cjs (v8.writeHeapSnapshot to a file; the write itself bloats RSS/PM by ~60-70 MB, so take the final snapshot AFTER the last csv sample you intend to use, and exclude snapshot minutes from slope fits).
TRAP: with an inspector client attached, tray Quit does NOT complete — main + gpu + utility processes linger until the debug client disconnects; kill the poller and the app exits cleanly within seconds. Don't misreport this as a quit hang.
Slope methodology: least-squares over the csv, computed separately for first-30 and last-30 minutes; the first ~30 min contain allocator warm-up (~0.2 MB/min) that converges to ~0. Endpoint-to-endpoint deltas over short windows (like round 275's 37 min) overstate the slope.
Baseline from this round (0.17.0 @ 6a0f1f0, sensevoice hands-free ~6 utterances/min): main heapUsed flat 16-18 MB, arrayBuffers 0, external oscillating 5-10 MB, PM plateau ≈495 MB, last-30-min PM slope ≈0.
Segment-count is the strongest single signal for Silero segmentation fixes on the 2.2s-gap fixture (silero256d.wav, 32 short sentences): broken hangover → ~12 heavily-merged segments; fixed → ~41; peak baseline → ~37 with all heads intact. Count segments from history.json before eyeballing text.
Two runs in one app session can share one history.json: record the pre-run count N and slice history.slice(0, len-N) for the delta — avoids restarting the app between A and B when only the Enhanced VAD toggle changes (toggle applies live, no restart needed).
Freshly launched SpeakType steals foreground focus; ALWAYS screenshot-verify Notepad focus after every app (re)launch before Ctrl+A/Delete/Alt+Q. A run started with SpeakType focused pastes nothing ("No text field in focus" toast) but still writes history — history-based counting stays valid, on-screen text does not.
Residual Silero anomaly signature to look for even after fixes: pairs of 需要换行处理。 (head lost) followed by segments ending …我/我们 (that head glued to previous tail) — count them as pairs, not independent errors.
Round 257 learnings (ASR language + Silero model preflight)
settings.language MUST be zh for Chinese fixture tests. The user's real config has language: "en"; with en, SenseVoice transcribes Chinese TTS as pinyin garbage ("H Huang Hang.", "Lin Qi.") and command words never match — this silently invalidates whole VAD/command runs. Preflight: node -e "...settings.language" and set zh in test userData before any Chinese run.
Silero model preflight: fresh test userData needs %APPDATA%\SpeakType\vad\silero_vad.onnx (a copy lives at C:\Users\Administrator\silero_vad.onnx). The Enhanced VAD toggle turns green even when the model is missing and main.log logs NOTHING about silero — the only visible confirmation is the Settings hint text "Add-on ready — voice detection upgraded". Behavioral tell: with the model, 2.2s-gap fixtures produce merged 10-18s captures; without it, capture cuts at every gap like peak mode.
rkey.ps1 maps only rctrl/etc — it does NOT know lalt/q; Alt+Q must be sent with the computer-use key action (works as a global hotkey regardless of focus).
Alt+Q during an in-flight utterance still finalizes and pastes that utterance once (by design, like hold release); "no residual" means nothing further lands afterwards.
Looping a short (<2s) command WAV with no silence gap never finalizes (continuous speech); command fixtures need ≥2.5s trailing silence inside the file.
Clicking the Notepad taskbar icon sometimes raises SpeakType instead — always screenshot-verify focus before Ctrl+A/Delete (a stray Ctrl+A+Del into SpeakType is harmless but wastes a cycle).
Frameless-window drag region swallows clicks (general trap)
The main window's top strip (~y<45 when maximized) is a -webkit-app-region: drag zone: clicks on any in-app button/link that scrolls up into that strip do nothing (or activate a background window if the click lands outside the app). Symptom: repeated clicks on a visibly rendered button with no effect. Fix: scroll the page so the target sits below y≈100, then click. (Same root cause as the History Show all/Show less note below.) Note: builds after PR #352 give <main> a mt-10 margin so content can no longer scroll under the strip — this trap only applies to pre-#352 builds.
Launch (dev and installed)
Dev: cd desktop && npx electron out/main/index.js <flags> (build with npm run build first).
Required flags in this environment:
--no-proxy-server --use-fake-device-for-media-stream --use-fake-ui-for-media-stream --use-file-for-fake-audio-capture=C:\Users\Administrator\tts\sample.wav
plus --remote-debugging-port=9333 when you need CDP.
WHY --no-proxy-server: Electron follows Windows system proxy (registry points at socks 127.0.0.1:1080 which is NOT listening here) → doubao.com fails with ERR_PROXY_CONNECTION_FAILED. Chrome on this box is launched without proxy so it works; Electron needs the explicit override.
Fake mic works exactly like Chrome; the wav loops, sentence: 「帮我跟老板说那个方案需要再改一下明天上午之前给他答复」.
Crafting fake-mic WAVs (PR #44 lessons)
System.Speech WAVs contain a LIST chunk after fmt — the data chunk is NOT at offset 44. Naively rewriting sizes at offsets 4/40 yields a file Chrome's fake capture plays as pure silence (dictation logs maxPeak=0 voicedMs=0). Parse chunks, extract the data PCM, and rebuild a canonical 44-byte-header wav (see C:\Users\Administrator\tts\makehf2.ps1).
Continuous hands-free (Alt+Q) can be tested deterministically: fake-capture loops the file, so a wav of speech + ~4s trailing silence makes every loop one auto-finalized sentence (silence > vadSilenceMs=2000 triggers finalize, hands-free auto-restarts). A pure-silence wav drives the 6×10s no-voice rounds → auto-exit toast in ~62s.
Verify toasts objectively with CDP polling of the toast.html target (C:\Users\Administrator\tts\toastpoll44.cjs, needs --remote-debugging-port).
Doubao login inside Electron (independent cookie jar)
No doubao credentials on the box. Workaround: copy Chrome's doubao.com cookies into the Electron session via CDP (C:\Users\Administrator\tts\cookie_xfer.js: reads Chrome's cookies on port 29229/2513, writes via Electron's 9333 using Storage.setCookies). Reload the bridge window after injecting.
Then the real activation flow works: home 去激活 → doubao window → click its built-in voice input once (fake mic) → WS hook captures api_app_key into the store (doubaoAppKeyCache).
Store / userData locations
Dev: %APPDATA%\Electron\speaktype.json.
Installed/packaged since 0.8.x: %APPDATA%\SpeakType\speaktype.json (older builds used %APPDATA%\SpeakType 语音输入法\ — the app auto-migrates from it on first launch, so when testing a "fresh config" you must rename both dirs away, otherwise the legacy config is migrated back in and the test is invalid: look for migrated legacy userData from … in main.log).
Offline model lives in %APPDATA%\SpeakType\models\sensevoice-small\{model.int8.onnx (239MB), tokens.txt}; keep a copy outside the config dir so a fresh-config run can restore it instead of re-downloading. downloadLocalModel skips files that already exist, so copying them in before clicking 下载 makes the card finish instantly (still logs local model … downloaded).
To simulate "not activated": quit app, set doubaoAppKeyCache to "", relaunch → hold RightCtrl shows Chinese toast 「还没拿到豆包语音入口…」. Back up the file first.
Offline (SenseVoice) live captions
Offline live captions are approximated by re-decoding the buffered audio every ~1s (asr.ts PARTIAL_* consts): preview starts after 1s of audio and stops once the buffer exceeds 20s; the final result is still decoded on release. To prove it, zoom on the floating panel at ~8s and ~16s (text must grow) and again at ~25s (frozen/shorter than the final pasted text) during a >27s hold.
Decoding is synchronous in the main process; during long re-decodes the whole UI (and CDP) can stall briefly. If the window stays white and all CDP targets time out, the main process is wedged — kill and relaunch (seen once on 0.8.3).
Auto-learn corrections (0.8.5 watchedit.ts)
Trigger conditions: autoLearn && autoPaste && !failed && text contains Chinese (dictation.ts finalize). Watcher = hidden powershell.exe -NonInteractive child of SpeakType polling the UIA focused element every 700ms for 15s (deadline starts inside PS, so add ~2-3s startup slack).
Notepad classic Edit works via the Name-property fallback (full text incl. Chinese; verified with a standalone probe replaying the same UIA script).
To make a correction that actually learns: the changed segment after common-prefix/suffix trimming must be a 2-6 char pure-Chinese word on the "right" side. Replacing 方案→草案 does NOT learn (shared 案 → diff becomes 方→草, 1 char). Use replacements sharing no boundary chars (答复→回执). Make the edit within ~3-13s after paste (before 15s deadline, after PS's first sample).
Learning evidence: main.log auto-learn: "X" -> "Y" (mojibake in console is normal — read speaktype.json as UTF-8/base64), settings.hotwords appended, history text replaced. Since 6eaa42a the renderer refreshes immediately after a learn (learnCorrection calls pushSettings); Dictionary/History should show the new state without touching settings.
When counting watcher processes, exclude your own shell: filter Win32_Process powershell.exe by ParentProcessId ∈ SpeakType PIDs, and never put match-strings like 'FocusedElement' in your own command line (it self-matches).
Cross-window false positive (found in 0.8.5 first cut): typing Chinese in ANOTHER focused control during the 15s window could be mislearned. Fixed in 6eaa42a — the observer binds to the baseline control's UIA RuntimeId; re-verified that typing in SpeakType's search box after paste no longer learns.
Incremental learning (0.9.1, 6f03041): each ~1.5s edit pause settles a round (toast per word), baseline rolls, LCS splits one big edit into multiple words. Testing traps: (a) the observation window is only WATCH_SECONDS=20 from the last text change — it does NOT extend while the user is merely idle, so edits starting >~20s after paste are silently missed (found as a real gap vs the "25s still learns" claim); batch click+select+type into ONE computer-use call, tool round-trips of 3-4s easily blow the window; (b) word pairs must share NO boundary chars or prefix/suffix trimming shrinks the diff below the 2-char threshold (明天→后天 shares 天 → only 明→后 → not learned, by design); (c) the History page "Clear all" button next to the search box wipes ALL history with no confirm — don't click it to clear the search filter.
App-based persona rules
Personas page → 「按应用自动切人设」→ 添加规则 → match text (matches process name OR window title, case-insensitive) + persona dropdown. The rule is evaluated at record start from the foreground window, and with no LLM configured it only shows up as the personaName on the history item (History list shows HH:MM · <persona> · <secs> · Local offline).
Always run the counter-test (foreground = a non-matching window, e.g. the SpeakType window itself) — it must record the global persona instead, otherwise a hardcoded persona would look like a pass.
Synthetic hotkeys (uiohook sees SendInput)
Use C:\Users\Administrator\tts\rkey.ps1 -Seq "down:rctrl,sleep:8000,up:rctrl" (scancode SendInput). Works for RightCtrl hold, Alt+Space, Alt+digits (d1..d9), Alt+Q (q key added for PR #16), f6, esc (added for 0.8.6 hotkey-capture tests). If a key is missing from the script's $map, the script silently no-ops — add the VK/scancode pair before concluding a hotkey is broken.
Hotkey capture (0.8.6 「录一个键」): the 10s capture window is easily missed if you press the key via a fresh exec shell (PowerShell startup can exceed 10s) — the button silently reverts and it looks like capture is broken. Press within the window via the computer-use key action (F6/Escape work) or a pre-warmed shell. Middle mouse: computer-use middle_click on any blank area is seen by uiohook. Auto-repeat bug found in 509ba94 (holding the OLD hotkey during capture triggered a real dictation via key-repeat keydowns) was fixed in c025093 (captureSwallowKeycode swallows that key until its keyup) — re-verified: hold 1.8s captures cleanly with no finalize, and normal dictation works right after.
Persona toast lasts 2.6s — screenshot immediately, or verify via store settings.personaId change.
Home page stats/activation card do NOT live-refresh in all cases; stats refresh on app restart.
Known traps
Alt+Space is the Windows system-menu key of the focused app (e.g. Notepad) — a paste issued while that menu modal is open is lost. Default toggle is Alt+Q since PR #15; if testing Alt+Space anyway, note AutoGLM (AutoGLM.exe, auto-running on this box) ALSO swallows it globally — kill it (taskkill /f /im AutoGLM.exe) first.
Fixed in PR #15 (don't re-report): tray icon missing from the installed package; error panel not auto-hiding (now hides after 5s); home activation card not refreshing after key capture.
Several stale orange tray icons from killed electron.exe dev runs linger in the tray overflow; hover to clear them.
This VM has NO audio output device (Get-CimInstance Win32_SoundDevice empty, taskbar speaker shows red X) — the "mute other apps while recording" switch (VK_VOLUME_MUTE) has no observable effect here; mark it untested rather than inferring.
System locale resolves to English, so with uiLanguage: "system" (fresh store) the app starts in English — useful baseline for i18n tests (since PR #16).
Dev runs show the Electron version (e.g. 43.3.0) as the app version in sidebar/About (app.getVersion()); only the packaged app shows the real product version.
Computer-use type action into the app sometimes drops shifted/capital letters (e.g. "SpeakType" → "peakype") — an input-injection artifact, not an app bug; verify chip text before reporting.
Launching rkey.ps1 via Start-Process -WindowStyle Hidden can steal the foreground window: the release-Ctrl+V then pastes into the hidden PowerShell instead of the target (looks like "paste failed" while transcription/history are fine). Workaround: re-click the target window during the hold, or diagnose with C:\Users\Administrator\tts\fg_watch.ps1.
Rewrite-selection (0.8.7): C:\Users\Administrator\tts\mock_chat.mjs is a local OpenAI-compatible chat-completions mock (http://127.0.0.1:8975/v1, logs each prompt, returns fixed "[REWRITTEN-BY-MOCK]" text) — use it for the rewrite E2E when no real LLM key works. secret:personal:DEEPSEEK_API_KEY was Insufficient Balance (HTTP 402) as of 2026-08-10; verify with the settings-page Test connection button before relying on it. The computer-use type action drops : in URLs (https// bug) — paste base URLs from clipboard instead. Rewrite toasts last ~2.6s: time the screenshot ~1s after F8 release (fail path) or during the hold (no-model/no-selection paths).
C:\Users\Administrator\tts\mock_whisper.mjs now matches the exact path /v1/audio/transcriptions (404 otherwise) — usable for both the success path and the HTTP-404 error branch of the ASR test-connection button. It can also return arbitrary text from (useful for long/punctuation test cases).
Window-state (maximize/hidden) verification
Judge maximized objectively with user32 IsZoomed(MainWindowHandle) + GetWindowRect (maximized rect = -8,-8,workarea+8 on this box), not by the title-bar glyph — at this scaled resolution the restore glyph is ambiguous in screenshots.
--hidden only takes effect when settings.startMinimized=true (index.ts); hidden state is provable via Get-Process SpeakType all MainWindowHandle=0 plus a desktop screenshot.
launchAtLogin writes HKCU Run value name SpeakType pointing at the launched exe with --hidden; clean up with reg delete HKCU\Software\Microsoft\Windows\CurrentVersion\Run /v SpeakType /f after tests, or the packaged test build will autostart on the next VM boot.
AutoGLM on this box can spontaneously pop a 智谱 login window into the foreground mid-test (not only swallow Alt+Space) — kill AutoGLM.exe before recording window-visibility tests.
Remote mic relay (relay/ Cloudflare Worker) local testing
npx wrangler dev in relay/ needs Node >= 22 (VM default 20.19 fails). Prepend C:\hostedtoolcache\node\24.0.1\x64 to PATH first. Serves http://localhost:8787.
The desktop app connects with wss:// only. Local wrangler has no TLS: run the TLS-terminating proxy C:\Users\Administrator\tts\wss_proxy.mjs (8443 -> 8787, reuses the app's self-signed cert) and launch Electron with NODE_TLS_REJECT_UNAUTHORIZED=0; point the relay URL setting at https://127.0.0.1:8443.
C:\Users\Administrator\tts\relay_pipe_test.mjs objectively verifies Worker pass-through (binary fidelity, start/stop/status/peer, room-occupied close 1008) against ws://localhost:8787 without the desktop app.
CDP Runtime.evaluate with top-level const fails on re-run — wrap injected snippets in an IIFE.
Real phone mousedown on the Chrome phone page steals foreground; use CDP Input.dispatchMouseEvent for hold/release so pasted text lands in the target window.
A real relay deployment exists at https://speaktype-relay.wookat520.workers.dev — use it directly for production-path relay tests (no local wrangler/TLS proxy needed). Do NOT pass --ignore-certificate-errors to the fake-phone Chrome when targeting workers.dev, so the real TLS chain is verified too.
PWA /app pairing (0.9.0+): the fixed pair code == settings.remoteRelayRoom (12 hex chars, generated once when Internet relay is first enabled; shown under the QR in Settings→General→Audio→Phone as microphone). Phone page stores it in localStorage['speaktype-room']; refresh must skip the pairing UI and auto-reconnect. cdp_phone_hold090.js (variant of cdp_phone_hold.js) also matches /app targets — note after installing the PWA there are TWO /app CDP targets (tab + standalone window); the script grabs the first, so close the tab if you need the standalone one. beforeinstallprompt DOES fire in Chrome-for-Testing here — clicking 添加到主屏幕 opens the native Install dialog and the installed standalone window works end-to-end. Console/SW checks: C:\Users\Administrator\tts\cdp_pwa_check.js (port 9444) prints SW registration, 4 asset statuses and console errors.
0.9.2 (2324483) autolearn window: watcher now renews the deadline while the FOCUSED control's UIA runtime id equals the anchor id (watchedit.ts L54-58), WATCH_SECONDS=45 / MAX=300 - idle 25-40s after paste then editing DOES learn (verified). The old '20s from last text change' trap only applies to <=0.9.1. History 'Clear all' now needs an inline confirm (Clear all history? -> red Clear / Cancel); Cancel preserves entries - safe to click once, but the red Clear still wipes everything.
Deleting %APPDATA%\SpeakType\speaktype.json is NOT enough to simulate a brand-new user: src/main/migrate.ts copies the config back from any legacy dir matching SpeakType * that contains a speaktype.json (e.g. SpeakType ?????). Renaming with a suffix (SpeakType xxx-off) still matches the glob - move the legacy dir OUT of %APPDATA% entirely, and move it back (original name) during cleanup.
Verify via main.log: expect "no legacy userData to migrate" instead of "migrated legacy userData from ...".
To make the Home missing-model banner appear for the default model, also stash the model files out of %APPDATA%\SpeakType\models\<model> and restore afterwards.
Capturing loader-phase UI states (badge/spinner)
SenseVoice offline transcription finishes in <1s, so the post-release loader is nearly impossible to screenshot. To hold the loader open, run a local OpenAI-compatible polish server that delays ~6s before responding (node http server on 127.0.0.1, returning {choices:[{message:{content:...}}]}), set polishBaseUrl/ApiKey/Model in speaktype.json, then screenshot during the "Polishing..." spinner. Restore polish settings afterwards.
Fake mic and audio service traps (round 30)
The fake microphone is NOT a system device: launch SpeakType.exe with Chromium flags --use-fake-device-for-media-stream --use-fake-ui-for-media-stream --use-file-for-fake-audio-capture=C:\Users\Administrator\tts\sample.wav. Without these flags hotkeys appear dead: recording aborts instantly with a brief "Microphone unavailable / No microphone found" toast (bottom-center, ~2.6s, easy to miss).
Windows Audio services (Audiosrv/AudioEndpointBuilder) may be DISABLED on this VM; the Chromium fake-capture flags work regardless. Re-enable with sc.exe config Audiosrv start= auto etc. only if real audio devices are needed.
To debug the packaged main process, launch with --inspect=9229 and use Runtime.evaluate over the Node inspector WebSocket; require app modules via path.join(process.resourcesPath, 'app.asar', 'node_modules', ...).
To prove a localCleanup/polish text transform really fired, screenshot the live-caption capsule mid-hold: it shows the raw ASR partial BEFORE cleanup (e.g. costs$11), then compare with the final pasted text (costs $11). Same build, before/after distinguishable.
Custom English fake-mic WAVs can be generated offline with SAPI TTS: SpFileStream + SpAudioFormat.Type=22 (16kHz mono) writes a wav directly usable as --use-file-for-fake-audio-capture source.
When launched with fake-mic flags, the SpeakType main window steals foreground: before dictating into Notepad, click its taskbar icon and confirm focus via the Ln/Col status bar.
Stats-accounting changes (round 48, PR #103): back up %APPDATA%\SpeakType\history.json, zero the stats block while the app is closed, then dictate a sentence whose word count differs wildly from its char count (e.g. 5 words / 28 chars) and cross-check history.json numbers against the Home UI. This box's SAPI has English voices only (David/Zira) — mixed CJK/Latin test sentences cannot be TTS-generated naturally.
Export dialogs and encoding traps (round 50, PR #106)
Renderer Blob downloads (Dictionary/History export) open a native Save dialog in Electron: click Save in the GUI first, then verify the file byte-for-byte via shell. The file does not exist until Save is clicked.
PowerShell 5 reads UTF-8-without-BOM files as ANSI by default and can corrupt Chinese text when writing back. Always use Get-Content -Raw -Encoding UTF8 / .NET UTF-8 APIs when touching speaktype.json or history.json, and back them up before any script edits.
Two-step confirm buttons and persona storage (round 51, PR #107)
Two-step confirm buttons change label width on first click, which shifts the button center: leave >=0.3s between clicks and aim the second click at the widened label. Zero-interval synthetic double clicks may be swallowed.
Personas/rules/current selection live in speaktype.json: top-level personas[] (custom ids like custom-), settings.personaId, settings.appPersonas. Cascade-cleanup assertions are most reliable via direct shell inspection of that file.
Prompt capture via local mock endpoint (round 52, PR #109)
To assert exact polish/rewrite prompt content, run a local Node HTTP mock at 127.0.0.1:18099 that logs the POST JSON body and returns a fixed OpenAI-compatible completion, then point polishBaseUrl at it.
Remember polishEnabled=true routes ordinary dictation through the mock too: disable polish when testing local-only hotword correction, or the fixed mock output overwrites the text you want to inspect.
Use full local model ids (e.g. parakeet-tdt-0.6b-v3, not parakeet); a wrong id silently shows the download banner instead of erroring.
Remote mic (phone-as-mic) testing (round 54, PR #110)
Open the pairing page in a browser launched with --no-proxy-server --ignore-certificate-errors plus the three fake-mic flags; the LAN page uses a self-signed cert and the box has a dead system proxy.
LAN/relay mode switching works with the pairing page still connected (since PR #112 stopRemoteMic force-closes lingering connections); verify with netstat | findstr 43117 plus the remote mic listening/relaying via lines in main.log.
Single-box "text lands in Notepad" checks race pasteText (Ctrl+V to the foreground window ~60ms after mouseup): use a SendInput script that Alt+Tabs within ~25ms of mouseup, or the text lands in the browser.
computer-tool coordinates are 1024x768 scaled; the real display is 1280x720 — multiply x by 1.25 and y by 0.9375 when driving SendInput scripts from screenshots.
Use a real cmd.exe window as the paste target (activeApp reports cmd.exe, which is in TERMINAL_APPS); clicking inside a cmd window enters "select" mark mode that swallows the paste — press Esc before dictating.
SAPI-generated short-sentence wav files looped as fake mic input are sometimes misrecognized on the first loop (e.g. "Speak type" → "speight"); assert on the second loop's stable output or lengthen the hold.
Hotkey loss-rate sampling is only valid when the target window has been ready >=3s before injecting (a keypress injected ~2s after Notepad launch can be swallowed by focus churn — environment flake, not a product bug; 20/20 clean under the steady-state rule).
launchAtLogin assertion: reg query HKCU\Software\Microsoft\Windows\CurrentVersion\Run /v SpeakType (value includes --hidden). startMinimized only hides the main window when launched WITH --hidden; without it the window shows as usual.
Repeated Stop-Process kills accumulate ghost tray icons; sweep the tray overflow with the mouse to clear ghosts before double-clicking the real icon to restore a hidden main window. Prefer tray Quit for cleanup.
Caption-overlay visual checks (mask/line-height/scroll): zoom the region around [340,555,690,645] (bottom-center) and keep same-utterance zooms of the old build for before/after comparison. The "not overflowing → no mask" state must be captured ~3-4s after entering hands-free (partial still <= configured lines; overflows by ~15s). For 6-line captures avoid the #120 50s segmentation moment (caption restarts and shrinks) — sample at 30-45s or 25s+ into the second segment.
Audio render endpoint: this box has no physical audio device, but a VB-CABLE virtual sound card is now installed (PR #217 round) so CoreAudio GetMute/SetMute work. If GetMute ever throws 0x80070490 again, reinstall: driver pack at C:\Users\Administrator\tts\vbcable\, run pnputil /add-driver vbMmeCable64_win10.inf /install then C:\Users\Administrator\tts\scream\Install\helpers\devcon-x64.exe install vbMmeCable64_win10.inf VBAudioVACWDM and Restart-Service audiosrv. (Scream 4.0 does NOT install — its .cat certificate is expired.)
Mute probes: C:\Users\Administrator\tts\getmute.ps1 and setmute.ps1 <0|1>. Both were fixed in the PR #217 round: the old getmute.ps1 had 12 vtable filler methods (off-by-one — it actually called GetVolumeStepInfo and always returned True); the correct IAudioEndpointVolume layout is 11 fillers before SetMute(slot14)/GetMute(slot15). If you copy this pattern, verify with SetMute(1)/(0) → GetMute round-trips first. muted-by-recording crash-recovery flag lives at %APPDATA%\SpeakType\muted-by-recording.
Probing mute DURING a hold-key recording: computer-tool hold_key blocks, so start a background PowerShell that sleeps ~3-4s then probes/kills (tts\pr217_probe.ps1, tts\pr217_kill.ps1), then immediately start the hold. keepFailedAudio pruning only fires when a finish-stage recognition failure persists audio — needs fault injection (temporarily corrupt a model file), not network blocking.
Model fault injection, hotkeys, migrations (rounds 64-66)
To create a finish-stage recognition failure, corrupt a whisper ggml model (whisper-server is a separate process: exited(3) → failed entry lands + prune runs, main app survives). Since PR #125, corrupting a sherpa model (sensevoice/parakeet) no longer crashes the app — a wrong-size file is treated as not-downloaded (guard proof: process alive 30s+ AND no sherpa worker started in log; pre-#125 builds silently abort). For the parakeet channel inject joiner.int8.onnx (only 6MB, fastest to back up/restore).
Before any model injection record Get-FileHash SHA256 and rename the original aside; restore and re-verify the hash afterwards. Pre-seeding failed-audio for prune tests: node-generated valid RIFF wavs + fs.utimesSync to control mtime (prune sorts by mtime; drives the 20-count/7-day/50MB rules precisely).
The speech model status/download button lives in Settings→Speech (the AI model tab is polish settings). HF downloads go through a local proxy cache here — 234MB completes in seconds, so re-download tests can just wait for completion and compare SHA against the original.
Hands-free hotkey tests always use Alt+Q: Alt+Space was removed from the product in PR #126 (Windows system-menu key; synthetic injection never triggered it either). Verifying read-time migrations needs three-way proof: UI shows the new value + the file on disk keeps the old value (node-read while the app runs) + the new value actually works (injection test).
Persona hotkey injection uses rkey.ps1's d1/d2/d3 key names (down:alt,down:d2,...); persona switching writes settings.personaId, so restoring the backed-up config is enough for cleanup.
Config-corruption recovery toast ("Settings rebuilt", with .bad backup) appears ~ready+1.5s and lasts 4s — screenshot ~2s after Start-Process. BOM-only corruption is repaired silently in place with zero data loss (no toast, no .bad). For window-bounds persistence compare GetWindowRect physical pixels, not the DIP values stored in speaktype.json (DPI differs).
To exercise resume-copy/progress logic without a real interrupted download, fake a .part: write the first N bytes of the real file to dest.part plus dest.part.json containing {"url":"fake","total":<real total bytes>} — partialProgress only reads the .part size and meta.total (download.ts), so the UI shows N/total ("Resume download (43% done)"). Delete both and restore originals afterwards.
"Fresh no-model" state: rename the whole model directory to .bakNNN (rename back after). Since PR #128, a corrupt single file (wrong size) makes the button show "Resume download (99% done)" instead of the full-download copy; the banner body text still says "One-time download (~660MB)".
Missing vs corrupt are different modelPartialPercent branches — inject both separately (delete a whole file vs write a wrong-size file). Since PR #130 both show Resume with distinct percentages (delete 652MB encoder → 2%, corrupt 6MB joiner → 99%), a strong assertion pair. Download failure state: inject with icacls %APPDATA%\SpeakType\models /deny "Administrator:(OI)(CI)W" (cleanup /remove:d, verify no DENY left). The "Downloading x% + progress bar" window needs a big file (652MB encoder ≈ 40s; a 6MB file finishes too fast to capture).
UI-language dropdown via synthetic clicks tends to miss (opens the adjacent Theme dropdown or misses the item). Standard method: click to open the menu → screenshot to confirm item coordinates → separate single call to click the target item → screenshot to confirm it took effect.
Polish/rewrite mock, history injection, toast capture (round 70)
Test AI polish / F8 rewrite with a local OpenAI-compatible mock: C:\Users\Administrator\tts\mock-polish-8098.cjs (127.0.0.1:8098, fixed response + logs each request); set polishBaseUrl=http://127.0.0.1:8098/v1 in settings. F8 must be HELD ≥400ms (holdDelayMs=120 threshold — an instant tap does not trigger).
Failed-entry Retry can be injected synthetically: set a history entry status="failed" + error + audioFile pointing at a real wav copied under failed-audio; Retry re-recognizes and normalizes the entry in place.
500-entry full-load: rewrite the history array in history.json directly (top-level shape {history,stats} — leave stats alone).
rkey.ps1's key map lacks ctrl/a/letter keys (type:/ctrl+a throw NullArray) — use computer-native injection for text and Ctrl+A. Short-lived toasts (persona switch, rewriteNoModel) require screenshotting immediately after computer-native key injection; the rkey→screenshot pipeline is too slow.
If the local browser can't reach the internet, the cause is usually the dead system proxy (socks 127.0.0.1:1080, nothing listening) — launch msedge --no-proxy-server; do NOT change the system proxy.
Before relying on hosts-file blocking (e.g. simulating GitHub API rate-limit silence), verify with ping: this box's hosts ACL may lack NETWORK SERVICE/Users read access, making the DNS Client ignore hosts entirely. Fix temporarily with icacls hosts /grant "NT AUTHORITY\NETWORK SERVICE:R" "BUILTIN\Users:R" + Restart-Service Dnscache; afterwards /remove:g and delete the entries. Resolve-DnsName always bypasses hosts — verify with ping.
Hands-free soft-segment (50s) evidence: main.log dictation finalize: durationMs=5xxxx (>50000 means the soft-segment path); a 60s session on the looping fake mic triggers it.
Remote-mic LAN pairing can be tested without a phone: open the pairing URL in msedge --no-proxy-server on this box. Token rotation (re-open the desktop pairing dialog) invalidates the old page: it must stop after exactly 8 reconnect attempts and show the rescan prompt; a wrong token gets HTTP 403.
20-min hands-free soak evidence: compare main.log dictation finalize count vs History new-entry count (must match; zero-loss proof). Memory sampling in a background loop: when cleaning up with Stop-Process filtered by command line, exclude your own PID — a naive filter matched the sampler itself and killed it.
Memory sampler MUST NOT use Start-Job from a one-shot exec shell — the job dies when the parent shell exits and leaves no CSV. Use a detached hidden process instead: Start-Process powershell -WindowStyle Hidden -ArgumentList '-Command', $samplerLoopWritingToCsv. At minimum capture a manual first/last WorkingSet snapshot as fallback evidence.
After cold start, Alt+Q within the first ~5-8s may be ignored (sherpa worker not ready yet) — re-send after waiting; not a product bug.
Launching win-unpacked WITHOUT the four fake-mic flags shows "Microphone unavailable — No microphone found" and no finalize in logs; easily misread as broken hotkeys. Always pass the flags.
Comparing history.json text in Windows PowerShell requires gc -Raw -Encoding UTF8 (default ANSI reads UTF-8 as mojibake, giving a false mismatch).
polishEnabled=true routes ALL dictation through the polish endpoint — when a mock endpoint is attached, keep it off except for F8 cases. F8 rewrite only checks polishBaseUrl (dictation.ts startRewrite); the toggle does not affect it.
Since PR #134 "no paste target" = foreground window class Progman/WorkerW (or no hwnd). To trigger: click empty desktop and dictate, expect the "saved to History" toast and no Ctrl+V. SpeakType's own windows (Chrome_WidgetWin_1) count as valid targets.
Test the pasteText 350ms clipboard-restore race with C:\Users\Administrator\tts\clip-race.ps1 (launch with -STA, hidden window): it tight-polls the clipboard, and after detecting the dictated text it MUST wait ~200ms before writing the simulated user copy — writing immediately can land before Ctrl+V and pollute the paste itself (a sentinel false failure). Only assert on runs whose log timestamps show the write happened before the restore point (write + 60 + 350ms).
When killing the sentinel, filter with 'clip[-]race\.ps1' AND exclude $PID — a plain command-line filter matches the killer itself (same self-kill trap as round 72).
Since PR #136 the restore is conditional (clipboard.readText() === text): user copy during the window survives; with no contention the old clipboard is restored; with an empty previous clipboard the dictated text stays.
Test image-clipboard restore with C:\Users\Administrator\tts\imgclip.ps1 (must run -STA): set = 64x64 green bitmap with a red corner block; verify = ContainsImage + size + two pixel probes. Assert on pixel values, not just ContainsImage=True — otherwise you can't distinguish "restored a different image".
Sentinel polling logs may show rapid [] ↔ text flapping right after a write (WinForms GetText jitter); take the final value from a separate -STA verify process, not the polling log.
Since PR #138: image previous is snapshotted only when text previous is empty; restore is three-branch — clipboard rewritten by others → give up; text previous → writeText; non-empty image → writeImage. File lists and other formats are still not preserved (declared boundary).
history.json top-level shape is {history:[...],stats:...}; latest entry text is $h.history[0].text.
Model punctuation only runs with polishEnabled=false and enhancedPunct=true (polish.ts — AI polish skips the punct model entirely).
Model lives at %APPDATA%\SpeakType\models\punct-ct\model.onnx (~281MB, downloaded via Settings→Speech; HF direct takes ~1 min).
Log signature for model punctuation is punct worker started; rule-based fallback has no such line.
Settings-page status updates live via the onPunctStatus push channel: the punctuate failure path pushes downloaded:false (#140) so external model deletion reverts the UI without restart; the success path does NOT push — verifying "back to ready" requires remounting the page (switch tabs).
Long Chinese test wavs: local TTS voices are English-only. Build them by PCM-concatenating C:\Users\Administrator\tts\sample.wav — read the bytes, take [byte[]]$data = $b[44..($b.Length-1)] (must cast to byte[], otherwise BinaryWriter writes a tiny corrupt file), repeat, and hand-write a 44-byte RIFF header.
TXT/SRT export goes through queued system "Save As" dialogs: save each one fully before clicking the next — a queued dialog intercepts later clicks/typing and can overwrite your source wav. Right after transcription completes, the Cancel button's position becomes the SRT button; screenshot to confirm the progress row is still there before clicking.
Warm sensevoice/parakeet transcribe ~70s audio in only 3-5s — cancel test cases need ≥5min audio to have a window.
ggml-tiny-q5_1.bin is already downloaded on this box (models root); to exercise the "model missing" banner pick base-q5_1 or small-q5_1.
Settings tab ids are general/voice/model/about: "voice" is Speech recognition (local dictation models), "model" is the AI polish model tab (#144 fixed the banner jump to use "voice").
TXT vs copied-text comparison: Windows clipboard converts to CRLF; normalize line endings before asserting equality.
Verifying completion-instant UI (e.g. the #146 settling window): do one warm-up run first to measure the exact transcribe duration (log started/done delta), then time the action as "press Enter → wait duration+0.1s → act immediately → screenshot now AND re-screenshot after 5s". Queued Save-As dialogs can pop several seconds late, so a "no dialog appeared" assertion must include the delayed re-screenshot.
ffmpeg 8.1.2 (gyan essentials) is installed on this box and on PATH. Container-matrix fixtures are one-liners: mp3 -b:a 64k, m4a -c:a aac, ogg/webm -c:a libopus, mp4 -f lavfi color=black + -shortest -c:v libx264 -c:a aac. Multi-hour wavs generate in seconds with -stream_loop N -i src.wav -t <sec> -c copy (3h ≈ 330MB — delete after).
Release acceptance (round 80): Setup is a per-user NSIS installer → %LOCALAPPDATA%\Programs\SpeakType, wizard buttons sit bottom-right (~601,543), and an overwrite install shows "Will reinstall/upgrade". On the finish page UNCHECK "Run SpeakType" before Finish, then launch manually with the fake-mic flags. Silent uninstall: "Uninstall SpeakType.exe" /S (user data in %APPDATA%\SpeakType survives uninstall). The update banner comes from main/index.ts fetching the GitHub latest release ~5s after startup (api.github.com direct) — an amber "New version vX is available" bar on About, hidden when versions match. Download release assets with direct and byte-compare against the GitHub API asset size.
Este SKILL.md es muy grande, por eso SkillsMP muestra aqui solo la primera seccion.Ver en GitHub
Download model
Resume download (x% done)
C:\Users\Administrator\tts\mock_text.txt
Clicking buttons on the floating panel: always mouse_move to the target first, then click — a direct click sometimes doesn't land in the panel window at all. To verify a click actually hit, inject a CDP click probe into panel.html and check for events.
In hands-free/hold modes, VAD auto-ends ~2s after sample.wav's speech finishes (history items ~10s). When testing cancel/long-hold behaviors, separate the auto-stop timing from your own action, or a working cancel will look broken.
PR #39 (8c7166a) round: this RDP VM reports matchMedia('(hover: hover)')=false, and Tailwind v4 wraps every hover:/group-hover: rule in @media (hover: hover) — so hover-reveal UI (History card Copy/Correct/Delete) can NEVER appear visually here even with real cursor moves. Verify hover styles via keyboard (focus-within, Tab from the search box) plus a CDP selector check (el.matches('.group-hover\\:opacity-100:is(:where(.group):hover *)') while dispatching Input.dispatchMouseEvent moves); don't report hover as broken. Hotkey capture takes the FIRST keydown: modifiers/Space are valid single keys, so pressing a combo (Ctrl+Alt+Space) just captures LeftCtrl — trigger the amber "unsupported" warning with a digit/arrow key instead. Generate English test wavs locally with System.Speech (16k mono SpeechAudioFormatInfo); wavs loop during long holds, so a "short English" utterance becomes long if held >wav length — keep the hold under the wav duration. History Retry button only renders on FAILED entries with kept audio (keepFailedAudio default true); the easiest failure is an OpenAI-compatible provider pointed at http://127.0.0.1:9/v1. whisper tiny-q5_1 (~32MB) downloads in seconds via the Local model card and switches workers with no restart. Window bounds persist only via the window 'close' event (click X to tray) — taskkill /F skips persisting.
Dev-mode launch (no packaged build needed): cd desktop; npx electron-vite dev (set NO_PROXY=*). Fake-mic Chromium flags cannot be passed this way - use a separate Chrome (--use-fake-device/-ui/-file flags) as the phone-mic source via the relay instead when the desktop app itself doesn't need a mic. PR #37+: official relay = https://speaktype.zalize.com/relay (relayBase = host+path); the exact old URL https://speaktype-relay.wookat520.workers.dev is auto-migrated back to the new default on every getSettings() read, so to test the old endpoint enter it scheme-less (speaktype-relay.wookat520.workers.dev) which bypasses the exact-match migration. Both domains hit the same Worker, so a phone on the old domain can pair with a desktop on the new one (same room). Relay URL edits only take effect after toggling Connection LAN->Internet relay. When editing speaktype.json from PowerShell use BOM-less UTF8 ([IO.File]::WriteAllText) - Set-Content writes a BOM that crashes electron-store JSON.parse.
Round-3 (c181c63) testing: startup toasts only display ~2.6s (about 2.0-4.5s after launch) — pixel screenshots almost always miss them; instead poll the toast window's document.visibilityState via CDP (visible -> hidden timeline + DOM text is objective proof the toast fired). To verify polish.ts English punctuation rules deterministically, do NOT import the whole file (pulls in electron-store and crashes under plain node) — extract addEnglishPunctuation into a standalone harness and run it with npx tsx; reusable harness at C:\Users\Administrator\tts\polish_harness.ts.
Release acceptance (v0.13.0 round): the installed app is a plain Electron binary, so the fake-mic Chromium flags work on it too — launch %LOCALAPPDATA%\Programs\SpeakType\SpeakType.exe --use-fake-device-for-media-stream --use-fake-ui-for-media-stream --use-file-for-fake-audio-capture=<wav> (no flags = dictation silently produces nothing on this mic-less box; change utterance = restart with a new wav). NSIS Setup supports silent install/uninstall via /S (installs to %LOCALAPPDATA%\Programs\SpeakType, uninstaller Uninstall SpeakType.exe). The About upgrade banner depends on an api.github.com latest-release prefetch that fails SILENTLY on anonymous rate-limit 403 (no banner on first launch) — just restart the app and check main.log for latest release prefetched: vX.Y.Z before concluding the banner is broken.
Round-5 lessons: (a) hold/rewrite hotkey taps shorter than holdDelayMs (default 120ms) are swallowed by hotkey.ts anti-misfire and trigger NOTHING (incl. exiting hands-free) — always hold synthetic keys >=0.2s before concluding a hotkey path is broken; (b) the CDP toast-poll process binds to a specific target and silently goes stale across app restarts — restart the poller together with the app or you get false 'no toast' results.
Round-6 lessons: (a) the CDP toast poller script (toastpoll*.cjs) hard-exits after ~90s — before asserting "toast did not appear", confirm the poller process is still alive (or restart it), otherwise you get false negatives (nearly misreported a P1); (b) for fresh-userData tests, renaming %APPDATA%\SpeakType alone is NOT enough: if the legacy dir %APPDATA%\SpeakType 语音输入法 exists, startup auto-migrates it into the new profile (main.log shows "migrated legacy userData") — move both dirs aside.
PR #51 (VAD/ORT) lesson: to prove the app no longer loads a legacy native binding from a user-writable dir (e.g. the old %APPDATA%\SpeakType\vad\onnxruntime_binding.node), plant FAKE placeholder files with those exact names in the dir and restart — the app must stay healthy and log no load errors; byte-identical legacy files are not needed for the "present but not loaded" claim.
PR #52/#53 (ITN / double-tap) lessons: (a) SenseVoice raw output has BUILT-IN ITN — it emits Arabic numerals for almost everything (二十三岁→"23岁", 幺三八零零→"13800", 三点半→"3点半" with an Arabic hour), so app-layer text-transform rules must be E2E-tested with sentences SenseVoice won't pre-convert (approximate numbers like 四五十个, idioms like 千万别) using an on/off toggle DIFF of the pasted text; also probe the rule module directly (esbuild the .ts to cjs and run cases under node). (b) Synthetic double-tap that beats anti-misfire: rkey.ps1 with ~70ms press duration + ~200ms gap reliably lands inside holdDelayMs=120 / DOUBLE_TAP_MS=400 windows; a single 70ms tap must do nothing (anti-misfire regression check).
PR #55 (round-8 fixes) lessons: (a) combo-vs-double-tap testing: rkey.ps1's $map now includes c (vk=0x43/scan=0x2E); simulate "shortcut usage" as Ctrl held <120ms with another key tapped during the hold — this must NOT enter hands-free while a pure 2×70ms double-tap must. (b) SenseVoice itself mis-converts ambiguous number sentences at the ASR layer (两千五百分之五十 → raw "2550%"), so app-layer ITN ambiguity protection can only be verified by probing the rule module directly (esbuild itn.ts to cjs); E2E never sees the Chinese raw for such sentences.
PR #57 (unified download / silent-partial filter) lessons: (a) in long hands-free silence runs, a manual exit hotkey RACES the ~1-minute 6-silent-round auto-exit — pressing "exit" after auto-exit RE-ENTERS hands-free and looks like "can't exit"; assert exit within 1 minute of entering, or double-prove via main.log finalize durationMs<10s + capsule gone. (b) Download failure paths: block sources via hosts entries 127.0.0.1 huggingface.co / hf-mirror.com / cdn-lfs*.hf.co (hosts writes occasionally hit file locks — retry), assert UI shows the error AND the target dir has no .part leftovers; restore hosts by marker lines afterwards (script: C:\Users\Administrator\tts\hostsblock57.ps1).
PR #59 (Parakeet / multi-model sherpa) lesson: when verifying a new sherpa-family model, use main.log sherpa worker started (<modelId>) to prove which model the worker loaded (switching models must log a NEW line), and drive it with a fake-mic WAV in the model's language (English: b2.wav); transducer engines ignore the Recognition language setting, so no language-dropdown change is needed to verify.
PR #66 (site screenshots) lesson: to capture clean app screenshots for the website use CDP Page.captureScreenshot (script C:\Users\Administrator\tts\cdp_shot.cjs, with wincap.ps1 MoveWindow to set window size first) — this machine's screen is only 1280x720 and taskbar auto-hide doesn't work, so full-window GDI capture always includes the taskbar. Also: PowerShell variable names are case-insensitive ($h clobbers a $H parameter).
PR #64 (dark root-cause) lesson: verifying theme/palette fixes must NOT rely on screenshot impressions — dark cards can mask a still-light body background. Assert numerically with CDP getComputedStyle(document.body).backgroundColor (helper: C:\Users\Administrator\tts\cdp_eval.cjs against port 9222, index.html page). In dark mode also open every native select's popup: the popup is system-white and option text may inherit light theme variables → low contrast.
PR #61 (dark mode) lesson: to test "follow system" theme sync, flip the registry directly — HKCU:\Software\Microsoft\Windows\CurrentVersion\Themes\Personalize values AppsUseLightTheme/SystemUsesLightTheme (0=dark/1=light); Electron matchMedia reacts live, no restart or Windows Settings UI needed. Restore both to 1 afterwards.
PR #49/#50 (enhanced punctuation) lessons: (a) SenseVoice CHINESE raw is essentially always self-punctuated (tried normal and +45% rate edge-tts long run-on sentences) so polish.ts needsPunctuation never fires for zh with SenseVoice - the ct-transformer model path can only be shown E2E with ENGLISH raw (e.g. b2.wav, raw has no mid-sentence punctuation); check main.log for "punct worker started" to prove the model (not rules) ran. (b) No local Chinese TTS voice exists (only David/Zira en-US, and python has no pip); generate Chinese WAVs online with npm i msedge-tts + node script (zh-CN-XiaoxiaoNeural, mp3) -> ffmpeg -ar 16000 -ac 1 -> rebuild canonical wav (see C:\Users\Administrator\tts\edgetts49.mjs / buildzh49.ps1); mkdir the toFile output dir first. (c) The punct model download from HF is fast (~5s) and real progress % is visible - click and screenshot within ~1.5s to catch mid-progress. (d) PowerShell here runs .ps1 files as ANSI: any script containing Chinese must be re-saved as UTF-8 WITH BOM before running or it fails to parse.
iwr
Transcribe persistence (#151+): finished results land in %APPDATA%\SpeakType\transcribe-last.json ({segments,fileName}; only written on completion with segments>0 — cancel/failure never write). To verify restart-restore, use two fixtures with different segment counts (e.g. 40s→2 segs vs 19s→1 seg) so you can tell which run was restored. Cleanup must delete transcribe-last.json.
Renderer history list is App.tsx-level cached state: it refreshes on dictation status→idle and (since #154) on a pushed transcribe completion state. For other main-process writes to history.json, trigger one dictation or restart before asserting UI. When asserting clipboard equality for multiline text in PowerShell, captured stdout splits lines and creates false diffs — use [Windows.Forms.Clipboard]::GetText() → WriteAllText to a file, then compare with CRLF normalized.
Warm sensevoice file transcription runs at ~30x realtime (300s of audio finishes in ~10s), so any "cancel mid-transcribe" test needs a fixture of ≥1200s (20min, ~38MB wav: ffmpeg -stream_loop 120 -t 1200 -c copy). If the Cancel click lands after the log's done timestamp, that round is invalid — retest instead of reporting a legitimate completion as a cancel leak.
Testing history search / personaName filters: use fixture filenames whose mixed-case token only exists in personaName (e.g. EchoDelta40.wav) as an adversarial key. Dictation entries record the persona actually applied by app rules (Notepad dictation often logs as "Auto translate" instead of the persona shown on Home). For computer-use hold-to-talk, hold Control_R — "ctrl" is left Ctrl and won't trigger dictation.
Timed Undo/toast UIs (e.g. #158's 6s delete-undo bar): put trigger→screenshot→click-undo in a single computer-tool action batch — separate calls usually exceed the window and you click nothing. Verify expired rounds against on-disk data and redo them as invalid instead of reporting the timing as a product bug. Such tests really delete data: back up history.json before and restore the whole file after.
Editing UTF-8 (no BOM) data files with Chinese text (e.g. history.json): use [IO.File]::ReadAllText/WriteAllText with UTF8Encoding($false) — PowerShell 5 Get-Content/Out-File default encodings mojibake the Chinese. The delete-Undo pill (#158/#160) sits bottom-right bottom-6 right-6; the Undo button is ~ (876,672) at 1024x768 default window — clicks elsewhere in the pill do nothing. Testing "Show more" pagination needs >50 history entries; back up then inject synthetic items with decreasing at.
DOM/ARIA/focus forensics on the packed build: launch win-unpacked with --remote-debugging-port=9223, then node + desktop\node_modules\ws against the page target from http://127.0.0.1:9223/json doing Runtime.evaluate (returnByValue) — CDP for evidence only, all UI input stays native keyboard/mouse. For focus-transfer assertions, don't query activeElement after the fact (cross-tool-call latency usually exceeds the 10s Undo window): pre-inject a capturing focusin listener logging {ts,tag,text}, run the keyboard sequence in one batch, then read the log for a millisecond-level focus trail. The Transcribe drop zone sits after the GitHub/MIT footer links in Tab order; its focus ring is an orange outline.
Since #164 the Audio section (mic picker/Test/phone-as-mic) lives at the top of Settings→Speech, not General; the Home phone-mic entry jumps via voice#remote-mic (settings/index.tsx "tab#anchor" syntax; MicSection scrollIntoView + brief indigo ring on anchor hit — capture the ring in the same action batch as the entry click). The "AI model" tab is renamed AI polish / AI 润色.
Long-text/wrapping UI needs two adversarial fixtures: a >12k-char CJK entry (verifies the History edit textarea rows clamp at 14) and an ≥800-char no-space ASCII string (verifies break-words — ordinary CJK sentences wrap anyway and prove nothing about overflow). Assert overflow via CDP el.scrollWidth > el.clientWidth plus a screenshot of the card's right edge. Prefix injected synthetic entry ids uniformly (e.g. r90-*) so zero-residue can be confirmed programmatically after restore.
Cloud-ASR network-error paths: dead-port method (asrProvider="openai" + asrBaseUrl="http://127.0.0.1:43998/v1", confirm the port isn't listening) requires a dummy asrApiKey — an empty key takes the "Speech recognition not set up" config branch and never issues a request. speaktype.json settings keys live under the top-level settings object ($j.settings.asrProvider); writing $j.asrProvider fails silently. A successful Retry converts the failed entry in place to a normal text entry (error field cleared) — usable as the recovery assertion. History Show all/Show less buttons can scroll up near the title bar; scroll them below y>100 before clicking.
Release-acceptance gotchas: the About update banner uses the unauthenticated GitHub API (60 req/h/IP) and this box is often rate-limited — check api.github.com/rate_limit core.remaining/reset first; after reset, switch General→About to remount and refetch. Downloads gets cleaned between rounds — re-download old Setups from releases/download/v{ver}/ instead of assuming they exist. Edge follows the dead system proxy (ERR_PROXY): launch site screenshots with msedge --no-proxy-server <url>. Most reliable Undo-pill forensics: delete → immediately mouse_move to hover the pill (pauses the 10s timer) → screenshot → click Undo — survives cross-tool-call latency.
Main-process pure functions (e.g. the polish.ts punctuation pipeline) can be verified against the shipped code: grep the asar's out/main/index.js for the new literals, then extract the real function via indexOf + vm.runInContext and run discriminating cases — far more trustworthy than re-typing the regexes. The preload global is window.speaktype (not window.api) for CDP Runtime.evaluate IPC calls. To block the GitHub API for cache tests add 127.0.0.1 api.github.com to hosts — but NEVER edit hosts with a same-file Get-Content | Set-Content pipe (Set-Content truncates first, wiping hosts); use [IO.File]::ReadAllLines then WriteAllLines, and ipconfig /flushdns after. Isolate "this launch" log assertions by recording the log file byte offset before starting.
Tray testing: the notification-area overflow accumulates ghost SpeakType icons from previously-exited test runs (Windows caches them; sweeping the mouse over them clears them) — expand the ^ overflow, sweep to clear ghosts, then right-click the single live icon (live icon shows the Open / Set up speech recognition / Quit menu). In the computer tool the hold hotkey must be Control_R ("ctrl" is left Ctrl and won't trigger). Since #174 the tray "Set up speech recognition" item opens Settings→Speech; the Doubao bridge window only opens from the Doubao provider card's button.
Two-step-confirm UIs: the discriminating assertion is that data is untouched after the first click (old behavior applied immediately). Order: first click → screenshot the red confirm state + re-check the count in speaktype.json unchanged → wait past the reset timeout and screenshot the reverted button → second click → json count zero. The 4s reset window is long enough across tool calls; no same-batch tricks needed. Dictionary hotwords inject under settings.hotwords (string array); empty-state copy is "No hotwords yet". Dictionary Export lands in Downloads as speaktype-dictionary-YYYY-MM-DD.txt; verify CJK content by byte comparison (the PowerShell console renders it as ???).
failed-audio retention testing: prune only looks at .wav extension + mtime + size (no decoding), so build discriminating fixtures from zero-byte-filled files with (Get-Item).LastWriteTime backdating; the trigger is any failure save with keepFailedAudio=true (one dead-port dictation suffices). To discriminate "50MB counts kept items only", include legit small files OLDER than the over-limit big file — old logic cascade-deletes them, new logic keeps them. Stats live in history.json top-level stats.{sessions,words,durationMs} (not speaktype.json); the Home guide card folds at sessions≥10. Keep fixture count <20 so the 20-item rule doesn't interfere with byte-rule assertions.
After an app restart the SpeakType main window often steals foreground focus: before fake-mic dictation, click the Notepad taskbar icon then its text area (or verify the foreground window after AppActivate); if the text lands nowhere, assert the round's ASR/correction result from the top entry of history.json and redo one visible dictation. ASCII hotword nearKey tolerance (#182): only keys ≥6 chars get fuzz, insertions are rejected (plurals stay), exact match wins over fuzzy — reuse the five discriminating cases DevOs / javescript / speaktypes / gitlab / "devop s". English TTS fixtures: edge-tts to mp3 then ffmpeg to 16k wav; speaking "dev ops" / "java script" as split words reliably exercises the fuzzy path.
F8 rewrite is a hold-to-talk hotkey (same holdDelay as RightCtrl): with the computer tool you must hold F8 for 5-6s while dictating the instruction — a tap does nothing. Flow: select text → hold F8 and speak → release replaces the selection. Hung-endpoint mock: a Node http server that receives the request but never calls res.end (since #190, AbortSignal.timeout(30s) fires the catch branch); print the request-arrival time in the mock log and compare to screenshot timestamps to assert the timeout. Toasts only show ~3-5s, so bracket the 30s mark with wait 26s + 3-4 rapid screenshots at 1.5s intervals, proving behavior first via state/log/inserted text. Polish config keys: settings.polishEnabled/polishBaseUrl/polishModel/polishApiKey; Test connection is in Settings→AI polish.
Multi-case LLM-reply testing: use a file-driven mock — the mock server re-reads a reply file (UTF-8 no BOM) on every request, so switching cases is just rewriting the file (no mock/app restart). stripLlmWrapper (since #192) discriminating five-pack: "Here is the polished text: X"→X, 「以下是润色后的文本：X」→X, no-prefix stays verbatim, quoted-only strips quotes, "Here is the text:" (strips to empty) falls back to the original reply — verify one case each on both call sites (rewriteSelection via F8 and polishText via dictation).
Auto-learn (watchedit) testing: the observation window is 45s and settles immediately on BLUR — clicking the SpeakType main window is the fastest settle trigger; the "New word learned + Undo" toast appears ~1-2s after blur, bracket with 0.6-0.8s rapid screenshots. Pure insertions (empty wrong) and edits after the window expires do NOT learn — use same-length homophone replacements as the discriminating edit. Single-CJK-char homophone edits learn a boundary-expanded 2-char entry (方案→方按 learns 「按需」 not 「方按」) — don't assert the dictionary content equals the hand-edited word; the correction still lands correctly at dictation time. TTS fixtures get deleted by prior rounds' cleanup: dir C:\Users\Administrator\tts\*.wav first (english_long.wav/sample.wav are the stable residents).
Testing shipped pure functions from app.asar: the internal path uses backslashes (out\main\index.js via @electron/asar extractFile); slice the function by matching function name( with bracket pairing, prepend the consts it references (CJK/ASCII_WORD etc.), and inject real deps from node_modules (e.g. pinyin-pro) — enough to run correctHotwords/extractCorrections discriminating cases in a Node vm without launching the app.
Since #196, single-CJK-char boundary expansion only fires when exactly one side is CJK: edits sandwiched between CJK on both sides (园→圆 inside 公_散) do NOT learn — that's the discriminating positive case; sentence-start/punctuation-adjacent single-char edits still learn. Design discriminating cases in a vm run of shipped extractCorrections first, then confirm on UI. This test machine has NO audio output device (SAPI Speak throws 0x2, MediaPlayer HasTimeSpan never true): measure.ps1 dictation/F8 metric segments cannot produce data here — only parsing/startup/Uri paths are verifiable. Taskbar icon order shifts with process launch order: screenshot before clicking the Notepad icon, and AppActivate('Untitled - Notepad') fails once the title carries a * dirty marker.
Verifying CJK literals inside app.asar: do NOT pass CJK strings on a PowerShell command line (console encoding mangles them to ????); write a temp Node .mjs that does readFileSync(asar).toString("utf8").indexOf("日本語/中文 literal") for byte-level checks. The UI-language dropdown lives at the bottom of Settings→App behavior and needs two clicks (first focuses, second opens the options). Since #188 the Home guide card's expand/collapse button uses its own keys home.steps.expand/home.steps.collapse (no longer history.expand).
Config write-failure injection: attrib +R %APPDATA%\SpeakType\speaktype.json makes electron-store's atomic write (rename tmp -> target) throw EPERM; ALWAYS attrib -R and restore the backup afterwards. Since #186, a failed persist leaves the in-memory store unchanged, so UI edits snap back to the old value immediately — you cannot switch uiLanguage through the UI while read-only; remove +R, persist the language, then re-apply +R to capture localized toasts. To assert only "this launch" log lines, record main.log's byte length beforehand and read from that offset with a ReadWrite-shared FileStream (the running process holds the file open).
Opus-family (ogg/webm) decode adds ~2s of padding, shifting segment cut points vs the wav baseline — timestamps stay monotonic and text is correct; not a defect. A 2h59m wav (~328MB) takes 5-8s to decode; the Loader phase is not a hang.
Fake-mic wavs loop when the hold outlasts the file: the wrapped-around leading syllable pollutes the tail of the dictation and shifts watchedit word-boundary decisions — for auto-learn cases always use a padded wav (original audio + 8s trailing silence). suggestHotword (History Correct chips) has NO single-char CJK expansion: a 1-char edit (末→墨) yields no chip by design — test Chinese suggestions with full 2-char word edits. Before each round Get-Process notepad and kill leftovers (a stale Notepad steals paste focus). Auto-learn assertions: dictionary chips screenshot + settings.hotwords json + auto-learn log line count is the reliable trio; the short-lived toast is bonus evidence only. Regression retests are strongest when replaying the exact corpus/steps of the failing round. The computer tool key action rejects space-joined sequences ("Right Right" = unknown key) — issue one key per action; for in-sentence caret edits prefer pure keyboard (Ctrl+Home + N×Right + Shift+Right + type) over coordinate clicks, CJK fullwidth punctuation counts as 1 char.
Hands-free reopens capture per segment and a fake wav replays from the file head each time: multi-sentence flows cannot be built from one composite wav — assert segmentation via finalize interval/duration in main.log (old behavior one 50s chunk vs per-sentence ~10s segments). Long-silence cases: loop a pure-silence wav; each round is VAD_NO_VOICE_TIMEOUT_MS=10s and HANDS_FREE_MAX_SILENT_ROUNDS=6 ≈ 1 minute — count maxPeak=0 log lines for rounds; the auto-exit toast appears ~56-62s, burst-capture from 55s. handsFreeEnd/ByKey toasts show for only ~1-2s and the manual-exit one is nearly impossible to catch — assert behavior via overlay gone + finalize stopped; the toast is bonus evidence.
Hotkey conflict/disable discrimination: an 80ms tap (below the default 120ms holdDelayMs) exercises only the toggle branch without triggering hold — assert with the trio of no hands-free UI, unchanged target text, and no new finalize line in main.log. F10 activates the app menu bar on Windows (Notepad File highlight): press Escape to leave menu mode after any F10 hotkey test. Legacy configs may still carry the retired Alt+Space as hotkeyToggle; the UI falls back to displaying Alt+Q, so assert against speaktype.json, not the dropdown.
Config injection must write nested keys j.settings.* — top-level keys in speaktype.json are ignored (electron-store reads store.get("settings")); the symptom of top-level writes is F8 popping "polish model not configured" and jumping to Settings. For in-memory state-leak cases needing both silence and speech in ONE app instance (e.g. the #198 rewriteTarget leak), build a "N seconds silence + speech" composite wav for --use-file-for-fake-audio-capture: every recording session replays from the file head, so a short hold captures only silence and a long hold reaches the speech tail. ASR error-branch mock: switch Settings→Speech to OpenAI-compatible with BaseUrl at the mock and return 500 from /audio/transcriptions — triggers the finalize catch branch (error state + failed history entry). Dictionary Clear is two-click confirm and the layout shifts before the second click — never reuse pre-click coordinates.- "First hotkey after cold start" cases must each be: kill -> cold-launch the packed app -> wait for sherpa ready (~9s) -> the FIRST injected key is the target combo; any prior keyboard input destroys the discriminating premise (uiohook alt-state syncs after the first event). Repeat 3+ cold starts for stability.
Hands-free silent-round counting (since #207): vadAutoStop=true caps at 6 rounds (~61s), false caps at 30 rounds (~5min08s) �?" count maxPeak=0 log lines within the session time window. The exit toast window is only ~2s; burst-capture at 2-3s intervals starting 8s before the expected exit, and treat log round-count + overlay-gone as the primary assertion.
The computer tool caps a single wait at <215s; segment long waits with the wait tool instead.
Tray-menu cases: the SpeakType icon often lands in the Windows notification-area overflow (click ^ to expand), and prior rounds' Stop-Process kills leave ghost tray icons �?" sweep the mouse across the overflow panel to clear ghosts before right-clicking the live icon; pair tray-menu assertions with a zoomed screenshot.
Typing long URLs into settings text fields via the computer tool occasionally drops characters: zoom-verify immediately after typing and ctrl+a retype if wrong.
"Immediate refresh" tray discriminations (e.g. #209 label change on settings:update) must happen within one process: fill key -> blur -> right-click tray directly; any restart in between destroys the discrimination.
Release-acceptance hands-free runs (handsfree.wav loops): ALWAYS check settings.language BEFORE the run — under language=en the zh loops show alternating per-loop dropped chars (板/天/前), which perfectly mimics the #338 "dropped words on gaps" regression symptom (v0.16.0 round nearly misreported it). Switch Recognition language to 中文 via Settings→Speech and rerun; clean = 4 identical full sentences.
zh dictation test cases MUST explicitly set settings.language="zh" first: with recognition language=English, Chinese speech degrades silently (dropped chars like "tian/qu") on the mic-loopback path while the same audio transcribes near-perfectly via file Transcribe �?" the rounds-124-128 "mp3/wav jitter" theories were misattributions from unisolated variables. en cases stay correct under language=zh in the same run.
The test script's SoundPlayer stops playback when its process exits: subtitle/partial assertions must keep the player process alive through the screenshot (or use -PlaySync), otherwise you capture a silent session and misjudge the overlay as broken.
Error-state hotkey retry cases: the retry press must be >=120ms (below holdDelay it never enters start; the #203 short-tap non-trigger trick is a counterexample here) and within the 15s error-state linger after failure (RETRY_WINDOW is 60s but the state expires first). Retry rounds show voicedMs=100 in the log (MIN_VOICED_MS backfill) vs fresh recordings (~thousands), a clean discriminator.
failed-audio wav filename = history entry id and its creation time = first-failure moment �?" the strongest physical evidence for "in-place update vs new entry" assertions.
Standard ASR-failure construction: settings.asrProvider="openai" + asrBaseUrl="http://127.0.0.1:18099/v1" (dead port) + dummy key; failure toast reads "Cannot reach the speech recognition service".
DPI/resolution-change cases are untestable on this box: the virtual display (1280x720 single screen) enumerates zero modes via EnumDisplaySettings. Test the same protection chain via injection instead: write mainWindowBounds x=5000,y=5000 to simulate an unplugged monitor (offscreen guard should recenter), and use MoveWindow + process kill (not close) to prove the 800ms debounced bounds persistence survives crashes.
Home stats cards (Sessions/Words/Voice time) read the stats key inside history.json; the renderer only pulls them at init and on idle status broadcasts. For "stats refresh" changes, assert with a Home screenshot + shell read of h.stats (double evidence), and restore stats together with history.json.bak during cleanup.
History failed-entry layout: red error line with a purple Retry chip on the right; on successful Retry the entry upgrades in place (timestamp suffix flips Cloud API -> Local offline).
Get-Clipboard with Chinese text needs [Console]::OutputEncoding=UTF8 first, otherwise ?? output causes false negatives.
History Correct edit box: textarea autoFocus, Shift+Enter inserts newline, Esc cancels and discards (since #219); expand/collapse button copy is "Show all / Show less".
Inline PowerShell with $var passed through the exec tool gets eaten by the outer shell; write any assertion involving variables to a .ps1 file first and execute the file (pr219_check.ps1 / pr219_clip.ps1 pattern).
History search filters on three fields (text + raw + personaName); when asserting hit counts or Export line counts, tally each field via shell first (Export count = filtered count, not text-only hits). Export uses the Electron save dialog, defaults to Downloads\speaktype-history-YYYY-MM-DD.md - delete it during cleanup.
Literal Chinese regex inside a .ps1 without BOM is read as ANSI mojibake under PowerShell 5; build the pattern from code points instead (e.g. [char]0x660E).
sample.wav contains a LIST chunk (data starts at offset 70); when concatenating or appending silence tails you must parse chunks and patch the data length - a naive 44-byte-header assumption yields corrupt silence (symptom: finalize logs maxPeak=0 voicedMs=0).
