qa-critical-ux

name	qa-critical-ux
description	Select and run risk-based desktop QA for a branch, commit, diff, or reported regression. Use the complete auth, CloudSync, calendar, note, recording, speaker, chat, summary, and provider matrix only for an explicit release gate or full-app QA request.

QA: Critical User Experience

Default to the smallest faithful QA scope for the change. The complete checklist remains the release gate, but it is not the default for ordinary branch or regression validation.

Temporary non-blocking audio policy

Before release-gate or full-app QA, check the current Linear status of ANLG-98 and ANLG-222.

• While ANLG-98 is not completed, AEC quality is not a release gate. Do not run dedicated speaker-playback, residual-echo metric, double-talk, or AEC provider-matrix work solely for a release. If an AEC failure is encountered incidentally, record it as informational against ANLG-98; do not patch the release candidate or block publication. Basic microphone/system capture, recording start/stop, audio persistence, and freedom from crashes or hangs remain gates.
• While ANLG-222 is not completed, automatic speaker identification is not a release gate. Do not require automatic Lex Fridman / George Hotz naming or run a provider identity matrix solely for a release. Generic speaker labels are acceptable. Transcript integrity, provider diarization output, manual speaker correction, persistence, and freedom from silent data loss remain gates.
• If a candidate explicitly implements either ticket, use targeted regression mode for that ticket. Otherwise do not investigate, mitigate, or expand QA around the deferred area.
• Restore each gate only after its corresponding Linear issue is completed. A release-specific observation does not override this policy.

Choose the scope first

Before building or launching anything:

1. Inspect the exact candidate and its base. In a GitButler workspace, use but status and but show <commit-or-branch>; do not treat the synthetic workspace HEAD or every applied branch as the candidate.
2. Trace changed files to user-visible behavior and failure boundaries. Find the original reproduction in the current or past Codex task, linked issue, PR, support report, or regression test when available.
3. Write a short risk map with:
   • behavior directly changed;
   • plausible adjacent failures caused by the changed boundary;
   • lifecycle or persistence transitions affected;
   • checklist areas that are unrelated and will be skipped.
4. State the selected live checks before running them. Do not expand the run because a broad checklist is available.

Targeted regression mode (default)

Use this mode for a branch, commit, diff, bug fix, or change isolated to one product area. Run only:

• the closest automated tests required by the affected packages and CI paths;
• the original user reproduction and the expected fixed outcome;
• one bounded adjacent smoke check for each credible cross-cutting failure;
• restart or persistence checks only when startup, durable state, migrations, cleanup, or recovery behavior changed;
• provider, account, platform, or model variants only when the changed code branches on those variants.

A Rust or native-code diff does not justify the full user-experience matrix by itself. It does justify checking that the exact candidate launches, remains responsive, completes the changed native operation without a crash or hang, and emits no relevant panic, deadlock, or repeated-error loop. Test recording, transcription, speaker identity, chat, calendar, auth, and provider matrices only when the diff can affect them.

For example, a retained legacy-migration conflict that changes DB readiness and CloudSync gating should test: native launch and responsiveness; the conflict reproduction; the resulting Storage state and action; CloudSync readiness; retained recovery-file protection; and restart persistence. It should not run an in-meeting recording or provider matrix unless that code is also in the candidate.

If the original fixture or required account is unavailable, mark that specific check BLOCKED. Do not replace missing targeted evidence with unrelated broad testing. Stop when the risk map is covered.

Full release-gate mode

Use the complete setup, release-candidate order, and checklist below only when:

• the user explicitly asks for full-app QA or a release gate;
• the release-new-version skill is about to run; or
• the candidate is broad enough that the risk map genuinely includes most of the critical experience.

Every applicable item must pass or be explicitly waived by the user before a release. A targeted run is evidence for its stated scope, not release approval.

Full release-gate setup

1. Build and launch an authenticated native Dev bundle with AEC diagnostics:

   .agents/skills/qa-critical-ux/scripts/run-native-dev-qa.sh
   

   The script reads the currently deployed public Supabase configuration, derives the production app/API endpoints from desktop_cd.yaml, builds the native app identity needed for Computer Use, and launches with AUDIO_SYNC_PROBE=1 and LISTENER_DEBUG=1. The build runs with an allowlisted environment containing only the public frontend configuration and minimal process/toolchain values. It does not load .env.supabase, desktop .env files through dotenvx, server credentials, or unrelated channel configuration. Local environment files are also excluded from the provenance fingerprint.

   Native Dev release evidence is invalid unless VITE_APP_URL, VITE_API_URL, VITE_SUPABASE_URL, and VITE_SUPABASE_ANON_KEY all match the currently deployed production public values. Local development URLs or a different Supabase project require a rebuild and cannot pass this gate.

   The helper owns a clean, persistent Cargo cache under ~/Library/Caches/anarlog; never point it at the repository's target directory or clone that directory into the QA cache. The repository cache can contain enough old, provenance-tracked proc-macro dylibs for macOS assessment to stall rustc for minutes. Native builds use the full Xcode toolchain explicitly so Swift/MLX can invoke the Metal compiler instead of inheriting Command Line Tools. The helper also builds the workspace UI package before Tauri, so a clean checkout does not depend on generated packages/ui/dist output from an earlier build.

   A successful-build manifest fingerprints the complete app bundle, all desktop build inputs (including legacy crates), and the deployed public auth key. Before the manifest is written, the helper inspects the generated frontend runtimeEnv, rejects any non-allowlisted public variables, and binds that validated configuration fingerprint to the app bundle hash. This prevents --launch-only from running a stale bundle, a locally configured bundle, or a bundle whose production auth config rotated. Reuse an already-current bundle with:

   .agents/skills/qa-critical-ux/scripts/run-native-dev-qa.sh --launch-only
   

   Pin release-gate builds to the intended 40-character candidate commit:

   ANARLOG_QA_GIT_SHA=<candidate-commit-sha> \
     .agents/skills/qa-critical-ux/scripts/run-native-dev-qa.sh
   

Known fixture ground truth

The bundled audio is an excerpt from Lex Fridman Podcast #387 with George Hotz. Use the repo's time-aligned crates/data/src/english_10/pyannote.json as the transcription reference and the official episode transcript as the identity reference. Together they establish this canonical mapping:

Fixture speaker	Participant
SPEAKER_01	Lex Fridman
SPEAKER_00	George Hotz

Validate at least these opening checkpoints after aligning to the first recognized fixture phrase. Fixture timestamps are audio-relative, not session wall-clock timestamps:

Fixture time	Expected participant	Semantic checkpoint
2.925-13.005 s	Lex Fridman	Asks what George thinks about Llama being open sourced
14.145-19.025 s	George Hotz	Says Mark Zuckerberg is the good guy
20.125-29.665 s	Lex Fridman	Asks whether open source is ultimately good
30.365-36.065 s	George Hotz	Answers "Undoubtedly" and begins discussing AI safety people

Use turnLevelTranscription for turn ownership and wordLevelTranscription only when finer alignment is needed. Provider wording and punctuation may differ, so compare the meaning and speaker ownership rather than requiring byte-identical text. The generic fixture IDs are valid only inside the reference file; they are not acceptable participant labels in the app.

Release-candidate order

1. Run the complete Dev checklist from a clean checkout of the exact committed SHA. Do not clean, discard, or include unrelated GitButler workspace changes to produce release evidence; use a separate clean clone instead. Exploratory Dev runs may use modified build inputs, but a release-gate run requires git_dirty=false in the helper manifest. This means the fingerprinted build inputs match the candidate commit. Record git_head_sha, which is the candidate commit rather than GitButler's synthetic workspace HEAD. The manifest is ${ANARLOG_QA_TARGET_DIR:-$HOME/Library/Caches/anarlog/native-dev-qa-target-v2}/.anarlog-native-dev-qa-manifest.

2. Require every Dev gate to pass, then trigger desktop_cd.yaml with channel=staging from a branch or ref whose tip is that exact SHA. Verify the Actions run's head SHA matches the manifest before testing it.

3. Download the artifact from that specific run:

   gh run download <run-id> --name hyprnote-staging-macos-silicon
   

   Do not use a “latest staging” download for release evidence. Record the DMG SHA-256, install it, and repeat the core gates: sign-in, note creation, recording start/stop, transcript preservation, automated summary, and recording/chat CloudSync deferral. Run full-fixture AEC or speaker identity checks only when their temporary policy gate is active.

4. Stable is allowed only when Dev and that exact staging artifact pass for the final main SHA, including its changelog. Verify main still points to that SHA before triggering stable. Any rebuild or source change invalidates the prior evidence.

5. After stable publishes, download the matching architecture DMG from the desktop_v<version> GitHub release, record its SHA-256, install it, and verify the app reports that version. Use Computer Use to repeat the core sign-in, note, recording start/stop, transcript, summary, chat, and CloudSync gates against the installed stable app. Run three-minute fixture, AEC, and automatic speaker identity checks only when their temporary policy gate is active. Keep any fixture playback in the terminal. Do not mark the release complete until this stable pass succeeds.

CloudSync platform scope

The patched native CloudSync vendor bundle and its request-cancellation tests currently cover only macOS Apple Silicon and Intel. This evidence must not be used to approve Windows, Linux, or mobile.

Before enabling or releasing any of those lanes, rebuild every target architecture's bundled CloudSync native library from the patched source and run the equivalent cancellation/drain suite against that exact bundle. At minimum, prove stalled send/receive and manual/legacy network calls, logout, configuration cleanup/init, worker-idle fencing, and an immediate local write after cancellation. Rust-level fail-closed behavior or passing macOS bundle tests alone is not cross-platform evidence.

Checklist

1. Sign in, callback handoff, and sign out

• Start signed out and initiate sign-in from the desktop app.
• PASS when: reaching the browser's signed-in callback page automatically opens the native-protocol prompt without a click; accepting it signs the desktop app in.
• Sign out, repeat the flow, dismiss the automatic prompt, then click the page's Open Anarlog button. PASS when the button opens the same native prompt and completes desktop sign-in without a duplicate or expired-link error.
• Sign out once more with CloudSync enabled. The app must return promptly to its signed-out state without a stuck spinner, SQLite lock, or orphaned sync request.

2. Calendar connect, events, notifications

• Settings → Calendar (or onboarding): connect Apple Calendar and/or Google/Outlook via the integration flow.
• When creating a macOS Calendar test event, set its calendar to the [e-mail] Google account (John (Char) in Calendar), never Personal.
• PASS when: the calendar list renders the account's calendars, events for today/this week appear in the timeline/sidebar, and an upcoming-event notification (meeting-start notification or in-app banner) fires for a test event starting within the notification window.
• Also verify: toggling a calendar off hides its events; ignore/unignore on a timeline event sticks (no snap-back after rapid toggling).

3. Create a new note

• Create a note from the sidebar/new-note affordance.
• PASS when: the editor opens immediately (no blocking wait), typed content persists after switching notes and after app restart, and the note appears in the timeline.

4. Start a recording

• In the note, start listening/recording. Use a short recording to exercise the basic capture path. Play the repo audio fixture from the terminal only when the ANLG-98 or ANLG-222 gate is active.
• PASS when: the recording starts without error, live transcript words appear (when live transcription is enabled for the provider), and the recording indicator/timer runs. Mute/unmute must not wedge the session.
• Also verify both microphone and system-audio inputs carry nonzero signal. While ANLG-98 is open, AEC initialization, speaker leakage, and duplicate playback phrases are informational only. When the issue is completed, require AEC to initialize without an error or fallback and the transcript to follow the podcast once rather than duplicating speaker leakage.
• Provider diarization and automatic identity are separate concerns. When the full fixture is applicable, verify the provider emits exactly two RemoteParty speaker clusters and compare their turns with the Known fixture ground truth above. While ANLG-222 is open, incorrect or generic participant names are informational only and do not fail the diarization check.
• When the ANLG-222 gate is active, PASS speaker identification only when the live or settled transcript labels those clusters Lex Fridman and George Hotz automatically. Generic labels such as Speaker 1, swapped names, a third RemoteParty cluster, or names that appear only after a tester manually assigns them are failures. A transcript that names only the opening turns and later reverts to generic or inconsistent labels also fails. Verify both named assignments and the participant mappings survive app restart unchanged.
• When the ANLG-98 gate is active, treat audio_mic.wav as the post-AEC, post-VAD microphone track, not the raw microphone. For a playback-only run, require all of:
  • audio_mic.wav and audio_spk.wav are readable mono 16 kHz WAVs whose durations differ by less than 0.1 seconds.
  • RemoteParty has at least 400 transcript words.
  • DirectMic words are at most 10% of RemoteParty words.
  • At most 5% of DirectMic bigrams also appear on RemoteParty within one second.
  • Residual mic/speaker absolute correlation is at most 0.10 in every active 30-second sample, with median attenuation of at least 20 dB.
• When the ANLG-98 gate is active, any duplicated-bigram failure blocks release even when the processed microphone is quieter than the system-audio track. If the result is ambiguous, repeat a 90-second baseline with NO_AEC=1; enabling AEC must reduce duplicate bigrams by at least 80% and processed-mic RMS by at least 10 dB.

5. Chat and overlapping activity sync deferral

• With CloudSync enabled and no recording active, send a chat request that runs long enough to inspect status. Once its native lease is acknowledged, require activity_paused: true, deferred_for_capture: false, and a static Saved locally status. Streaming and SQLite persistence must remain immediate.
• No CloudSync send, receive, or E2EE witness request may start while the chat lease is held. Apply the same pre-existing-operation drain exception as the recording gate, then compare last_sync_at_ms and the SQLite Cloud request log until the assistant response and its SQLite writes settle.
• After the final chat persist plus the 750 ms trailing delay, require activity_paused: false and exactly one coalesced trailing sync cycle. Abort/error and regeneration paths must also release their leases.
• Run one overlap: start recording, then start chat, and stagger their completion. Ending the first activity must not resume sync; status remains Saved locally and network sync stays at zero. Only the final lease may resume CloudSync, with exactly one trailing sync cycle afterward.

6. Automated summary after recording

• Stop the recording.
• PASS when: an enhanced note/summary is generated automatically without manual triggering, the summary reflects the spoken content, and a title is generated for untitled notes. A transcript must be attached to the session.
• With CloudSync enabled, verify the final summary-and-title persistence acquires an enhance activity lease before its final database read. No CloudSync request may overlap those writes; release must eventually succeed after transient bridge failures, followed by one trailing sync.
• Capture the summary body hash after generation and again after app restart. The generated title and settled summary hash must remain unchanged.

7. Provider matrix — repeat steps 3–6 under each config

Config	How to set
On-device	Settings → AI: repeat the recording gates with every on-device STT model exposed for the QA Mac, using a local LLM; sign-out state is also worth one pass
Pro plan	Settings → AI: select Anarlog cloud (anarlog provider) with a Pro/trialing account

• PASS when: steps 3–6 behave identically in outcome under each config (transcript + chat + automated summary), with provider-appropriate quality.
• Do not sample one on-device model as representative. Record each exposed model ID, its hardware recommendation status, and its individual result. A model that cannot download, start, transcribe, or settle a recording is a failed on-device matrix row.
• Watch for: feature-gate prompts appearing for entitled users, silent summary failures (check the AI task state), and stalled live transcription (watchdog should batch-repair from the recording after stop).

Automation notes

• Prefer driving the app UI via the Browser/automation tooling available in the session; the Tauri webview is not reachable by the in-app Browser pane, so use screenshots/accessibility tooling or ask the user to perform mic-dependent steps.
• Fixture playback and stop timing must run from the terminal. A human is only needed for the optional double-talk phrase and OAuth consent screens; verify the results programmatically (transcript rows, summary documents, calendar events, and audio diagnostics).
• Useful signals: sessions, transcripts, and session_documents (kind = summary) tables via the app DB; console/log output from the dev server for stall-watchdog and enhance-task errors.
• When the ANLG-222 gate is active, inspect session_participants and the transcript's speaker_hints_json alongside the rendered UI. Count distinct provider_speaker_index values per channel, but use the visible names to gate identity attribution. Do not create user_speaker_assignment hints before recording the automatic-identification result.

Reporting

For targeted mode, report the candidate branch/commit, base, selected risk, PASS/FAIL/BLOCKED, and one-line evidence for each selected check. List unrelated checklist lanes once as NOT APPLICABLE, and say explicitly that the result is not full release certification.

For full release-gate mode, produce a table: checklist item × provider config and on-device model → PASS/FAIL with a one-line note. Include the Dev manifest's Git SHA/dirty state, staging run URL and head SHA, staging artifact SHA-256, stable release URL and artifact SHA-256, app version, speaker-cluster count, speaker-name result, and any explicit waiver. Any FAIL or SHA mismatch blocks release; file or fix before cutting.