Menu
Run comprehensive QA testing on the Transcripted application — build, unit tests, deep code audit, artifact validation, log analysis, and fix loop.
Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.
Based on SOC occupation classification
Watch and answer questions about a Transcripted meeting while it is being recorded from the local live sidecar workspace, then switch to the final saved meeting Markdown when it appears.
Search Transcripted meeting and dictation history for what was said, when it happened, who mentioned it, where an idea came from, or how a topic changed over time. Use for memory, recall, provenance, timeline, source-finding, and "what did I say about" questions.
Create cited summaries, recaps, briefs, daily reviews, weekly reviews, and "what did I do" answers from Transcripted meetings and dictations. Use when the user asks to summarize one capture, a date range, a person, a project, a topic, or mixed meeting and dictation history.
| name | transcripted-qa |
| description | Run comprehensive QA testing on the Transcripted application — build, unit tests, deep code audit, artifact validation, log analysis, and fix loop. |
You are a QA engineer testing the Transcripted macOS app. When this skill is invoked, run a comprehensive test pass across all tiers, audit changed code for bugs, report all findings in a structured report, and enter a fix loop for any failures.
git diff main --name-only to see what files changedSources/TranscriptedCore/Audio/** and Sources/Speech/** → audio, STT, recoverySources/TranscriptedCore/Pipeline/** and Sources/Meeting/** → meeting pipeline, retries, speaker namingSources/TranscriptedCore/Storage/** → saving, transcript formatting, artifact validationResources/AgentSkills/** → bundled agent skillsSources/TranscriptedCore/Stats/** → statsSources/TranscriptedCore/Models/FailedTranscription.swift and Sources/TranscriptedCore/Services/FailedTranscriptionManager.swift → retry queueSources/TranscriptedCore/Speaker/** → speaker DB, embeddings, clusteringSources/UI/** → settings, onboarding, menu bar, overlaySources/Capture/** → hotkeys and physical trigger routingSources/Observability/** → logs, Sentry, analytics, privacy sanitizersTools/** → standalone CLI, MCP, and QA tools~/Library/Application Support/Transcripted/captures/ and the app-owned state DBs when available.Run tiers in order. Stop and enter the fix loop if Tier 0 fails. Run independent tiers in parallel where possible (e.g., Tier 2 + Tier 3 can run alongside Tier 1).
cd /path/to/transcripted && bash build-deps.sh && bash build.sh 2>&1
build.sh is the authoritative app build on main. The old Transcripted.xcodeproj flow is historical and should not be used for current QA.
If dependency artifacts are missing, build-deps.sh is the required first step. If the build fails, extract the compiler or linker errors and enter the fix loop. Do NOT proceed to other tiers.
cd /path/to/transcripted && bash run-tests.sh 2>&1
run-tests.sh is the repo's fast custom swiftc regression suite, not XCTest. Capture the failing test names and assertion output exactly.
cd /path/to/transcripted && swift test 2>&1
Run this when the change touches Package.swift, Sources/TranscriptedCore/, or shared library interfaces. Also run it when you need additional confirmation that the package seam still compiles cleanly.
What's covered (by test file → source file):
| Test File | Source Under Test | What It Tests |
|---|---|---|
| Core Tests | ||
DateFormattingHelperTests | DateFormattingHelper | ISO dates, time-only, filename-precise formats |
DateParserTests | DateParser | Date string parsing from various formats |
DisplayStatusTests | DisplayStatus | Status enum transitions, progress values |
EmbeddingTests | SpeakerProfile | Embedding vector operations |
FailedTranscriptionManagerTests | FailedTranscriptionManager | Retry queue, persistence, auto-cleanup |
FailedTranscriptionTests | FailedTranscription | Model properties, serialization |
PipelineErrorTests | TranscriptionTypes | Error types, retryable classification |
RecordingValidatorTests | RecordingValidator | Disk space, permissions, path safety |
SpeechSegmentationTests | Transcription | Silence detection, segment merging |
StatsDatabaseTests | StatsDatabase | Schema, queries, aggregations |
TranscriptionTypesTests | TranscriptionTypes | Utterance, result, speaker segment models |
DiagnosticExporterTests | DiagnosticExporter | System info collection, GitHub issue URLs |
ModelDownloadServiceTests | ModelDownloadService | Error classification (12+ codes), error descriptions, disk space |
TranscriptExporterTests | TranscriptExporter | Markdown export, plain text export, filename generation, speaker prefix stripping |
TranscriptFormatterTests | TranscriptFormatter | YAML escaping, source labels, markdown generation, frontmatter construction |
TranscriptMetadataBuilderTests | TranscriptMetadataBuilder | CaptureQuality thresholds, RecordingHealthInfo |
TranscriptSaverTests | TranscriptSaver | File save, notifications |
TranscriptStoreTests | TranscriptStore | Transcript reading for tray UI |
AgentOutputTests | AgentOutput | JSON sidecar generation, index updates |
| Service Tests | ||
AudioResamplerTests | AudioResampler | 16kHz resampling, format conversion |
SpeakerDatabaseTests | SpeakerDatabase + SpeakerProfileMerger | Find by name, merge profiles, EMA update, WAL mode, confidence cap |
SpeakerEmbeddingMatcherTests | SpeakerEmbeddingMatcher | Cosine similarity (8 cases), L2 normalize, speaker matching with thresholds |
SpeakerProfileMergerTests | SpeakerProfileMerger | Name variants (15+ pairs), case insensitivity, empty strings, symmetry |
QwenServiceTests | QwenService | Response parsing, JSON extraction, markdown fence stripping, generic title filtering |
MeetingDetectorTests | TranscriptionTypes | SpeakerNameUpdate action enum cases |
EmbeddingClustererTests | EmbeddingClusterer | Cluster merging, absorption, DB-informed split |
| UI Tests | ||
PillStateManagerTests | PillStateManager | State machine transitions |
ContextualErrorBannerTests | ContextualErrorBanner | Error classification (6 types), icons, titles, recovery hints |
PillDimensionsTests | Design/Animations | All pill dimension constants, size relationships, animation timing |
PillSoundsTests | PillStateManager | System sound availability, sound preference toggle |
Parse output for: total tests run, passed, failed. For each failure: test class, method name, assertion message, file:line.
cd Tools/TranscriptedQA && swift run transcripted-qa validate-all --format json 2>&1
What's checked (by category):
| Category | Checks | What It Validates |
|---|---|---|
| Transcripts (per .md file) | yaml-present, yaml-required-keys, yaml-engine-stt, yaml-engine-diarize, yaml-sources, yaml-capture-quality, yaml-count-mic_utterances, yaml-count-system_utterances, yaml-count-total_word_count, body-has-sections, sidecar-exists, permissions | YAML frontmatter completeness, engine names match expected values, source types are valid, counts present and numeric, markdown has expected sections, .json sidecar exists alongside each .md, file permissions not world-readable |
| JSON Sidecars (per .json file) | json-valid, json-version, json-engine-stt, json-engine-diarize, json-duration, json-utterances-sorted, json-speaker-refs, md-match | Valid JSON, schema version present, engine names, duration is positive number, utterances sorted by start time, all speaker refs resolve, corresponding .md file exists |
| Speaker DB | speakers-integrity, speakers-wal-mode, speakers-schema, speakers-embedding-size, speakers-no-null-embeddings, speakers-valid-uuids, speakers-confidence-range, speakers-callcount-positive, speakers-name-source, speakers-permissions | SQLite integrity check, WAL journal mode, schema has expected columns, embeddings are 1024 bytes (256 floats), no NULL embeddings, UUIDs are valid format, confidence 0.0-1.0, call counts positive, name sources are known values, 0600 permissions |
| Stats DB | stats-integrity, stats-schema-recordings, stats-schema-daily, stats-positive-durations, stats-valid-dates, stats-permissions | SQLite integrity, recordings table schema, daily_activity table schema, no negative durations, dates are valid YYYY-MM-DD, 0600 permissions |
| Logs | jsonl-valid, jsonl-required-keys, jsonl-valid-levels, jsonl-valid-subsystems, jsonl-entry-count, jsonl-error-rate | Every line parses as JSON, required keys (ts, level, subsystem, message), levels are known values, subsystems are known values, reasonable entry count, error+warning rate below threshold |
| Index | json-valid, count-match, files-exist, no-duplicate-speakers | Valid JSON, transcript count matches array length, all referenced files exist on disk, no duplicate speaker IDs |
| Health | transcript-dir, logs-dir, qwen-model, disk-space, macos-version, no-crashes | ~/Documents/Transcripted exists, log directory exists, Qwen model files present, sufficient disk space, macOS 14.2+, no crash reports in DiagnosticReports |
After validating real data, also run the round-trip test:
cd Tools/TranscriptedQA && swift run transcripted-qa round-trip 2>&1
This generates clean test fixtures, validates them (86 checks, expect 0 failures), then runs 8 corruption tests — each one modifies the data in a specific way and verifies the validator catches it:
transcription_engine key → TranscriptValidator FAILduration_seconds → JSONSidecarValidator FAILtranscript_count → IndexValidator FAILRead ~/Library/Logs/Transcripted/app.jsonl directly. Check for:
error level entries — quote the message and subsystem~/Library/Logs/DiagnosticReports/ for any files containing "Transcripted"bash scripts/ui-smoke-test.sh
If Transcripted is not running, the script will attempt to launch it. Requires Accessibility permissions for Terminal/Claude Code.
What's checked (~15 checks):
| Check | What It Tests |
|---|---|
ui/app-running | App process is alive |
ui/menu-bar-only | App runs as accessory (no dock icon) |
ui/menu-bar-item | Status bar item exists |
ui/idle-no-windows | No windows open at idle state |
ui/hotkey-responsive | Cmd+Shift+R triggers log activity |
ui/hotkey-stop | Second Cmd+Shift+R stops recording |
ui/settings-window | Settings opens from menu and closes with Cmd+W |
ui/context-menu | Right-click menu is readable |
ui/menu-item-Start-Recording | "Start Recording" menu item present |
ui/menu-item-Settings | "Settings" menu item present |
ui/menu-item-Quit | "Quit" menu item present |
ui/idle-cpu | CPU usage < 5% at idle |
ui/idle-memory | Memory < 500MB at idle |
ui/defaults-* | UserDefaults keys readable |
ui/accessibility | App responds to accessibility queries |
If the script cannot get accessibility access, checks will WARN (not FAIL). This tier is informational — failures here don't trigger the fix loop.
For each changed source file (NOT test files), launch a subagent to:
git diff main for that fileFor each changed source file, check:
Run the full E2E test script:
bash scripts/ui-e2e-test.sh /tmp/transcripted-e2e
Requires: Transcripted running, Accessibility + Screen Recording permissions for the terminal app. Takes screenshots at every step for visual verification. Uses AppleScript to drive the UI via menu items (not hotkeys — hotkeys can be intercepted by focused apps like Chrome).
Flows tested (~30 checks):
| Flow | What It Does | What It Verifies |
|---|---|---|
| E1: Idle State | Check app at rest | No spurious windows, CPU < 5%, memory < 500MB |
| E2: Context Menu | Click menu bar item | "Start Recording", "Open Transcripts", "Settings", "Quit" all present |
| E3: Settings | Open Settings via menu, then Cmd+W | Window opens with UI content, closes cleanly |
| E4: Recording | Start via menu, wait 8s, stop via menu | Log activity confirms recording, new .md + .json created |
| E5: Transcript Verification | Read newest transcript | YAML has required keys, JSON sidecar valid, CLI validator passes |
| E6: Short Recording | Start and stop within 0.5s | App doesn't crash |
| E7: Rapid Stress | 5x rapid start/stop cycles | App survives, CPU recovers |
| E8: YouTube Comparison | Open a known YouTube video (JFK Moon Speech), record 30s of system audio, compare transcript | Word count > 0, system utterances captured, expected words from speech found in transcript, JSON sidecar valid |
Flow E8 details: Opens Chrome to JFK's 1962 Rice University "Moon Speech" at the 1:30 mark (NASA Video). Lets autoplay run, starts Transcripted recording for 30 seconds, stops both. Reads the transcript and checks for expected words: "condense", "history", "century", "animals", "caves", "years". Requires at least 3/6 matches to pass. This tests the full pipeline: system audio capture → diarization → Parakeet STT → transcript save.
After running, read screenshots from the output directory to visually verify:
After all tiers complete, produce this report:
## Transcripted QA Report — [today's date]
### Scope
[Changed files count, mapped domains, baseline transcript/speaker counts]
### Tier 0 — Build: PASS/FAIL
[Build time. If failed, show compiler errors]
### Tier 1 — Unit Tests: X/Y passed
[Test time. If failures: test name + assertion + file:line for each]
### Tier 2 — Artifact Validation: X passed, Y failed, Z warnings
[Show any FAIL or WARN results with details]
### Tier 3 — Log Analysis
[Total entries, error count by subsystem, notable errors quoted, timestamp gaps, crash reports]
### Tier 3.5 — UI Smoke Tests: X passed, Y warnings
[Results per check. WARN if accessibility permissions missing]
### Tier 4 — Code Audit
[Per-file findings table: file, issue, severity, line]
### Tier 5 — Coverage Analysis
[Per-file: tested methods vs untested, notable gaps]
### Tier 6 — End-to-End: X/Y flows passed (or SKIPPED)
[Results per flow. Include YouTube comparison: words captured, content match score, processing time]
[Read screenshots from /tmp/transcripted-e2e/ and verify UI state visually]
### Issues Found
1. [severity] [description] → [file:line] → [fix applied / suggested fix]
2. ...
### Warnings
- [notable warnings that don't block but should be tracked]
### What's NOT Tested
- [List of source files with zero test coverage that were changed]
- [List of important untested code paths in covered files]
When a tier fails:
Audio.swift or SystemAudioCapture.swift without asking the user — these are audio thread files| Artifact | Path |
|---|---|
| Transcripts | ~/Documents/Transcripted/*.md |
| JSON sidecars | ~/Documents/Transcripted/*.json |
| Index | ~/Documents/Transcripted/transcripted.json |
| Speaker DB | ~/Documents/Transcripted/speakers.sqlite |
| Stats DB | ~/Documents/Transcripted/stats.sqlite |
| Failed queue | ~/Documents/Transcripted/failed_transcriptions.json |
| Speaker clips | ~/Documents/Transcripted/speaker_clips/ |
| App logs | ~/Library/Logs/Transcripted/app.jsonl |
| Crash reports | ~/Library/Logs/DiagnosticReports/ |
| CLI tool | Tools/TranscriptedQA/ |
validate-all, round-trip, generate-fixtures, stress-test, plus per-category validatorsscripts/build-fluidaudio.sh --force if module errors occurWZyRbnpGyzQ at t=90s) — expects words: condense, history, century, animals, caves, years