hearing

Use when building the agent / "copilot" layer of the hearing meeting-transcription project — the thing that consumes a transcript and produces feedback (summaries, action items, decisions, follow-up research in batch; running notes, suggested questions, relevant docs, fact-checks live). Triggers on agent layer, meeting copilot, post-meeting summary, action items, decision extraction, follow-up research, live notes, suggested questions, real-time RAG, fact-check, queue consumer, asyncio.Queue of utterances, finalized segments, FrameProcessor, Pipecat, LiveKit Agents, RealTimeAgentPipeline, non-blocking transcript intercept, backpressure / decoupling the LLM from capture, feeding me/them + diarization labels into the prompt, vector store over meeting context, LlamaIndex, Qdrant, web search during a meeting, deep-research follow-up, SimpleLLMAgent, pluggable LLM backend, batch-vs-live unification. The agent defaults to Claude but stays pluggable.

Use when designing, wiring, or modifying the hearing project's pipeline architecture — the SSOT for HOW the four layers (audio capture, STT, diarization/speaker-ID, agents) compose by dependency injection. Triggers on capture source, channel split, mic vs system audio, STT engine facade, transcribe(audio)->segments, streaming transcription, diarization interface, speaker label, TranscriptSegment, agent consumer, pipeline assembly, batch vs live, recipe-style composition, swap/substitute a component, Protocol/dataclass contracts, asyncio.Queue between stages, backpressure/decoupling, progressive disclosure of the pipeline, and any "how do the hearing-* skills fit together" or "what's the shared data model" question. This is the contract every other hearing-* skill (hearing-audio-capture, hearing-stt, hearing-diarization, hearing-agents) implements against.

Use when capturing audio for meeting transcription on macOS (and the iOS limits) — getting BOTH the microphone AND system audio (the other participants coming out of your speakers) into a Python pipeline. Triggers on system audio capture, loopback, BlackHole, Aggregate Device, Multi-Output Device, Audio MIDI Setup, Core Audio taps, AudioHardwareCreateProcessTap, AudioTee, ScreenCaptureKit, SCStream, PyObjC SCK -3805, sounddevice InputStream, soundcard, PyAudio, ffmpeg avfoundation, channel slicing / mapping, "record what I hear", "capture the meeting audio", separate mic vs system channels, the "me vs them" channel trick, virtual audio driver, sample-rate / clock / drift mismatch, screen-recording permission for audio, and iOS/iPhone call/app audio recording restrictions (ReplayKit, iOS 18.1 call recording).

Use when working on speaker attribution in the "hearing" meeting-transcription library — turning a transcript into a who-spoke-when, speaker-labeled (diarized) transcript, separating remote participants, and deciding which library to run. Triggers on diarization, speaker diarization, who spoke when, speaker labels, Speaker 1/Speaker 2, speaker turns, speaker segmentation, speaker attribution, "me vs them", local vs remote speaker, channel energy / channel trick, dual VAD, speaker identification, voice fingerprinting, voice enrollment, speaker embedding, pyannote.audio, pyannote community-1, speaker-diarization-3.1, NVIDIA NeMo Sortformer/MSDD, WhisperX diarize, SpeechBrain ECAPA/x-vector, DER (diarization error rate), or feeding speaker labels into the agent prompt. Distinguish from raw transcription (that is hearing-stt) and from audio capture/channel layout (that is hearing-audio-capture).

Use when building or designing the TypeScript/React frontend for the "hearing" meeting-transcription project — the transcript viewer, the live AI-copilot overlay, the agent-feedback panel (notes, suggested questions, surfaced docs, fact-checks), or the meeting/project-context UI. Covers schema-driven UI with zodal (defineCollection, Zod v4, content-metadata bifurcation routing transcript metadata to a DB and audio/recording content to object storage), command-dispatch with acture (one command → palette + hotkeys + AI tool + MCP + tests), how the Python backend (argh CLI + HTTP layer) feeds the UI, and how live finalized utterances stream to the client. Triggers on transcript view, copilot overlay, agent panel, feedback panel, suggested questions, surfaced docs, fact-check, meeting collection, segment schema, speaker/me-vs-them labels, zodal collection, acture command, bifurcation, lk.transcription, SSE/WebSocket transcript stream, novice-friendly frontend, declarative/schema UI for hearing. Explain JS/TS deci

Use when building the streaming / live / real-time milestone of the hearing meeting-transcription library — turning the batch (record-then-transcribe) path into a low-latency loop that emits transcript segments as people speak. Triggers on streaming STT, real-time transcription, live captions, RealtimeSTT, whisper_streaming, SimulStreaming, WhisperLive, OpenAI Realtime API (gpt-realtime-whisper), Pipecat, LiveKit Agents, VAD / voice activity detection, silero-vad, webrtcvad, utterance / turn finalization, partial vs final segments, word-level timestamps for keyword triggers, feed_audio, asyncio.Queue decoupling, backpressure between capture/STT/diarization/agent stages, latency budget (VAD + STT + LLM TTFT + tool/RAG + transport), semantic turn detection, mid-sentence pause handling, and any task that adds a streaming capture/sink without changing the STT or agent interfaces.

Use when choosing, wiring, swapping, or benchmarking a speech-to-text (STT/ASR) engine behind the hearing project's transcription facade — the pluggable layer that turns audio into segments so the rest of the architecture never names a specific engine. Triggers on transcribe(audio)->segments, ASR engine selection, Whisper / whisper.cpp / faster-whisper / WhisperX / distil-whisper, NVIDIA Parakeet-TDT / Canary, Moonshine, Apple-Silicon MLX paths (lightning-whisper-mlx, mlx-audio, Lightning-SimulWhisper), cloud STT APIs (OpenAI gpt-4o-transcribe / -mini / gpt-realtime-whisper / gpt-4o-transcribe-diarize, Deepgram Nova-3, AssemblyAI Universal, Google Chirp, Azure Speech, Speechmatics, Rev), per-minute STT pricing, local-vs-cloud / on-device decision, WER tolerance, concurrency, self-hosting break-even (~100k min/month), CTranslate2 / int8 quantization, model-name versioning and the ~June-2026 OpenAI transcribe retirements, and "which engine should I use for meeting transcription". Diarization (who-spoke-when) is