yandex-speechkit

name

yandex-speechkit

description

Work with Yandex SpeechKit speech-to-text, especially STT v3 over gRPC. Use when Codex needs to choose between streaming, synchronous, or asynchronous recognition; prepare audio for SpeechKit; generate or write clients from the cloudapi proto files; configure STT v3 request options; troubleshoot authentication, folder ID, format, or session issues; or post-process SpeechKit transcripts into plain text, timestamps, subtitles, or structured output.

Yandex SpeechKit

Overview

Use this skill to implement or troubleshoot speech recognition with Yandex SpeechKit. Default to STT v3 for new gRPC integrations and choose the recognition mode based on whether audio is live, short and pre-recorded, or long and pre-recorded.

Workflow Decision

Choose the recognition mode before writing code:

• Use streaming recognition for live microphone audio, assistants, bots, and real-time captions.
• Use synchronous recognition for short pre-recorded voice messages when a single request-response flow is enough.
• Use asynchronous recognition for long recordings, meetings, calls, subtitles, and other offline transcription jobs.

If the task starts from a recorded file, do not force it through streaming unless the user explicitly needs real-time emulation.

Read references/mode-selection.md when the mode is not obvious.

Default Choices

• Prefer API v3 for new projects.
• Prefer gRPC clients generated from yandex/cloud/ai/stt/v3/stt_service.proto.
• Prefer service-account authentication when possible.
• For SpeechKit scripts, prefer YANDEX_IAM_TOKEN / --iam-token over YANDEX_API_KEY / --api-key; API keys are fallback credentials.
• Treat Object Storage credentials and SpeechKit credentials as separate. S3-compatible Object Storage upload uses static access/secret keys, while SpeechKit recognition uses IAM token or API key and folder context.
• Prefer explicit audio preparation over guessing server-side behavior.
• Prefer handling partial, final, and final_refinement separately in streaming clients.
• Prefer asynchronous recognition for long files and batch workflows.

Streaming STT v3

When implementing streaming recognition:

1. Generate or use a client for Recognizer.RecognizeStreaming.
2. Send session_options as the first client message.
3. Stream audio chunks after the session is configured.
4. Keep chunk cadence close to real time.
5. Stop or reconnect if the session hits timing or payload limits.
6. Accumulate partial results for UI feedback and final or final_refinement for committed transcript text.

Use silence_chunk when the client already knows the silence duration and wants to avoid transmitting literal silence bytes. Use explicit eou controls only when the application owns utterance boundary detection.

Read references/stt-v3-protocol.md before implementing request and response handling.

File Recognition

For recorded audio files:

1. Decide whether the file is short enough for synchronous recognition or should go through async recognition.
2. Normalize the input audio format before sending it.
3. Submit the recognition request with explicit settings instead of relying on defaults.
4. Poll or fetch the final result for async operations.
5. Convert the result into the user's requested shape: plain transcript, timed words, subtitles, speaker-separated text, or structured JSON.

When the user asks for subtitles, meeting notes, or call transcripts, assume asynchronous recognition is the default unless there is a strong reason not to.

If the project has Object Storage settings in .env or process env, treat asynchronous recognition as mandatory for file transcription workflows. In that case do not route recorded files through the local synchronous chunking path unless the user explicitly asks for low-level debugging of the sync endpoint.

Audio Preparation

Before sending audio:

• Identify whether the source is live audio or a file.
• Confirm the container and codec instead of inferring from the file extension.
• Convert unsupported or inconvenient inputs before calling SpeechKit.
• Preserve channel and timing information when the requested output depends on it.

Use the bundled scripts when helpful:

• scripts/inspect_audio.py to inspect codec, sample rate, channels, duration, and a recommended SpeechKit path.
• scripts/prepare_audio.py to convert inputs with ffmpeg into WAV, LINEAR16, OGG Opus, or MP3.
• scripts/split_audio_by_size.py to break large local recordings into chunks under a maximum file size before upload or batch processing.
• scripts/transcribe_local_in_parts.py to split a large local recording and run synchronous recognition over each chunk when you want to stay entirely on local files.
• scripts/transcribe_file_async.py to prepare a local file, upload it to Object Storage, create a private URL, and run async recognition end-to-end.
• scripts/transcribe_file_async.py --prepared-file ... --skip-prepare to reuse converted audio after a failed upload or recognition request.
• scripts/transcribe_file_async.py --skip-upload --source-url ... to reuse an already uploaded object URL.
• scripts/transcribe_file_async.py <work_dir> --operation-id ... to resume polling/fetching an already submitted async operation.

When Object Storage env is configured, prefer this order:

1. scripts/prepare_audio.py
2. scripts/object_storage_upload.py or scripts/transcribe_file_async.py
3. scripts/object_storage_presign.py when a private URL is needed explicitly
4. scripts/speechkit_async_recognize_v3.py or the full wrapper scripts/transcribe_file_async.py

If you need protocol-level format guidance or common conversion pitfalls, read references/gotchas.md.

Recognition Options

Adjust recognition options only when the task needs them:

• Enable normalization when the user needs cleaner final text.
• Keep raw recognition behavior when downstream code needs the original surface form.
• Configure language restrictions only when the domain is narrow enough to benefit from them.
• Enable speaker or conversation analysis only when the output actually needs it.
• Use summarization only when the workflow explicitly asks for summary output from the transcript pipeline.

Avoid turning on every advanced option by default. Start with the smallest settings set that satisfies the request.

Speaker Information

When the user wants speaker information in the transcript, prefer speaker-aware output instead of a flat transcript.

• Enable speaker labeling or conversation analysis when the API path supports it.
• If exact speaker names are unknown, use stable labels such as Спикер 1, Спикер 2, Спикер 3.
• If the recording or filename gives enough context to identify a participant confidently, replace generic labels with names or roles.
• If speaker assignment is uncertain, keep the generic speaker label and say that diarization was inferred automatically.

Default speaker-aware output for meetings and calls:

• One paragraph or block per speaker turn.
• Optional timestamp per turn when timing is available or requested.
• Cleaned text inside each speaker block when the user asks for readable output.

Preferred shapes:

• [00:12] Спикер 1: ...
• Спикер 2: ...
• Structured JSON with speaker, start, end, and text only when another system will consume it.

Do not over-claim diarization quality. If the audio contains overlap, crosstalk, or poor channel separation, say that speaker attribution may be approximate.

Validation Loop

For any non-trivial integration:

1. Validate authentication first.
2. Validate audio format second.
3. Validate the selected recognition mode third.
4. Validate response parsing against real partial and final payloads.
5. Re-run with a short known-good audio sample before debugging large inputs.

If behavior is unclear, inspect the exact request and response shape before changing model or audio settings.

Useful command starters:

• python scripts/inspect_audio.py path/to/file
• python scripts/prepare_audio.py input.m4a output.wav --format wav --sample-rate 16000 --channels 1
• python scripts/split_audio_by_size.py meeting.mp3 chunks/ --max-size-mb 20
• python scripts/transcribe_local_in_parts.py meeting.mp3 workdir/ --max-size-mb 20 --lang ru-RU
• python scripts/transcribe_file_async.py meeting.mp4 workdir/
• python scripts/transcribe_file_async.py meeting.mp4 workdir/ --prepared-file workdir/prepared/meeting.ogg --skip-prepare
• python scripts/transcribe_file_async.py meeting.mp4 workdir/ --skip-prepare --skip-upload --source-url "https://..."
• python scripts/transcribe_file_async.py workdir/ --operation-id <operation-id>
• python scripts/speechkit_sync_recognize.py file.wav --folder-id <folder-id> --format lpcm --sample-rate-hertz 16000
• python scripts/speechkit_async_recognize_v3.py --uri https://storage.yandexcloud.net/.../file.wav --folder-id <folder-id> --poll

Gotchas

• Service-account authentication and folder handling behave differently from user-token flows; do not blindly send folder_id in every request.
• HTTP 403 PermissionDenied usually means wrong SpeechKit principal, missing SpeechKit rights, wrong YANDEX_FOLDER_ID, or a key/token from a different cloud/folder. Retry with --iam-token when an API key fails.
• Object Storage upload success does not prove SpeechKit permission. Check both credential sets independently.
• Streaming is for real-time audio. For recorded files, synchronous or asynchronous modes are usually the correct choice.
• In streaming mode, the first message must configure the session before audio chunks are sent.
• If the client stops sending messages for too long, the streaming session is terminated.
• final_refinement is a separate response type and may arrive after final when normalization is enabled.
• Large gRPC results can exceed default client receive-size limits.
• The protocol exposes advanced analysis features; do not wire them into the default path unless the user asks for them.

Read references/gotchas.md when debugging failures or inconsistent output.

Output Shapes

Default to the smallest useful output:

• Plain text transcript for simple transcription requests.
• Segment timestamps when the user asks for subtitle-like output.
• Word timings only when explicitly needed.
• Structured JSON when the result feeds another system.
• Speaker-separated output only when speaker labeling or analysis is enabled.

State any assumptions about normalization, timestamps, and speaker separation in the final result.

Output Location And Naming

Write final user-facing results into ~/transcription.

• Create ~/transcription if it does not exist.
• Treat this directory as the default destination for raw transcripts, cleaned transcripts, subtitles, notes, and summaries produced from a recording.
• Keep temporary chunking or scratch artifacts outside ~/transcription unless the user explicitly asks to retain them there.

Filename rules:

• Include the event date first when it can be inferred from the source file, metadata, or user request.
• Include a short Russian description of the event after the date.
• Use 2 to 6 Russian words that identify the event, for example meeting topic, call type, or speaker/context.
• Use lowercase words joined by underscores.
• Keep the filename stable and descriptive instead of generic names like transcript.txt or meeting.txt.

Preferred pattern:

• YYYY-MM-DD_описание_события.txt
• YYYY-MM-DD_описание_события_cleaned.txt
• YYYY-MM-DD_описание_события_summary.txt
• YYYY-MM-DD_описание_события_notes.txt

If the exact event title is unclear, infer a conservative Russian description from the recording context and mention that it was inferred.

Transcript Post-Processing

When the user asks to clean up a transcript after recognition, choose the lightest transformation that satisfies the request:

• Use light cleanup for raw ASR text that should stay close to the original wording.
• Use readable rewrite when the user wants a clean written-text version of spoken language.
• Use structured extraction when the user wants notes, tasks, decisions, or a summary instead of the transcript itself.

For light cleanup:

• Remove obvious ASR noise such as repeated words, connection checks, filler bursts, and empty fragments.
• Keep the original order and meaning.
• Do not silently invent missing content.
• Preserve profanity, jargon, and product names unless the user asks to sanitize them.

For readable rewrite:

• Turn spoken phrasing into normal written Russian or the requested language.
• Split long ASR blocks into paragraphs.
• Fix punctuation, casing, and obvious recognition artifacts.
• Keep uncertain fragments conservative; mark uncertainty instead of guessing when meaning is unclear.
• Preserve speaker boundaries when the user asked for speaker information.

For structured extraction:

• Derive the structure from the request: summary, meeting notes, roadmap, action items, decisions, risks, or Q&A.
• Keep a link back to the raw or cleaned transcript when traceability matters.
• Separate facts said in the recording from your own inferences.

Template-Based Transcript Analysis

When the user provides or references an analysis template, use it as the target output schema for the transcript post-processing step. The template is not just formatting; it defines what signals to extract, how to group them, and what level of interpretation is allowed.

Default flow:

1. Produce or locate the raw transcript.
2. Make a lightly cleaned transcript if ASR noise would make extraction unreliable.
3. Read the requested template and identify its required sections, examples, wording rules, and forbidden shortcuts.
4. Extract only transcript-backed signals that fit the template.
5. Fill the template with structured analysis.
6. Mark weakly supported or inferred points explicitly instead of presenting them as facts.
7. Save the analysis as a separate derivative file next to the transcript, for example YYYY-MM-DD_описание_события_ux_analysis.md.

For UX scenario templates such as /Users/n.baryshnikov/Projects/18studio/records/ux-scenario-prompt.md, extract UX signals rather than summarizing the whole meeting:

• user type and context
• user goal or job to be done
• current user path and concrete steps
• expectations at each step
• friction, confusion, errors, manual workarounds, drop-off points
• consequences for the user
• success criteria
• recommendations that follow from the observed problem

Write UX analysis as an observer:

• Describe what the user does, sees, expects, and misunderstands.
• Separate fact, problem, consequence, and recommendation.
• Avoid vague adjectives such as "convenient", "clear", "intuitive", or "beautiful" unless the transcript gives concrete evidence.
• Do not turn every comment into a product requirement. Preserve uncertainty when the transcript is ambiguous.
• Do not mix a UX problem with a technical implementation unless the template explicitly asks for technical recommendations.

If the template and transcript conflict, keep the template structure but state that the transcript does not contain enough evidence for the missing section.

When producing multiple forms, prefer this order:

1. Raw transcript
2. Cleaned transcript
3. Structured derivative such as notes, summary, or template-based analysis

If the transcript quality is poor, say so explicitly and state whether the problem appears to come from audio quality, speaker overlap, domain vocabulary, or recognition errors.