This skill helps an LLM generate correct audio code with @ax-llm/ax. Use when the user asks about ai.transcribe(), ai.speak(), signature audio inputs or outputs, agent audio behavior, .chat() conversational audio, OpenAI audio or realtime models, Gemini Live native audio, Grok Voice Agent models, voices, formats, transcripts, or how audio fits with structured outputs.
This skill helps an LLM generate correct audio code with @ax-llm/ax. Use when the user asks about ai.transcribe(), ai.speak(), signature audio inputs or outputs, agent audio behavior, .chat() conversational audio, OpenAI audio or realtime models, Gemini Live native audio, Grok Voice Agent models, voices, formats, transcripts, or how audio fits with structured outputs.
version
23.0.14
Audio I/O Codegen Rules (@ax-llm/ax)
Use this skill for audio in Ax. Pick the smallest audio surface that matches the job:
Use ai.transcribe(...) for batch speech-to-text.
Use ai.speak(...) for batch text-to-speech.
Use speech:audio signature outputs for structured programs that should return synthesized audio artifacts.
Use .chat() audio config for conversational or realtime audio turns.
Core Rules
Input :audio is an audio input value: { data, format?, mimeType?, sampleRate?, channels? }.
Output :audio is a scripted audio artifact. The model returns plain text for that field; Ax synthesizes it after structured output parsing.
Output audio JSON schema is model-facing string, not a binary object.
Agents transcribe input audio fields before planner/executor/responder stages by default, so agent stages see text instead of base64 audio.
Realtime and conversational audio still use .chat() and modelConfig.audio.
Batch signature audio artifacts use forward-time speech options, not .
The model emits a text script for speech; Ax replaces it with AxChatAudioOutput after result selection. If the field already contains an audio artifact with { data } or { id }, Ax leaves it alone.
The agent runtime transcribes recording first and passes the transcript through the internal agent stages. Use direct ax(...) or .chat() when you specifically want native audio understanding in the model call.
Conversational .chat() Audio
Use modelConfig.audio for conversational audio turns where audio is part of the chat response instead of a structured signature field.
Gemini Live uses a one-turn WebSocket call under .chat(). It expects PCM input for native audio turns; use format: 'pcm16' or mimeType: 'audio/pcm;rate=16000'.
Grok Voice Defaults
Use axAIGrokVoiceDefaultConfig() for xAI Grok Voice Agent:
Grok Voice uses a one-turn WebSocket call under .chat(). It expects PCM input for spoken input turns; use format: 'pcm16' or mimeType: 'audio/pcm'.
Streaming Audio
OpenAI audio chat, OpenAI Realtime, Gemini Live, and Grok Voice all default to non-streaming, but each can stream deltas when you pass { stream: true }.
Use signature audio outputs for structured speech artifacts:
const gen = ax('question:string -> answer:string, speech:audio');
Use .chat() audio when the response itself is a conversational audio turn. Do not combine .chat() audio output with provider-native structured response formats unless that provider explicitly supports the combination.
