Use this skill when building real-time, bidirectional streaming applications with the Gemini Live API. Covers WebSocket-based audio/video/text streaming, voice activity detection (VAD), native audio features, function calling, session management, ephemeral tokens for client-side auth,...
Instrucciones de origen · Vista previa de solo lectura
name
gemini-live-api-dev
description
Use this skill when building real-time, bidirectional streaming applications with the Gemini Live API. Covers WebSocket-based audio/video/text streaming, voice activity detection (VAD), native audio features, function calling, session management, ephemeral tokens for client-side auth,...
Use this skill when building real-time, bidirectional streaming applications with the Gemini Live API. Covers WebSocket-based audio/video/text streaming, voice activity detection (VAD), native audio features, function calling, session management, ephemeral tokens for client-side auth,...
Overview
The Live API enables low-latency, real-time voice and video interactions with Gemini over WebSockets. It processes continuous streams of audio, video, or text to deliver immediate, human-like spoken responses.
[!NOTE]
The Live API currently only supports WebSockets. For WebRTC support or simplified integration, use a partner integration.
Models
gemini-3.1-flash-live-preview — Optimized for low-latency, real-time dialogue. Native audio output, thinking (via thinkingLevel). 128k context window. This is the recommended model for all Live API use cases.
Output: Raw PCM, little-endian, 16-bit, mono. 24kHz sample rate.
[!IMPORTANT]
Use send_realtime_input / sendRealtimeInput for all real-time user input (audio, video, and text). send_client_content / sendClientContent is only supported for seeding initial context history (requires setting initial_history_in_client_content in history_config). Do not use it to send new user messages during the conversation.
[!WARNING]
Do not use media in sendRealtimeInput. Use the specific keys: audio for audio data, video for images/video frames, and text for text input.
Quick Start
Authentication
Python
from google import genai
client = genai.Client(api_key="YOUR_API_KEY")
from google.genai import types
config = types.LiveConnectConfig(
response_modalities=[types.Modality.AUDIO],
system_instruction=types.Content(
parts=[types.Part(text="You are a helpful assistant.")]
)
)
asyncwith client.aio.live.connect(model="gemini-3.1-flash-live-preview", config=config) as session:
pass# Session is active
[!IMPORTANT]
A single server event can contain multiple content parts simultaneously (e.g., audio chunks and transcript). Always process all parts in each event to avoid missing content.
Python
asyncfor response in session.receive():
content = response.server_content
if content:
# Audio — process ALL parts in each eventif content.model_turn:
for part in content.model_turn.parts:
if part.inline_data:
audio_data = part.inline_data.data
# Transcriptionif content.input_transcription:
print(f"User: {content.input_transcription.text}")
if content.output_transcription:
print(f"Gemini: {content.output_transcription.text}")
# Interruptionif content.interrupted isTrue:
pass# Stop playback, clear audio queue
JavaScript
// Inside the onmessage callbackconst content = response.serverContent;
if (content?.modelTurn?.parts) {
for (const part of content.modelTurn.parts) {
if (part.inlineData) {
const audioData = part.inlineData.data; // Base64 encoded
}
}
}
if (content?.inputTranscription) console.log('User:', content.inputTranscription.text);
if (content?.outputTranscription) console.log('Gemini:', content.outputTranscription.text);
if (content?.interrupted) { /* Stop playback, clear audio queue */ }
Live Translation (Gemini Live Translate)
The Live API supports real-time, low-latency streaming translation of speech (audio) across 70+ languages. For full details on options and capabilities, see the Live Translate Guide.
Model
gemini-3.5-live-translate-preview — The recommended translation model for all Live Translate use cases.
Configuration (TranslationConfig)
To enable translation, specify a TranslationConfig object inside your live session setup:
Python SDK: Configure the connection using translation_config on LiveConnectConfig:
Async function calling — Not yet supported; function calling is synchronous only. The model will not start responding until you've sent the tool response.
Proactive audio — Not yet supported in Gemini 3.1 Flash Live. Remove any configuration for this feature.
Affective dialogue — Not yet supported in Gemini 3.1 Flash Live. Remove any configuration for this feature.
Code execution — Not supported
URL context — Not supported
Migrating from Gemini 2.5 Flash Live
When migrating from gemini-2.5-flash-native-audio-preview-12-2025 to gemini-3.1-flash-live-preview:
Model string — Update from gemini-2.5-flash-native-audio-preview-12-2025 to gemini-3.1-flash-live-preview.
Thinking configuration — Use thinkingLevel (minimal, low, medium, high) instead of thinkingBudget. Default is minimal for lowest latency.
Server events — A single event can contain multiple content parts simultaneously (audio + transcript). Process all parts in each event.
Client content — send_client_content is only for seeding initial context history (set initial_history_in_client_content in history_config). Use send_realtime_input for text during conversation.
Turn coverage — Defaults to TURN_INCLUDES_AUDIO_ACTIVITY_AND_ALL_VIDEO instead of TURN_INCLUDES_ONLY_ACTIVITY. If sending constant video frames, consider sending only during audio activity to reduce costs.
Async function calling — Not yet supported. Function calling is synchronous only.
Proactive audio & affective dialogue — Not yet supported. Remove any configuration for these features.
Best Practices
Use headphones when testing mic audio to prevent echo/self-interruption
Enable context window compression for sessions longer than 15 minutes
Implement session resumption to handle connection resets gracefully
Use ephemeral tokens for client-side deployments — never expose API keys in browsers
Use send_realtime_input for all real-time user input (audio, video, text). Reserve send_client_content only for seeding initial context history
Send audioStreamEnd when the mic is paused to flush cached audio
Clear audio playback queues on interruption signals
Process all parts in each server event — events can contain multiple content parts
Documentation Lookup
When MCP is Installed (Preferred)
If the search_docs tool (from the Google MCP server) is available, use it as your only documentation source:
Call search_docs with your query
Read the returned documentation
Trust MCP results as source of truth for API details — they are always up-to-date.
[!IMPORTANT]
When MCP tools are present, never fetch URLs manually. MCP provides up-to-date, indexed documentation that is more accurate and token-efficient than URL fetching.
When MCP is NOT Installed (Fallback Only)
If no MCP documentation tools are available, fetch from the official docs index:
The Live API supports 70 languages including: English, Spanish, French, German, Italian, Portuguese, Chinese, Japanese, Korean, Hindi, Arabic, Russian, and many more. Native audio models automatically detect and switch languages.
