pipecat

name	pipecat
description	Pipecat realtime voice/multimodal bots. Covers pipelines/frames, transports, RTVI, Pipecat Cloud deploy. Use when building real-time voice bots (STT/LLM/TTS pipelines), multimodal AI agents, WebRTC/WebSocket transports, or deploying to Pipecat Cloud. Keywords: pipecat, pipecat-ai, RTVI, WebRTC, voice bot.
metadata	{"version":"1.6.0","release_date":"2026-07-21"}

Pipecat

Pipecat is an open-source Python framework for building real-time voice and multimodal bots. It composes streaming speech/LLM/TTS services into a low-latency pipeline, connected via transports (WebRTC/WebSocket) and client SDKs using the RTVI message standard.

Links

• Documentation
• Changelog
• GitHub
• PyPI (framework)
• PyPI (cloud SDK)
• Full-text extract used for this skill

Quick navigation

• Installation (packages/extras/CLI): references/installation.md
• Migration to 1.0: references/migration-1-0.md
• Concepts & architecture: references/core-concepts.md
• Session initialization (runner/bot/client): references/session-initialization.md
• Pipeline & frames: references/pipeline-and-frames.md
• Transports: references/transports.md
• Speech input & turn detection: references/speech-input-and-turn-detection.md
• Client SDKs + RTVI messaging: references/client-sdks-rtvi.md
• CLI (init/tail/cloud): references/cli.md
• Function calling (server): references/function-calling.md
• Context management: references/context-management.md
• LLM inference: references/llm-inference.md
• Text to speech (TTS): references/text-to-speech.md
• Deployment (pattern/platforms): references/deployment.md
• Server APIs (supported services): references/server-services.md
• Server Utilities (runner): references/server-runner.md
• Server APIs (pipeline/task/params): references/server-pipeline-apis.md
• Pipecat Cloud ops: references/pipecat-cloud.md
• Troubleshooting: references/troubleshooting.md

Mental model (cheat sheet)

• Pipeline: ordered processors that consume/emit frames.
• Frames: the streaming units (audio/text/video/context/events) flowing through the pipeline.
• Transport: connectivity + media IO + session state (WebRTC/WebSocket/provider realtime).
• Runner: HTTP service that starts sessions and spawns a bot process with transport credentials.
• Client SDK: starts the bot, connects transport, sends messages/requests, receives events.

Recipes

1) Keep secrets server-side

• Put provider API keys (LLM/STT/TTS) only on the server/bot container.
• The client should call a server start endpoint (startBot / startBotAndConnect) to receive transport credentials (e.g., a room URL + token), not provider keys.

2) Use WebRTC for production voice

• Prefer a WebRTC transport (e.g., Daily) for resilience and media quality.
• Use a WebSocket transport mostly for server↔server, prototypes, or constrained environments.

2b) Design for streaming + overlap

• Keep the pipeline fully streaming (avoid batching whole turns when you can).
• If your services support it, start TTS from partial LLM output to reduce perceived latency.

3) Initialize and evolve context via RTVI

• Initialize the bot’s pipeline context from the server start request payload.
• For ongoing interaction, prefer a dedicated “send text” style API (when available) instead of deprecated context append methods.

4) Function calling: end-to-end flow

• LLM requests a function call.
• Client registers a handler by function name.
• Client returns a function-call result message back to the bot.

5) Pipecat Cloud deployment basics

• Build/push an image that matches the expected platform (Pipecat Cloud requires linux/arm64 in the docs).
• Use a deployment config file for repeatability.
• Configure pool sizing with min_agents (warm capacity) and max_agents (hard limit).

Critical gotchas / prohibitions

• Do not embed sensitive API keys in client apps.
• Expect and handle “at capacity” responses (HTTP 429) when the pool is exhausted.
• Plan for cold-start latency if min_agents = 0.
• Ensure secrets and image-pull credentials are created in the same region as the deployed agent.
• Do not assume deprecated import shims or service-specific context classes still exist in 1.0.0; audit imports before upgrading.
• Do not keep VAD/turn-detection logic on transport params; current releases route that control through LLMUserAggregator strategies.
• Do not assume OpenAIResponsesLLMService is HTTP-based anymore; WebSocket is now the default implementation.
• Do not send a single button field in the RTVI dtmf client message; as of 1.6.0 it requires buttons (a list), and RTVI.PROTOCOL_VERSION is 2.1.0.
• Do not filter OTel dashboards on old GenAI span attribute names (az.ai.openai, xai, mistral, gen_ai.usage.reasoning_tokens, bare tokens.*); 1.6.0 renamed these to standard gen_ai.* conventions.

Release Highlights (1.6.0)

New transport and reasoning

• MOQTransport: a Media over QUIC transport giving bots a bidirectional, low-latency audio + RTVI channel over QUIC instead of WebRTC/WebSocket (pip install "pipecat-ai[moq]"). The bot can run as its own MoQ server for local dev; the development runner gained --moq-serve, --moq-bind, and TLS flags.
• reasoning in OpenAI Responses services: OpenAIResponsesLLMService / OpenAIResponsesHttpLLMService accept a ReasoningConfig(effort=..., summary=...); encrypted reasoning round-trips across turns and tool calls automatically, and summaries surface as thought frames / on_assistant_thought.

Flows, evals, and new services

• Pipecat Flows NO_RESPONSE: a function can return (result, NO_RESPONSE) to finish the call without an LLM run or node transition, letting the next user utterance drive the next response.
• Eval scenarios absent: true: an expectation that passes only if no matching event arrives within within_ms, useful for catching duplicate-output regressions.
• New services: CrusoeLLMService and BasetenLLMService (OpenAI-compatible LLM services for Crusoe Cloud and Baseten), DeepgramFluxTTSService (websocket TTS for Deepgram Flux, token-streamed by default).
• Audio token usage: LLMTokenUsage gains input_audio_tokens / output_audio_tokens / cache_read_input_audio_tokens, populated by OpenAIRealtimeLLMService (and Azure realtime) and GeminiLiveLLMService.

Breaking changes and deprecations

• RTVI dtmf client message uses buttons (a list) instead of button; RTVI.PROTOCOL_VERSION is now 2.1.0.
• OTel GenAI span attributes were renamed to standard conventions: gen_ai.provider.name values for Azure/xAI/Mistral, and gen_ai.usage.reasoning_tokens -> gen_ai.usage.reasoning.output_tokens; OpenAI Realtime and Gemini Live token attributes are now standardized gen_ai.usage.* names instead of ad hoc tokens.*.
• ElevenLabs services default to TTS model eleven_flash_v2_5 instead of the now-deprecated eleven_turbo_v2_5.
• PronunciationDictionaryLocator is deprecated in favor of text_transforms / replace_text (removal in 2.0.0).
• Turn-strategy reset() is deprecated in favor of handle_user_turn_started() / handle_user_turn_stopped() lifecycle callbacks (removal in 2.0.0).

Release Highlights (0.0.109 -> 1.2.0)

Runtime and service additions

• OpenAIResponsesLLMService now defaults to a persistent WebSocket connection; the prior HTTP behavior moved to OpenAIResponsesHttpLLMService.
• Inworld Realtime LLM adds a WebSocket cascade STT/LLM/TTS path with semantic VAD and function calling.
• MistralTTSService adds streaming Voxtral TTS, and TTS/STT services gained more runtime-update and sample-rate options.
• The development runner now exports a module-level FastAPI app for custom routes before main().

Tooling and context changes

• Function calling now supports grouped parallel tool batches, async tool completion after interruption, and streaming intermediate tool results.
• Context editing now has LLMMessagesTransformFrame, and the framework standardizes on universal LLMContext / LLMContextAggregatorPair.
• OpenAI tool schemas can now include provider-specific custom_tools.
• 1.2.0 adds add_tool_change_messages for LLM aggregators, widens tool_resources into deprecated app_resources, and extends async-tool compatibility across more realtime providers.

Turn-taking and client protocol

• 1.2.0 adds explicit inference/finalization turn hooks (on_user_turn_inference_triggered, LLMTurnCompletionUserTurnStopStrategy, FilterIncompleteUserTurnStrategies) for smarter end-of-turn gating.
• RTVI grows first-class UI Agent Protocol support with ui-event, ui-snapshot, ui-cancel-task, ui-command, and ui-task, bumping the protocol to 1.3.0.
• The development runner and runner arguments now carry a stable session_id, which is useful for per-session tracing across local and cloud-like flows.

Breaking migrations

• Deprecated service-specific context classes, transport params, RTVI shims, frame aliases, and interruption/VAD helpers were removed across the stack.
• Turn detection and mute behavior moved toward LLMUserAggregator strategies instead of transport-level configuration.
• Some legacy providers and helpers were removed entirely (OpenPipeLLMService, TTSService.say(), FrameProcessor.wait_for_task(), older beta/alias modules).

Release Highlights (1.3.0 -> 1.5.0)

• Workers and multi-agent pipelines (1.3.0): PipelineTask / PipelineRunner are renamed toward PipelineWorker / WorkerRunner, and pipecat.workers makes pipelines peers on a typed-message bus for @job dispatch, handoffs, sidecars, UI workers, and distributed Redis/PGMQ patterns.
• UIWorker and RTVI UI protocol (1.3.0): UIWorker can observe client accessibility snapshots and drive UI commands over RTVI; the UI worker vocabulary moves from task/agent to job/worker (ui-task -> ui-job-group, cancelUITask -> cancelUIJobGroup).
• Development runner (1.3.0): one runner can serve WebRTC, Daily, telephony, and plain WebSocket clients; /start accepts a transport field, /ws-client supports protobuf WebSocket clients, /status reports enabled transports, and the Daily redirect moved to /daily.
• Service surface (1.3.0): adds Vonage Video Connector transport, Inception Mercury 2 LLM service, Cartesia turn-based STT, Rime coda TTS defaults, Soniox endpoint-delay settings, LLMService.append_system_instruction(), and STTService.supports_ttfs. Optional service/transport extras now raise ImportError when their dependency is missing, and transformers is no longer a base dependency.
• Behavioral eval framework (1.4.0): a testing framework for evaluating bot behavior, plus on_user_turn_message_added event handlers and realtime LLM-service metadata frames.
• Pipecat Flows in core (1.5.0): Pipecat Flows is integrated into the main package, so structured conversation flows no longer require a separate install. New services (1.5.0): Together AI STT/TTS services and NVIDIA per-sentence synthesis, with TTFA (time-to-first-audio) metrics for latency tracking.