Skip to main content Home Creators foolhardy45 portfolio voice-ai-development
voice-ai-development You are an expert in building real-time voice applications. You think in terms of latency budgets, audio quality, and user experience. You know that voice apps feel magical when fast and broken when slow.
Jump to install Skills Marketplace Discover and explore AI skills built by the community.
Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.
Show prompt details A direct command skips the review prompt. Inspect the source before running it.
npx skills add https://github.com/foolhardy45/portfolio --skill voice-ai-developmentThe command stays on one line. Scroll horizontally to inspect it before copying.
Prefer a local copy? Download the files currently available to SkillsMP.
Download Zip Downloading... More from this repository
Related occupations SOC
Based on SOC occupation classification
name voice-ai-development description You are an expert in building real-time voice applications. You think in terms of latency budgets, audio quality, and user experience. You know that voice apps feel magical when fast and broken when slow. risk unknown source vibeship-spawner-skills (Apache 2.0) date_added 2026-02-27
Voice AI Development
Role : Voice AI Architect
You are an expert in building real-time voice applications. You think in terms of
latency budgets, audio quality, and user experience. You know that voice apps feel
magical when fast and broken when slow. You choose the right combination of providers
for each use case and optimize relentlessly for perceived responsiveness.
Capabilities
OpenAI Realtime API
Vapi voice agents
Deepgram STT/TTS
ElevenLabs voice synthesis
LiveKit real-time infrastructure
WebRTC audio handling
Voice agent design
Latency optimization
Requirements
Python or Node.js
API keys for providers
Audio handling knowledge
Patterns
OpenAI Realtime API
Native voice-to-voice with GPT-4o
When to use : When you want integrated voice AI without separate STT/TTS
import asyncio
import websockets
import json
import base64
OPENAI_API_KEY = "sk-..."
async def voice_session ():
url = "wss://api.openai.com/v1/realtime?model=gpt-4o-realtime-preview"
headers = {
"Authorization" : f"Bearer {OPENAI_API_KEY} " ,
"OpenAI-Beta" : "realtime=v1"
}
async with websockets.connect(url, extra_headers=headers) as ws:
await ws.send(json.dumps({
"type" : "session.update" ,
"session" : {
"modalities" : ["text" , "audio" ],
"voice" : "alloy" ,
"input_audio_format" : "pcm16" ,
"output_audio_format" : "pcm16" ,
"input_audio_transcription" : {
"model" : "whisper-1"
},
"turn_detection" : {
"type" : "server_vad" ,
"threshold" : 0.5 ,
"prefix_padding_ms" : 300 ,
"silence_duration_ms" : 500
},
"tools" : [
{
"type" : "function" ,
"name" : "get_weather" ,
"description" : "Get weather for a location" ,
"parameters" : {
"type" : "object" ,
"properties" : {
"location" : {"type" : "string" }
}
}
}
]
}
}))
async def send_audio (audio_bytes ):
await ws.send(json.dumps({
"type" : "input_audio_buffer.append" ,
"audio" : base64.b64encode(audio_bytes).decode()
}))
async for message in ws:
event = json.loads(message)
if event["type" ] == "resp
Vapi Voice Agent Build voice agents with Vapi platform
When to use : Phone-based agents, quick deployment
from flask import Flask, request, jsonify
import vapi
app = Flask(__name__)
client = vapi.Vapi(api_key="..." )
assistant = client.assistants.create(
name="Support Agent" ,
model={
"provider" : "openai" ,
"model" : "gpt-4o" ,
"messages" : [
{
"role" : "system" ,
"content" : "You are a helpful support agent..."
}
]
},
voice={
"provider" : "11labs" ,
"voiceId" : "21m00Tcm4TlvDq8ikWAM"
},
firstMessage="Hi! How can I help you today?" ,
transcriber={
"provider" : "deepgram" ,
"model" : "nova-2"
}
)
@app.route("/vapi/webhook" , methods=["POST" ]
def vapi_webhook ():
event = request.json
if event["type" ] == "function-call" :
name = event["functionCall" ]["name" ]
args = event["functionCall" ]["parameters" ]
if name == "check_order" :
result = check_order(args["order_id" ])
return jsonify({"result" : result})
elif event["type" ] == "end-of-call-report" :
transcript = event["transcript" ]
save_transcript(event["call" ]["id" ], transcript)
return jsonify({"ok" : True })
call = client.calls.create(
assistant_id=assistant.id ,
customer={
"number" : "+1234567890"
},
phoneNumber={
"twilioPhoneNumber" : "+0987654321"
}
)
web_call = client.calls.create(
assistant_id=assistant.id ,
type ="web"
)
Deepgram STT + ElevenLabs TTS Best-in-class transcription and synthesis
When to use : High quality voice, custom pipeline
import asyncio
from deepgram import DeepgramClient, LiveTranscriptionEvents
from elevenlabs import ElevenLabs
deepgram = DeepgramClient(api_key="..." )
async def transcribe_stream (audio_stream ):
connection = deepgram.listen.live.v("1" )
async def on_transcript (result ):
transcript = result.channel.alternatives[0 ].transcript
if transcript:
print (f"Heard: {transcript} " )
if result.is_final:
await handle_user_input(transcript)
connection.on(LiveTranscriptionEvents.Transcript, on_transcript)
await connection.start({
"model" : "nova-2" ,
"language" : "en" ,
"smart_format" : True ,
"interim_results" : True ,
"utterance_end_ms" : 1000 ,
"vad_events" : True ,
"encoding" : "linear16" ,
"sample_rate" : 16000
})
async for chunk in audio_stream:
await connection.send(chunk)
await connection.finish()
eleven = ElevenLabs(api_key="..." )
def text_to_speech_stream (text: str ):
"""Stream TTS audio chunks."""
audio_stream = eleven.text_to_speech.convert_as_stream(
voice_id="21m00Tcm4TlvDq8ikWAM" ,
model_id="eleven_turbo_v2_5" ,
text=text,
output_format="pcm_24000"
)
for chunk in audio_stream:
yield chunk
async def tts_websocket (text_stream ):
async with eleven.text_to_speech.stream_async(
voice_id="21m00Tcm4TlvDq8ikWAM" ,
model_id="eleven_turbo_v2_5"
) as tts:
async for text_chunk in text_stream:
audio = await tts.send(text_chunk)
yield audio
final_audio = await tts.flush()
yield final_audio
Anti-Patterns
❌ Non-streaming Pipeline Why bad : Adds seconds of latency.
User perceives as slow.
Loses conversation flow.
Instead : Stream everything:
STT: interim results
LLM: token streaming
TTS: chunk streaming
Start TTS before LLM finishes.
❌ Ignoring Interruptions Why bad : Frustrating user experience.
Feels like talking to a machine.
Wastes time.
Instead : Implement barge-in detection.
Use VAD to detect user speech.
Stop TTS immediately.
Clear audio queue.
❌ Single Provider Lock-in Why bad : May not be best quality.
Single point of failure.
Harder to optimize.
Instead : Mix best providers:
Deepgram for STT (speed + accuracy)
ElevenLabs for TTS (voice quality)
OpenAI/Anthropic for LLM
Limitations
Latency varies by provider
Cost per minute adds up
Quality depends on network
Complex debugging
Related Skills Works well with: langgraph, structured-output, langfuse
When to Use This skill is applicable to execute the workflow or actions described in the overview.
