Expert in voice synthesis, speech processing, and vocal production using ElevenLabs and professional audio techniques. Specializes in TTS, voice cloning, podcast production, and voice UI design.
Game audio middleware (Wwise, FMOD) → sound-engineer
Music composition/production → DAW tools
Live concert/event audio → specialized domain
MCP Integrations
MCP Tool
Purpose
text_to_speech
Generate speech from text with voice selection
speech_to_speech
Transform voice recordings to different voices
voice_clone
Create instant voice clones from audio samples
search_voices
Find voices in ElevenLabs library
speech_to_text
Transcribe audio with speaker diarization
isolate_audio
Separate voice from background noise
create_agent
Build conversational AI agents with voice
Expert vs Novice Shibboleths
Topic
Novice
Expert
TTS quality
"Any voice works"
Matches voice to brand; considers emotion, pace, style
Voice cloning
"Upload any audio"
Knows 30s-3min of clean, varied speech needed; single speaker
Loudness
"Make it loud"
Targets -16 to -19 LUFS for podcasts; -14 for streaming
De-essing
"Doesn't matter"
Knows sibilance lives at 5-8kHz; frequency-selective compression
Compression
"Squash it"
Uses 3:1-4:1 for dialogue; slow attack (10-20ms) to preserve transients
High-pass
"Never use it"
Always HPF at 80-100Hz for voice; removes rumble, plosives
True peak
"Peak is peak"
Knows intersample peaks exceed 0dBFS; targets -1 dBTP
ElevenLabs models
"Use default"
eleven_multilingual_v2 for quality; eleven_flash_v2_5 for speed
Common Anti-Patterns
Anti-Pattern: Uploading Noisy Audio for Voice Cloning
What it looks like: Voice clone from phone recording with background noise, echo
Why it's wrong: Clone learns the noise; output has artifacts
What to do instead: Use isolate_audio first; record in quiet space; provide 1-3 min of varied speech
Anti-Pattern: Ignoring Loudness Standards
What it looks like: Podcast at -6 LUFS, then normalized by platform → crushed dynamics
Why it's wrong: Each platform normalizes differently; too loud = distortion, too quiet = inaudible
What to do instead: Master to -16 LUFS for podcasts; -14 LUFS for streaming; always check true peak < -1 dBTP
Anti-Pattern: TTS Without Voice Matching
What it looks like: Using default robotic voice for premium product
Why it's wrong: Voice IS brand; wrong voice = wrong emotional connection
What to do instead: search_voices to find matching tone; consider custom clone for brand consistency
Anti-Pattern: No De-essing on Processed Voice
What it looks like: "SSSSibilant" speech after compression and EQ boost
Why it's wrong: Compression brings up sibilance; EQ boost at 3-5kHz makes it worse
What to do instead: De-ess at 5-8kHz before compression; use frequency-selective compression
Anti-Pattern: Single Take, No Editing
What it looks like: Podcast with 20 "ums", breath sounds, long pauses
Why it's wrong: Listeners fatigue; unprofessional; reduces engagement
What to do instead: Edit out filler words; gate or manually cut breaths; tighten pacing
Evolution Timeline
Pre-2020: Robotic TTS
Concatenative synthesis (spliced recordings)
Obvious robotic quality
Limited voice options
2020-2022: Neural TTS Emerges
Tacotron, WaveNet improve naturalness
Still detectable as synthetic
Voice cloning requires hours of data
2023-2024: AI Voice Revolution
ElevenLabs instant voice cloning (30 seconds)
Near-human quality in TTS
Real-time voice transformation
Voice agents for customer service
2025+: Current Best Practices
Emotional TTS (control tone, pace, emotion)
Cross-lingual voice cloning
Real-time voice transformation in apps
Personalized voice agents
Voice authentication integration
Core Concepts
ElevenLabs Voice Selection
Model comparison:
Model
Quality
Latency
Languages
Use Case
eleven_multilingual_v2
Best
Higher
29
Production, quality-critical
eleven_flash_v2_5
Good
Lowest
32
Real-time, voice UI
eleven_turbo_v2_5
Better
Low
32
Balanced
Voice parameters:
# Stability: 0-1 (lower = more expressive, higher = more consistent)# Similarity boost: 0-1 (higher = closer to original voice)# Style: 0-1 (higher = more exaggerated style)# For natural speech:
stability = 0.5# Balanced expression
similarity = 0.75# Close to voice but natural
style = 0.0# Neutral (increase for dramatic)
Voice Cloning Best Practices
Audio requirements:
Duration: 1-3 minutes (more = better, diminishing returns after 3min)
True Peak = Maximum level including intersample peaks
Always measure with K-weighting (ITU-R BS.1770)
Conversational AI Agents
ElevenLabs agent configuration:
create_agent(
name="Support Agent",
first_message="Hi, how can I help you today?",
system_prompt="You are a helpful customer support agent...",
voice_id="your_voice_id",
language="en",
llm="gemini-2.0-flash-001", # Fast for conversation
temperature=0.5,
asr_quality="high", # Speech recognition quality
turn_timeout=7, # Seconds before agent responds
max_duration_seconds=300# 5 minute call limit
)
Voice UI considerations:
Use fast model (eleven_flash_v2_5) for real-time
Keep responses concise (< 30 seconds)
Add pauses for natural conversation flow
Handle interruptions gracefully
Quick Reference
Voice Selection Decision Tree
Brand/professional content? → Custom clone or curated voice
Real-time/interactive? → eleven_flash_v2_5 model
Quality-critical? → eleven_multilingual_v2 model
Multiple languages? → Check language support per voice
Most ASR models trained on fluent speech. Disfluencies cause:
Word boundary detection errors
Repetitions transcribed literally ("I I I want" vs "I want")
Collapsed syllables missed entirely
Timing models confused by irregular pace
Solutions & Workarounds
1. Model selection (best to worst for disfluencies):
Whisper large-v3 - Most robust to disfluencies
ElevenLabs speech_to_text - Good with varied speech
Google Speech-to-Text - Decent with enhanced models
Fast/lightweight models - Usually worst
2. Pre-processing:
# Normalize speech rate before ASR# Use librosa to stretch irregular segments toward target rateimport librosa
y, sr = librosa.load("disfluent.wav")
y_stretched = librosa.effects.time_stretch(y, rate=0.9) # Slow down
3. Post-processing:
Remove duplicate words: "I I I want" → "I want"
Filter common fillers: "um", "uh", "like", "you know"
Use LLM to clean transcripts while preserving meaning
4. Fine-tuning Whisper (advanced):
# Fine-tune on disfluent speech dataset# Datasets: FluencyBank, UCLASS, SEP-28k (stuttering)from transformers import WhisperForConditionalGeneration, WhisperProcessor
model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-large-v3")
# Fine-tune on your speech samples with corrected transcripts# Training loop with disfluent audio → fluent transcript pairs
5. ElevenLabs voice cloning approach:
Clone your voice from fluent segments
Use TTS for fluent output with your voice
Great for pre-recorded content, not live
Accessibility Considerations
Always provide manual transcript correction option
Consider hybrid: ASR + human review
For voice UI: longer timeout, confirmation prompts
Test with actual users from target population
Performance Targets
Operation
Typical Time
TTS (100 words)
2-5 seconds
Voice clone creation
10-30 seconds
Speech-to-speech
3-8 seconds
Transcription (1 min audio)
5-15 seconds
Audio isolation
5-20 seconds
Integrates With
sound-engineer - For spatial audio, game audio, procedural SFX
native-app-designer - Voice UI implementation in apps
vr-avatar-engineer - Avatar voice integration
For detailed implementations: See /references/implementations.md
Remember: Voice is intimate—it speaks directly to the listener's brain. Match voice to brand, process for clarity not loudness, and always respect the platform's loudness standards. With ElevenLabs, you have instant access to professional voice synthesis; use it thoughtfully.
