ElevenLabs Core Workflow B — Speech-to-Speech, Sound Effects & Audio Isolation
Overview
Secondary ElevenLabs workflows beyond TTS: (1) Speech-to-Speech voice conversion, (2) Sound Effects generation from text descriptions, (3) Audio Isolation for noise removal, and (4) Speech-to-Text transcription.
Prerequisites
- Completed
elevenlabs-install-auth setup
- For STS: source audio file in MP3/WAV/M4A format
- For audio isolation: noisy audio file to clean
Instructions
Step 1: Speech-to-Speech (Voice Changer)
Transform audio from one voice to another using POST /v1/speech-to-speech/{voice_id}:
import { ElevenLabsClient } from "@elevenlabs/elevenlabs-js";
import { createReadStream, createWriteStream } from "fs";
import { Readable } from "stream";
import { pipeline } from "stream/promises";
const client = new ElevenLabsClient();
async function speechToSpeech(
sourceAudioPath: string,
targetVoiceId: string,
outputPath: string
) {
const audio = await client.speechToSpeech.convert(targetVoiceId, {
audio: createReadStream(sourceAudioPath),
model_id: "eleven_english_sts_v2",
voice_settings: JSON.stringify({
stability: 0.5,
similarity_boost: 0.8,
style: 0.0,
}),
remove_background_noise: true,
});
await pipeline(Readable.fromWeb(audio as any), createWriteStream(outputPath));
console.log(`Voice-converted audio saved to ${outputPath}`);
}
await speechToSpeech(
"my_recording.mp3",
"21m00Tcm4TlvDq8ikWAM",
"converted.mp3"
);
cURL equivalent:
curl -X POST "https://api.elevenlabs.io/v1/speech-to-speech/21m00Tcm4TlvDq8ikWAM" \
-H "xi-api-key: ${ELEVENLABS_API_KEY}" \
-F "audio=@my_recording.mp3" \
-F "model_id=eleven_english_sts_v2" \
-F 'voice_settings={"stability":0.5,"similarity_boost":0.8}' \
-F "remove_background_noise=true" \
--output converted.mp3
Step 2: Sound Effects Generation
Generate cinematic sound effects from text descriptions using POST /v1/sound-generation:
async function generateSoundEffect(
description: string,
outputPath: string,
options?: {
duration?: number; // 0.5-30 seconds (null = auto)
promptInfluence?: number; // 0-1 (default 0.3, higher = follows prompt more closely)
loop?: boolean; // Seamless looping (default false)
}
) {
const audio = await client.textToSoundEffects.convert({
text: description,
duration_seconds: options?.duration,
prompt_influence: options?.promptInfluence ?? 0.3,
});
await pipeline(Readable.fromWeb(audio as any), createWriteStream(outputPath));
console.log(`Sound effect saved to ${outputPath}`);
}
await generateSoundEffect(
"Heavy rain on a tin roof with distant thunder",
"rain.mp3",
{ : , : }
);
(
,
,
{ : , : }
);
(
,
,
{ : , : }
);
cURL equivalent:
curl -X POST "https://api.elevenlabs.io/v1/sound-generation" \
-H "xi-api-key: ${ELEVENLABS_API_KEY}" \
-H "Content-Type: application/json" \
-d '{
"text": "Heavy rain on a tin roof with distant thunder",
"duration_seconds": 10,
"prompt_influence": 0.6
}' \
--output rain.mp3
Step 3: Audio Isolation (Voice Isolator)
Remove background noise from audio using POST /v1/audio-isolation:
async function isolateVoice(
noisyAudioPath: string,
cleanOutputPath: string
) {
const cleanAudio = await client.audioIsolation.audioIsolation({
audio: createReadStream(noisyAudioPath),
});
await pipeline(
Readable.fromWeb(cleanAudio as any),
createWriteStream(cleanOutputPath)
);
console.log(`Clean audio saved to ${cleanOutputPath}`);
}
await isolateVoice("noisy_interview.mp3", "clean_interview.mp3");
Streaming variant for large files (POST /v1/audio-isolation/stream):
async function isolateVoiceStreaming(
noisyAudioPath: string,
cleanOutputPath: string
) {
const stream = await client.audioIsolation.audioIsolationStream({
audio: createReadStream(noisyAudioPath),
});
const writer = createWriteStream(cleanOutputPath);
for await (const chunk of stream) {
writer.write(chunk);
}
writer.end();
}
cURL equivalent:
curl -X POST "https://api.elevenlabs.io/v1/audio-isolation" \
-H "xi-api-key: ${ELEVENLABS_API_KEY}" \
-F "audio=@noisy_interview.mp3" \
--output clean_interview.mp3
Step 4: Speech-to-Text (Transcription)
Transcribe audio with speaker diarization using POST /v1/speech-to-text:
async function transcribeAudio(audioPath: string) {
const result = await client.speechToText.convert({
audio: createReadStream(audioPath),
model_id: "scribe_v1",
});
console.log("Transcription:", result.text);
if (result.words) {
for (const word of result.words) {
console.log(`[${word.start.toFixed(2)}-${word.end.toFixed(2)}] ${word.text}`);
}
}
return result;
}
await transcribeAudio("podcast_episode.mp3");
API Endpoint Summary
| Feature | Method | Endpoint | Billing |
|---|
| Speech-to-Speech | POST | /v1/speech-to-speech/{voice_id} | Per character |
| Sound Effects | POST | /v1/sound-generation | Per generation |
| Audio Isolation | POST | /v1/audio-isolation | 1,000 chars/min of audio |
| Audio Isolation Stream | POST | /v1/audio-isolation/stream | 1,000 chars/min of audio |
| Speech-to-Text | POST | /v1/speech-to-text | Per audio minute |
Sound Effect Tips
- Be specific: "wooden door creaking slowly open in a quiet room" beats "door sound"
- Specify quantity: "three quick gunshots" vs "gunshots"
- Set mood: "eerie", "cheerful", "aggressive" changes the output character
- Use
prompt_influence: 0.6-0.8 for precise results, 0.2-0.4 for creative variation
- Max duration: 30 seconds per generation
Audio Isolation Limits
| Aspect | Limit |
|---|
| Max file size | 500 MB |
| Max duration | 1 hour |
| Supported formats | MP3, WAV, M4A, FLAC, OGG, WEBM |
| PCM optimization | Use file_format: "pcm_s16le_16" for lowest latency |
Error Handling
| Error | HTTP | Cause | Solution |
|---|
model_can_not_do_voice_conversion | 400 | Wrong model for STS | Use eleven_english_sts_v2 |
audio_too_short | 400 | STS input under 1 second | Use longer audio clip |
audio_too_long | 400 | STS input over limit | Trim to under 5 minutes |
invalid_sound_prompt | 400 | Nonsensical SFX description | Write descriptive, specific prompts |
file_too_large | 413 | Audio isolation over 500MB | Compress or split the file |
quota_exceeded | 401 | Character/generation limit hit | Check usage dashboard |
Resources
Next Steps
For common errors, see elevenlabs-common-errors. For SDK patterns, see elevenlabs-sdk-patterns.
