fal-audio

name

fal-audio

description

Complete fal.ai audio system. PROACTIVELY activate for: (1) Whisper speech-to-text, (2) Transcription with timestamps, (3) Translation to English, (4) F5-TTS voice cloning, (5) ElevenLabs premium TTS, (6) Kokoro multi-language TTS, (7) XTTS open-source cloning, (8) Subtitle generation (SRT), (9) Audio file formats. Provides: STT/TTS endpoints, language codes, voice cloning setup, timestamp formatting. Ensures accurate transcription and natural speech synthesis.

Quick Reference

STT Model	Endpoint	Speed	Accuracy
Whisper	fal-ai/whisper	Medium	Highest
Whisper Turbo	fal-ai/whisper-turbo	Fast	High
Whisper Large v3	fal-ai/whisper-large-v3	Slow	Highest

TTS Model	Endpoint	Voice Clone	Quality
F5-TTS	fal-ai/f5-tts	Yes	High
ElevenLabs	fal-ai/elevenlabs/tts	Via API	Highest
Kokoro	fal-ai/kokoro/american-english	No	Good
XTTS	fal-ai/xtts	Yes	Good

Whisper Task	Use Case
transcribe	Same language text
translate	Non-English → English

Whisper Parameter	Value
chunk_level	"segment" for timestamps
language	ISO code (e.g., "en")

When to Use This Skill

Use for audio processing:

• Transcribing audio/video to text
• Generating subtitles with timestamps
• Translating speech to English
• Cloning voices from reference audio
• Generating natural speech from text

Related skills:

• For video with audio: see fal-text-to-video
• For API integration: see fal-api-reference
• For model comparison: see fal-model-guide

---

fal.ai Audio Models

Complete reference for speech-to-text (STT) and text-to-speech (TTS) models on fal.ai.

Speech-to-Text Models

Whisper (OpenAI)

Endpoint: fal-ai/whisper Best For: Accurate transcription and translation

The industry-standard speech recognition model with support for 99+ languages.

import { fal } from "@fal-ai/client";

const result = await fal.subscribe("fal-ai/whisper", {
  input: {
    audio_url: "https://example.com/speech.mp3",
    task: "transcribe",
    language: "en",
    chunk_level: "segment"
  }
});

console.log(result.text);
console.log(result.chunks);  // With timestamps

import fal_client

result = fal_client.subscribe(
    "fal-ai/whisper",
    arguments={
        "audio_url": "https://example.com/speech.mp3",
        "task": "transcribe",
        "language": "en",
        "chunk_level": "segment"
    }
)
print(result["text"])
for chunk in result["chunks"]:
    print(f"[{chunk['timestamp'][0]:.2f}-{chunk['timestamp'][1]:.2f}] {chunk['text']}")

Whisper Parameters:

Parameter	Type	Values	Description
audio_url	string	-	Audio file URL
task	string	"transcribe", "translate"	Transcribe or translate to English
language	string	ISO code	Source language (optional, auto-detected)
chunk_level	string	"segment"	Return timestamps
version	string	"3"	Whisper version

Response Structure:

interface WhisperOutput {
  text: string;  // Full transcription
  chunks?: Array<{
    text: string;
    timestamp: [number, number];  // [start, end] in seconds
  }>;
}

Whisper Turbo

Endpoint: fal-ai/whisper-turbo Best For: Fast transcription

const result = await fal.subscribe("fal-ai/whisper-turbo", {
  input: {
    audio_url: "https://example.com/podcast.mp3",
    task: "transcribe"
  }
});

Whisper Large v3

Endpoint: fal-ai/whisper-large-v3 Best For: Maximum accuracy

const result = await fal.subscribe("fal-ai/whisper-large-v3", {
  input: {
    audio_url: "https://example.com/meeting.mp3",
    task: "transcribe",
    language: "en"
  }
});

Whisper Usage Examples

Transcription with Timestamps:

const result = await fal.subscribe("fal-ai/whisper", {
  input: {
    audio_url: audioUrl,
    task: "transcribe",
    chunk_level: "segment"
  }
});

// Format as SRT subtitles
result.chunks.forEach((chunk, i) => {
  const start = formatTime(chunk.timestamp[0]);
  const end = formatTime(chunk.timestamp[1]);
  console.log(`${i + 1}\n${start} --> ${end}\n${chunk.text}\n`);
});

function formatTime(seconds: number): string {
  const h = Math.floor(seconds / 3600);
  const m = Math.floor((seconds % 3600) / 60);
  const s = Math.floor(seconds % 60);
  const ms = Math.floor((seconds % ) * );
   ;
}

Translation (Non-English to English):

const result = await fal.subscribe("fal-ai/whisper", {
  input: {
    audio_url: "https://example.com/french-speech.mp3",
    task: "translate",  // Translates to English
    language: "fr"
  }
});

console.log(result.text);  // English translation

Multi-Language Detection:

// Whisper auto-detects language if not specified
const result = await fal.subscribe("fal-ai/whisper", {
  input: {
    audio_url: "https://example.com/unknown-language.mp3",
    task: "transcribe"
    // language omitted - auto-detect
  }
});

Text-to-Speech Models

F5-TTS

Endpoint: fal-ai/f5-tts Best For: Voice cloning from reference audio

const result = await fal.subscribe("fal-ai/f5-tts", {
  input: {
    gen_text: "Hello! Welcome to our product demonstration. We're excited to show you what we've built.",
    ref_audio_url: "https://example.com/voice-sample.wav",
    ref_text: "This is a sample of my voice for cloning purposes.",
    model_type: "F5-TTS"
  }
});

console.log(result.audio_url);

result = fal_client.subscribe(
    "fal-ai/f5-tts",
    arguments={
        "gen_text": "Hello! Welcome to our product.",
        "ref_audio_url": "https://example.com/voice-sample.wav",
        "ref_text": "This is a sample of my voice."
    }
)
print(result["audio_url"])

F5-TTS Parameters:

Parameter	Type	Description
gen_text	string	Text to synthesize
ref_audio_url	string	Reference voice audio URL
ref_text	string	Transcript of reference audio
model_type	string	"F5-TTS" or "E2-TTS"
remove_silence	boolean	Remove silence from output

ElevenLabs TTS

Endpoint: fal-ai/elevenlabs/tts Best For: Premium voice quality

const result = await fal.subscribe("fal-ai/elevenlabs/tts", {
  input: {
    text: "Welcome to fal.ai! Let me tell you about our amazing AI models.",
    voice_id: "21m00Tcm4TlvDq8ikWAM",  // ElevenLabs voice ID
    model_id: "eleven_multilingual_v2"
  }
});

console.log(result.audio.url);

ElevenLabs Parameters:

Parameter	Type	Description
text	string	Text to synthesize
voice_id	string	ElevenLabs voice ID
model_id	string	TTS model version
stability	number	Voice stability (0-1)
similarity_boost	number	Voice similarity (0-1)

ElevenLabs Voice IDs (examples):

• 21m00Tcm4TlvDq8ikWAM - Rachel (female)
• AZnzlk1XvdvUeBnXmlld - Domi (female)
• EXAVITQu4vr4xnSDxMaL - Bella (female)
• ErXwobaYiN019PkySvjV - Antoni (male)
• VR6AewLTigWG4xSOukaG - Arnold (male)

Kokoro TTS

Endpoint: fal-ai/kokoro/american-english Best For: Multi-language, natural sounding

const result = await fal.subscribe("fal-ai/kokoro/american-english", {
  input: {
    text: "This is a test of the Kokoro text-to-speech system.",
    voice: "af_bella"  // Voice style
  }
});

console.log(result.audio.url);

Kokoro Variants:

• fal-ai/kokoro/american-english - American English
• fal-ai/kokoro/british-english - British English
• fal-ai/kokoro/japanese - Japanese
• fal-ai/kokoro/mandarin - Mandarin Chinese

Kokoro Parameters:

Parameter	Type	Description
text	string	Text to synthesize
voice	string	Voice style identifier
speed	number	Speech speed multiplier

XTTS (Coqui)

Endpoint: fal-ai/xtts Best For: Open-source voice cloning

const result = await fal.subscribe("fal-ai/xtts", {
  input: {
    text: "Hello, this is a cloned voice speaking.",
    audio_url: "https://example.com/voice-reference.wav",
    language: "en"
  }
});

XTTS Parameters:

Parameter	Type	Description
text	string	Text to synthesize
audio_url	string	Reference audio for cloning
language	string	Target language

Model Comparison

Speech-to-Text

Model	Speed	Accuracy	Languages	Best For
Whisper	Medium	Highest	99+	Accuracy critical
Whisper Turbo	Fast	High	99+	Speed needed
Whisper Large v3	Slow	Highest	99+	Maximum quality

Text-to-Speech

Model	Quality	Voice Clone	Languages	Best For
F5-TTS	High	Yes	Multiple	Voice cloning
ElevenLabs	Highest	Via API	Many	Premium quality
Kokoro	Good	No	Multiple	Multi-language
XTTS	Good	Yes	16	Open-source

Workflow Examples

Transcribe and Translate Pipeline

async function processAudio(audioUrl: string, targetLanguage: string = 'en') {
  // 1. Transcribe
  const transcription = await fal.subscribe("fal-ai/whisper", {
    input: {
      audio_url: audioUrl,
      task: "transcribe",
      chunk_level: "segment"
    }
  });

  // 2. If not English, translate
  let translation = null;
  if (targetLanguage === 'en') {
    translation = await fal.subscribe("fal-ai/whisper", {
      input: {
        audio_url: audioUrl,
        task: "translate"
      }
    });
  }

  return {
    original: transcription.text,
    translated: translation?.text,
    chunks: transcription.chunks
  };
}

Voice Cloning Pipeline

async function cloneVoiceAndSpeak(
  referenceAudioUrl: string,
  referenceText: string,
  textToSpeak: string
) {
  // Use F5-TTS for voice cloning
  const result = await fal.subscribe("fal-ai/f5-tts", {
    input: {
      gen_text: textToSpeak,
      ref_audio_url: referenceAudioUrl,
      ref_text: referenceText,
      remove_silence: true
    }
  });

  return result.audio_url;
}

Subtitle Generation

async function generateSubtitles(videoUrl: string): Promise<string> {
  // Extract audio and transcribe
  const result = await fal.subscribe("fal-ai/whisper", {
    input: {
      audio_url: videoUrl,  // Works with video URLs too
      task: "transcribe",
      chunk_level: "segment"
    }
  });

  // Generate SRT format
  let srt = '';
  result.chunks.forEach((chunk, i) => {
    srt += `${i + 1}\n`;
    srt += `${formatSrtTime(chunk.timestamp[0])} --> ${formatSrtTime(chunk.timestamp[1])}\n`;
    srt += `${chunk.text}\n\n`;
  });

  return srt;
}

function formatSrtTime(seconds: number): string {
  const date = new Date(seconds * 1000);
  return date.toISOString().substr(11, ).(, );
}

Audio Book Generation

async function generateAudioBook(chapters: string[], voiceId: string) {
  const audioUrls = [];

  for (const chapter of chapters) {
    // Split into manageable chunks
    const chunks = splitText(chapter, 5000);

    for (const chunk of chunks) {
      const result = await fal.subscribe("fal-ai/elevenlabs/tts", {
        input: {
          text: chunk,
          voice_id: voiceId,
          model_id: "eleven_multilingual_v2"
        }
      });
      audioUrls.push(result.audio.url);
    }
  }

  return audioUrls;
}

function splitText(text: string, maxLength: number): string[] {
  const chunks = [];
  let current = '';

  text.split('. ').forEach(sentence => {
    if ((current + sentence).length < maxLength) {
      current += sentence + ;
    }  {
      chunks.(current.());
      current = sentence + ;
    }
  });

   (current) chunks.(current.());
   chunks;
}

Parameter Reference

Speech-to-Text Input

interface STTInput {
  audio_url: string;
  task?: "transcribe" | "translate";
  language?: string;  // ISO 639-1 code
  chunk_level?: "segment";
  version?: string;
}

Text-to-Speech Input

interface TTSInput {
  // Common
  text?: string;
  gen_text?: string;

  // Voice cloning
  ref_audio_url?: string;
  ref_text?: string;
  audio_url?: string;  // XTTS

  // Voice selection
  voice_id?: string;  // ElevenLabs
  voice?: string;     // Kokoro
  model_type?: string; // F5-TTS

  // Control
  speed?: number;
  stability?: number;
  similarity_boost?: number;
  language?: string;
  remove_silence?: boolean;
}

Best Practices

Speech-to-Text

1. Audio Quality: Clean audio = better transcription
2. Specify Language: Provide language hint when known
3. Use Timestamps: Request chunk_level: "segment" for subtitles
4. Handle Long Audio: Whisper handles long files automatically
5. Translation: Use task: "translate" for non-English to English

Text-to-Speech

1. Reference Quality: For voice cloning, use 10-30 second clear samples
2. Reference Transcript: Accurate transcript improves cloning quality
3. Text Length: Split very long text into chunks
4. Punctuation: Proper punctuation improves prosody
5. Emotion Hints: Use punctuation (!, ?) to convey emotion

Common Supported Languages

Language	Code	STT	TTS
English	en	Yes	Yes
Spanish	es	Yes	Yes
French	fr	Yes	Yes
German	de	Yes	Yes
Italian	it	Yes	Yes
Portuguese	pt	Yes	Yes
Japanese	ja	Yes	Yes
Chinese	zh	Yes	Yes
Korean	ko	Yes	Yes
Russian	ru	Yes	Limited

File Format Support

Input Formats (STT)

Format	Extension	Supported
MP3	.mp3	Yes
WAV	.wav	Yes
M4A	.m4a	Yes
FLAC	.flac	Yes
OGG	.ogg	Yes
WebM	.webm	Yes
Video	.mp4	Yes (audio extracted)

Output Formats (TTS)

Model	Output Format
F5-TTS	WAV
ElevenLabs	MP3
Kokoro	WAV
XTTS	WAV

Error Handling

try {
  const result = await fal.subscribe("fal-ai/whisper", {
    input: { audio_url: audioUrl, task: "transcribe" }
  });
} catch (error) {
  if (error.status === 400) {
    console.error("Invalid audio file or URL");
  } else if (error.status === 413) {
    console.error("Audio file too large");
  } else {
    console.error("Transcription failed:", error.message);
  }
}