voicebox-voice-mcp-agent

# Voicebox Voice MCP Agent > Skill by [ara.so](https://ara.so) — MCP Skills collection. ## Overview Voicebox is a local-first AI voice studio that combines voice input (Whisper STT + global dictation) and voice output (7 TTS engines with voice cloning) in a single TypeScript/Rust application. All inference runs on your hardware by default; audio never leaves your machine unless explicitly configured. It provides: - **Seven TTS engines**: Qwen3-TTS, Qwen CustomVoice, LuxTTS, Chatterbox Multilingual/Turbo, HumeAI TADA, Kokoro - **Voice cloning**: Zero-shot cloning from short reference samples - **Global dictation**: System-wide hotkey with paste-into-focused-field - **MCP HTTP server**: Agent-driven voice workflows at `/mcp` - **23 languages**: Multilingual synthesis - **Audio effects**: Pitch, reverb, delay, chorus, compression, filters - **Local LLM refinement**: Bundled Qwen3 for transcript cleanup **Stack**: Bun monorepo, TypeScript, Tauri 2 (Rust), React 18, Zustand, TanStack Router, Vite ## Installation ### Prerequisites ```bash # Install Bun (JavaScript runtime + package manager) curl -fsSL https://bun.sh/install | bash # Install Rust (for Tauri desktop builds) curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh # Install FFmpeg (audio processing) # macOS: brew install ffmpeg # Ubuntu/Debian: sudo apt install ffmpeg # Windows: download from ffmpeg.org ``` ### Quick Setup ```bash # Clone repository git clone https://github.com/jamiepine/voicebox.git cd voicebox # Install all dependencies just setup # Start development mode (web UI) just dev ``` ### Development Modes ```bash # Web UI only (http://127.0.0.1:5173) bun run dev:web # Tauri desktop development bun run dev # Build for production bun run build # Desktop release build just build:desktop ``` ## Configuration ### Environment Variables Create `.env` in project root: ```bash # API Configuration VOICEBOX_HOST=127.0.0.1 VOICEBOX_PORT=17493 VOICEBOX_CORS_ORIGINS=http://localhost:3000,http://localhost:5173 LOG_LEVEL=info # HuggingFace (for gated models) HF_TOKEN=your_huggingface_token_here # Optional: Redis Persistence Cache (for web deployments) VOICEBOX_REDIS_ENABLED=false VOICEBOX_REDIS_URL=redis://127.0.0.1:6379 VOICEBOX_REDIS_KEY_PREFIX=voicebox: ``` ### MCP Client Configuration Create `.mcp.json` to configure MCP client access: ```json { "mcpServers": { "voicebox": { "url": "http://127.0.0.1:17493/mcp", "description": "Voicebox voice synthesis and dictation server" } } } ``` ## API Usage ### REST API The service exposes a REST API at `http://127.0.0.1:17493` when running. #### Generate Speech ```typescript // TypeScript example: Generate speech interface GenerationRequest { text: string; profileId: string; engine?: string; // optional, uses profile default effects?: AudioEffect[]; } interface AudioEffect { type: 'pitch' | 'reverb' | 'delay' | 'chorus' | 'compression'; params: Record<string, number>; } async function generateSpeech(text: string, profileId: string): Promise<Blob> { const response = await fetch('http://127.0.0.1:17493/generations', { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ text, profileId, effects: [ { type: 'pitch', params: { semitones: 2 } }, { type: 'reverb', params: { roomSize: 0.5, damping: 0.5 } } ] }) }); return await response.blob(); } // Usage const audioBlob = await generateSpeech( "Hello, this is a test of voice synthesis.", "default-profile-id" ); // Play audio const audioUrl = URL.createObjectURL(audioBlob); const audio = new Audio(audioUrl); audio.play(); ``` #### Stream Generation Progress ```typescript // Server-Sent Events for progress tracking async function streamGeneration(text: string, profileId: string) { const response = await fetch('http://127.0.0.1:17493/generations/stream', { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ text, profileId }) }); const reader = response.body!.getReader(); const decoder = new TextDecoder(); while (true) { const { done, value } = await reader.read(); if (done) break; const chunk = decoder.decode(value); const lines = chunk.split('\n'); for (const line of lines) { if (line.startsWith('data: ')) { const data = JSON.parse(line.slice(6)); console.log('Progress:', data.progress, '%'); console.log('Status:', data.status); } } } } ``` #### Voice Cloning ```typescript // Clone voice from reference audio async function cloneVoice( name: string, referenceAudio: File, targetEngine: string = 'qwen-customvoice' ): Promise<string> { const formData = new FormData(); formData.append('name', name); formData.append('reference', referenceAudio); formData.append('engine', targetEngine); const response = await fetch('http://127.0.0.1:17493/voices/clone', { method: 'POST', body: formData }); const { voiceId } = await response.json(); return voiceId; } // Usage const fileInput = document.querySelector('input[type="file"]') as HTMLInputElement; const audioFile = fileInput.files![0]; const voiceId = await cloneVoice('My Custom Voice', audioFile); console.log('Voice cloned with ID:', voiceId); ``` ### Service Layer (Internal) When extending Voicebox or adding features: ```typescript // app/src/services/tts/types.ts export interface TTSEngine { name: string; synthesize(text: string, options: SynthesisOptions): Promise<AudioBuffer>; clone?(reference: AudioBuffer): Promise<VoiceModel>; supportedLanguages: string[]; } export interface SynthesisOptions { voiceId?: string; language?: string; speed?: number; pitch?: number; } // Example: Implementing a custom TTS engine import type { TTSEngine, SynthesisOptions } from './types'; export class CustomTTSEngine implements TTSEngine { name = 'custom-tts'; supportedLanguages = ['en', 'es', 'fr']; async synthesize(text: string, options: SynthesisOptions): Promise<AudioBuffer> { // Call your TTS model here const response = await fetch('http://your-tts-service/synthesize', { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ text, voice_id: options.voiceId, language: options.language || 'en', speed: options.speed || 1.0, pitch: options.pitch || 0 }) }); const arrayBuffer = await response.arrayBuffer(); const audioContext = new AudioContext(); return await audioContext.decodeAudioData(arrayBuffer); } async clone(reference: AudioBuffer): Promise<VoiceModel> { // Implement voice cloning logic throw new Error('Cloning not implemented for this engine'); } } // Register engine // app/src/services/tts/registry.ts import { CustomTTSEngine } from './engines/custom'; export const ttsEngines: TTSEngine[] = [ new CustomTTSEngine(), // ... other engines ]; ``` ## Platform Adapters Voicebox uses platform adapters to abstract native capabilities: ```typescript // app/src/platform/types.ts export interface Platform { // Filesystem readFile(path: string): Promise<Uint8Array>; writeFile(path: string, data: Uint8Array): Promise<void>; // Audio capture startAudioCapture(): Promise<MediaStream>; stopAudioCapture(): void; // Hotkeys (desktop only) registerHotkey?(combo: string, callback: () => void): Promise<void>; unregisterHotkey?(combo: string): Promise<void>; // Updates (desktop only) checkForUpdates?(): Promise<UpdateInfo | null>; installUpdate?(): Promise<void>; } // Desktop implementation (Tauri) // tauri/src/platform/desktop.ts import { invoke } from '@tauri-apps/api/core'; import { register, unregister } from '@tauri-apps/plugin-global-shortcut'; export const desktopPlatform: Platform = { async readFile(path: string): Promise<Uint8Array> { return await invoke('read_file', { path }); }, async writeFile(path: string, data: Uint8Array): Promise<void> { await invoke('write_file', { path, data: Array.from(data) }); }, async startAudioCapture(): Promise<MediaStream> { return await navigator.mediaDevices.getUserMedia({ audio: true }); }, stopAudioCapture(): void { // Stop all tracks }, async registerHotkey(combo: string, callback: () => void): Promise<void> { await register(combo, callback); }, async unregisterHotkey(combo: string): Promise<void> { await unregister(combo); } }; ``` ## MCP Integration ### MCP Server Endpoints The MCP server runs at `http://127.0.0.1:17493/mcp` and exposes tools for agent-driven workflows: ```typescript // Example: MCP tool definitions { "tools": [ { "name": "synthesize_speech", "description": "Generate speech audio from text using a specific voice profile", "inputSchema": { "type": "object", "properties": { "text": { "type": "string", "description": "Text to synthesize" }, "profile_id": { "type": "string", "description": "Voice profile ID" }, "engine": { "type": "string", "description": "Optional TTS engine override" } }, "required": ["text", "profile_id"] } }, { "name": "clone_voice", "description": "Clone a voice from reference audio", "inputSchema": { "type": "object", "properties": { "name": { "type": "string", "description": "Name for the cloned voice" }, "reference_path": { "type": "string", "description": "Path to reference audio" }, "engine": { "type": "string", "description": "TTS engine to use for cloning" } }, "required": ["name", "reference_path"] } }, { "name": "start_dictation", "description": "Start voice dictation and paste into focused field", "inputSchema": { "type": "object", "properties": { "language": { "type": "string", "description": "Language code (e.g., 'en', 'es')" } } } } ] } ``` ### MCP Client Usage ```typescript // Example: Calling MCP tools from an agent interface MCPRequest { tool: string; arguments: Record<string, unknown>; } async function callMCPTool(tool: string, args: Record<string, unknown>) { const response = await fetch('http://127.0.0.1:17493/mcp/execute', { method: 'POST', headers: { 'Content-Type': 'application/json' }, body: JSON.stringify({ tool, arguments: args }) }); return await response.json(); } // Generate speech via MCP const result = await callMCPTool('synthesize_speech', { text: 'Hello from the MCP agent!', profile_id: 'default-voice', engine: 'qwen3-tts' }); console.log('Audio URL:', result.audioUrl); ```