phone-agent

name	phone-agent
description	Run a real-time AI phone agent using Twilio, Deepgram, and ElevenLabs. Handles incoming calls, transcribes audio, generates responses via LLM, and speaks back via streaming TTS. Use when user wants to: (1) Test voice AI capabilities, (2) Handle phone calls programmatically, (3) Build a conversational voice bot.

Phone Agent Skill

Runs a local FastAPI server that acts as a real-time voice bridge.

Architecture

Twilio (Phone) <--> WebSocket (Audio) <--> [Local Server] <--> Deepgram (STT)
                                                  |
                                                  +--> OpenAI (LLM)
                                                  +--> ElevenLabs (TTS)

Prerequisites

1. Twilio Account: Phone number + TwiML App.
2. Deepgram API Key: For fast speech-to-text.
3. OpenAI API Key: For the conversation logic.
4. ElevenLabs API Key: For realistic text-to-speech.
5. Ngrok (or similar): To expose your local port 8080 to Twilio.

Setup

1. Install Dependencies:

   pip install -r scripts/requirements.txt
   

2. Set Environment Variables (in ~/.moltbot/.env, ~/.clawdbot/.env, or export):

   export DEEPGRAM_API_KEY="your_key"
   export OPENAI_API_KEY="your_key"
   export ELEVENLABS_API_KEY="your_key"
   export TWILIO_ACCOUNT_SID=
    TWILIO_AUTH_TOKEN=
    PORT=8080