| name | livekit-prompt-builder |
| description | Guide for creating effective prompts and instructions for LiveKit voice agents. Use when building conversational AI agents with the LiveKit Agents framework, including (1) Creating new voice agent prompts from scratch, (2) Improving existing agent instructions, (3) Optimizing prompts for text-to-speech output, (4) Integrating tool/function calling capabilities, (5) Building multi-agent systems with handoffs, (6) Ensuring voice-friendly formatting and brevity for natural conversations, (7) Iteratively improving prompts based on testing and feedback, (8) Building industry-specific agents (debt collection, healthcare, banking, customer service, front desk). |
LiveKit Voice Agent Prompt Builder
Create production-ready prompts for LiveKit voice agents that work seamlessly with text-to-speech systems.
Quick Start
For most use cases, follow this simple workflow:
- Choose a template from
assets/templates/ that matches your use case
- Customize the template with your specific information
- Validate using
scripts/validate_prompt.py
- Test with your LiveKit agent
- Refine based on real conversations
Workflow
Step 1: Determine Your Agent Type
Select the template that best matches your needs:
General Purpose:
-
Basic Assistant (basic-assistant.txt) - Simple conversational agent
- Use for: General chat, demos, minimal functionality
- Complexity: Low
- Tools: Optional
-
Tool-Enabled Agent (tool-enabled-agent.txt) - Agent with function calling
- Use for: Weather, search, data lookup, single-domain assistants
- Complexity: Medium
- Tools: 1-5 tools
-
Customer Service (customer-service.txt) - Support agent with escalation
- Use for: Help desks, support lines, troubleshooting
- Complexity: Medium
- Tools: Lookup, search, escalation
-
Specialist Agent (specialist-agent.txt) - Task-specific agent (reservations, orders, etc.)
- Use for: Booking systems, order taking, appointments
- Complexity: Medium-High
- Tools: Action tools (create, update)
-
Multi-Agent Coordinator (multi-agent-coordinator.txt) - Routes to other agents
- Use for: Main entry point in multi-agent systems
- Complexity: Medium
- Tools: Transfer/handoff tools
Industry-Specific:
Step 2: Customize Your Template
Open your selected template and fill in the placeholders:
Example - Customizing basic-assistant.txt:
Template:
You are [NAME], a friendly voice assistant. You are [TRAIT_1] and [TRAIT_2], with [TRAIT_3].
Customized:
You are Casey, a friendly voice assistant. You are helpful and patient, with a sense of humor.
Key customization areas:
- Identity (name, role, personality)
- Domain context (business info, hours, policies)
- Tool descriptions (when and how to use each tool)
- Workflow steps (information collection sequence)
- Boundaries (what agent can't do)
Step 3: Apply Voice Optimization
Ensure your prompt includes these critical voice optimization rules:
Required elements:
Keep your responses brief - 1-3 sentences. Respond in plain text only - no emojis, asterisks, markdown, or special formatting. Ask one question at a time.
For agents that mention numbers, prices, or contact info:
Spell out numbers ("fifteen" not "15"). Spell phone numbers digit by digit ("5-5-5, 1-2-3-4"). For URLs, omit "https://" and say "dot" for periods.
For agents with tools:
If a tool call fails, explain the issue simply and suggest a fallback. Summarize tool results in conversational language - don't recite technical IDs or raw data.
Step 4: Validate Your Prompt
Run the validation script to check for common issues:
python scripts/validate_prompt.py your_prompt.txt
The validator checks for:
- Missing identity statement
- Lack of voice optimization rules
- TTS antipatterns (emojis, markdown, special characters)
- Excessive length (>500 words)
- Missing tool error handling
- Number formatting guidance
Fix any errors before deploying your prompt.
Step 5: Test with LiveKit
Deploy your prompt in a LiveKit agent and test with real conversations:
Test scenarios:
- Happy path - User provides information smoothly
- Interruptions - User changes topic or interrupts
- Tool failures - Simulate tool errors
- Edge cases - Ambiguous requests, out-of-scope questions
- Voice quality - Listen to TTS output for formatting issues
Step 6: Refine
Based on testing, refine your prompt:
Common refinements:
- Add specific guidance for observed failure modes
- Adjust tone or personality descriptors
- Add examples for complex tool usage
- Tighten response length constraints if agent is too verbose
- Add boundaries for out-of-scope requests
Key Concepts
Voice-First Design
Voice agents differ from text chatbots:
Critical differences:
- No visual formatting (users can't see bold, lists, etc.)
- Limited user patience (long responses cause frustration)
- TTS limitations (special characters cause problems)
- No "scroll back" (users can't review previous responses)
Implications for prompts:
- Always specify plain text only
- Enforce brevity (1-3 sentences typical)
- Spell out numbers and abbreviations
- Ask one question at a time
- Avoid complex structures (lists, tables)
The Three Essential Sections
Every voice agent prompt should have:
- Identity - "You are [name/role]. You are [personality]."
- Voice Rules - Plain text, brief responses, number formatting
- Core Functionality - What the agent does, tools it uses, workflows it follows
Additional sections (domain context, guardrails, handoffs) add as needed.
Template vs. Custom Prompts
Use templates when:
- Starting a new agent
- Following common patterns (support, booking, etc.)
- Want quick setup with best practices
Write custom when:
- Unique use case not covered by templates
- Very specialized domain
- Complex multi-step workflows
Even with custom prompts, reference the templates for voice optimization patterns.
Progressive Disclosure: When to Load References
Load reference documentation based on your specific needs:
For all users:
- Start with templates and this SKILL.md
Load prompt-components.md when:
- Building complex prompts from scratch
- Need detailed guidance on specific sections
- Want to understand the "why" behind best practices
- Customizing beyond simple template fill-in
Load voice-optimization.md when:
- Encountering TTS formatting issues
- Agent responses sound awkward or unnatural
- Need deep understanding of voice-specific constraints
- Working with multiple TTS providers
- Debugging number, URL, or email pronunciation
Load tool-integration.md when:
- Integrating function calling/tools
- Building multi-agent systems with handoffs
- Need patterns for error handling or parameter collection
- Tools aren't being used correctly
- Working with RAG or external data sources
Load examples.md when:
- Want to see complete, real-world prompts
- Need inspiration for similar use cases
- Comparing complexity levels
- Learning from production patterns
Load industry-best-practices.md when:
- Building agents for regulated industries (healthcare, banking, debt collection)
- Need compliance guidance (HIPAA, FDCPA, TCPA, etc.)
- Working on industry-specific use cases (front desk, customer service, sales)
- Require specialized tone and approach patterns
- Need security or privacy considerations
Load iterative-improvement.md when:
- Improving existing prompts based on feedback
- Analyzing test results and conversation data
- Conducting A/B tests on prompt variations
- Establishing metrics and measurement frameworks
- Need systematic approach to prompt refinement
- Want to preserve prompt structure while making improvements
Strategy: Start minimal (templates + SKILL.md). Load references only when you hit specific challenges or need deeper guidance.
Multi-Agent Systems
For systems with multiple agents, design each agent separately:
Pattern:
-
Coordinator/Greeter - Uses multi-agent-coordinator.txt
- Simple routing logic
- Minimal conversation
- Quick transfers
-
Specialist Agents - Use specialist-agent.txt
- Each handles one specific task
- Collects required information
- Takes action or transfers to next agent
Key considerations:
- Each agent should have a single, clear responsibility
- Define handoff criteria explicitly
- Decide what context to pass between agents (full chat history vs structured data)
- Test agent transitions for smoothness
Common Patterns
Pattern 1: Simple Conversational Agent
Template: basic-assistant.txt
Use for: Chat, demos, general assistance
Prompt structure:
- Identity + personality
- Voice optimization rules
- Optional: specific topics or capabilities
Example use cases:
- Demo voice assistant
- Companion chatbot
- General information assistant
Pattern 2: Information Retrieval Agent
Template: tool-enabled-agent.txt
Use for: Looking up data, answering questions from external sources
Prompt structure:
- Identity + domain
- Voice optimization
- Tool descriptions (when to use each tool)
- Result summarization guidance
Example use cases:
- Weather agent
- Product lookup
- Knowledge base search
- RAG-enabled assistants
Pattern 3: Action-Taking Agent
Template: specialist-agent.txt
Use for: Booking, ordering, scheduling, transactions
Prompt structure:
- Identity + role
- Voice optimization
- Information collection sequence
- Confirmation before action
- Tool usage for state changes
Example use cases:
- Reservation agent
- Order taking
- Appointment scheduling
- Registration
Pattern 4: Support Agent with Escalation
Template: customer-service.txt
Use for: Help desk, customer support, troubleshooting
Prompt structure:
- Identity + tone (empathetic, patient)
- Voice optimization
- Capabilities and tools
- Clear escalation criteria
- Error handling and fallbacks
Example use cases:
- Customer support
- Technical troubleshooting
- Account assistance
- Order status lookup
Troubleshooting
Issue: Agent uses markdown or special formatting
Symptoms: TTS says "asterisk asterisk hello" or skips characters
Solution: Ensure prompt includes:
Respond in plain text only - no emojis, asterisks, markdown, or special formatting.
Prevention: Run validation script
Issue: Responses too long
Symptoms: Users interrupt frequently, seem impatient
Solution: Add stricter constraints:
Keep responses very brief - 1-2 sentences maximum. Ask one question at a time.
Prevention: Test with real users early
Issue: Numbers sound awkward
Symptoms: "Dollar sign forty-nine point nine nine" or unclear prices/phone numbers
Solution: Add number formatting rules:
Spell out numbers ("forty-nine ninety-nine" for prices). Spell phone numbers digit by digit ("5-5-5, 1-2-3-4").
Issue: Agent doesn't use tools correctly
Symptoms: Tools called at wrong time, missing parameters, no error handling
Solution:
- Add explicit tool usage instructions in prompt
- Specify when to use each tool
- Define parameter collection flow
- Add error handling guidance
Check: Load tool-integration.md for detailed patterns
Issue: Agent goes off-topic or out-of-scope
Symptoms: Agent tries to handle requests beyond capabilities
Solution: Add boundaries and guardrails:
You can only help with [specific domain]. For questions about [out-of-scope topics], direct users to [alternative resource].
Issue: Multi-agent transfers feel awkward
Symptoms: Abrupt handoffs, lost context, repeated questions
Solution:
- Add transition messages before transfer
- Define what context passes between agents
- Ensure previous agent collects minimum required info
Check: See multi-agent examples in examples.md
Advanced Topics
Emotional Expression
For agents that should convey emotion, use descriptive language:
Good:
When users share good news, respond warmly and enthusiastically. When they express frustration, respond with empathy and patience.
Avoid: Trying to use emojis, asterisks, or formatting to convey emotion.
Trust TTS: Modern TTS systems convey emotion through intonation based on word choice.
Multiple TTS Providers
Different TTS providers have different characteristics:
- OpenAI TTS - Natural, handles most punctuation well
- Cartesia Sonic - Fast, low-latency, clear pronunciation
- Deepgram - Strong with accents and dialects
- ElevenLabs - Highly expressive, personality-driven
Universal best practices work across all providers:
- Plain text only
- Brief responses
- Spell out numbers
- Avoid special characters
Test your specific provider to identify any quirks.
Handling Sensitive Contexts
For healthcare, finance, or other regulated industries:
Add explicit constraints:
Never:
- Share customer information without verification
- Provide medical/legal/financial advice
- Process transactions without explicit confirmation
Be security-conscious:
For security, you'll need to verify identity before providing account information.
Escalate appropriately:
For [sensitive issue], immediately transfer to [appropriate specialist].
See healthcare and banking examples in examples.md.
Best Practices Summary
- Start with templates - They encode best practices
- Keep prompts concise - Under 300 words when possible
- Always include voice rules - Plain text, brevity, number formatting
- Test with real TTS - Listen to actual output
- Validate before deploying - Use the validation script
- One responsibility per agent - In multi-agent systems
- Iterate based on real usage - Refine after testing
- Error handling is critical - Always include fallback guidance
- Confirm before actions - Especially for state changes
- Boundaries prevent confusion - Clearly define scope
Resources
Templates:
General Purpose:
assets/templates/basic-assistant.txt - Simple conversational agentassets/templates/tool-enabled-agent.txt - Agent with function callingassets/templates/customer-service.txt - Support with escalationassets/templates/specialist-agent.txt - Task-specific workflowsassets/templates/multi-agent-coordinator.txt - Router/greeter agent
Industry-Specific:
assets/templates/front-desk-receptionist.txt - Professional call routing and receptionassets/templates/debt-collection-agent.txt - Compliant debt recovery (FDCPA/TCPA)
Reference Documentation:
references/prompt-components.md - Detailed component guidereferences/voice-optimization.md - TTS formatting deep divereferences/tool-integration.md - Function calling patternsreferences/examples.md - Complete real-world promptsreferences/industry-best-practices.md - Industry-specific patterns and compliancereferences/iterative-improvement.md - Systematic prompt refinement and testing
Tools:
scripts/validate_prompt.py - Validation script
External Resources:
Example: Complete Workflow
Scenario: Build a pizza ordering agent
Step 1: Choose template
- Task-specific with order taking →
specialist-agent.txt
Step 2: Customize
You are Marco, an order specialist at Mario's Pizza. You are friendly, efficient, and helpful.
Keep responses very brief - 1-2 sentences. Plain text only, no formatting. Ask one question at a time.
Our menu:
- Margherita: twelve dollars
- Pepperoni: fourteen dollars
- Veggie Supreme: fifteen dollars
Collect in order:
1. Pizza selection - "What pizza would you like?"
2. Size - "What size: small, medium, or large?"
3. Any modifications - "Any special requests or modifications?"
4. Name - "Name for the order?"
5. Phone - "Phone number?"
Confirm: "Let me confirm: [size] [pizza] for [name] at [phone]. Total is [price]. Is that correct?"
Use create_order(pizza, size, modifications, name, phone) after confirmation.
Spell out prices: "fourteen dollars" not "dollar sign fourteen".
Spell phone numbers: "5-5-5, 1-2-3-4".
Step 3: Validate
python scripts/validate_prompt.py pizza_prompt.txt
Step 4: Deploy and test with LiveKit
Step 5: Refine based on feedback
- If users often ask about delivery time → Add delivery info to domain context
- If tool fails occasionally → Add error handling for out-of-stock items
- If users interrupt with questions → Add guidance for handling interruptions
Iterative Improvement
Voice agent prompts improve through continuous refinement based on real-world usage:
Quick Improvement Workflow
- Collect feedback - Gather test results, user feedback, conversation metrics
- Identify patterns - Look for common issues across multiple conversations
- Make targeted changes - Modify specific sections while preserving structure
- Test changes - Validate improvements with test scenarios
- Measure impact - Compare before/after metrics
- Iterate - Repeat the cycle
Key Principles
- Structure preservation - Make minimal, targeted changes to address specific issues
- Data-driven - Use evidence from tests and conversations, not guesses
- Version control - Track changes and document reasons
- A/B testing - When possible, run old and new versions in parallel
- Regression testing - Ensure changes don't break existing functionality
For detailed guidance: See references/iterative-improvement.md
Industry-Specific Guidance
Different industries require specialized approaches:
Debt Collection:
- Empathetic but professional tone
- FDCPA/TCPA/CFPB compliance built-in
- Sentiment-aware escalation
- Payment arrangement workflows
Healthcare:
- HIPAA compliance and privacy protection
- Emergency detection and routing
- Professional, caring tone
- Limited scope (no medical advice)
Banking/Financial:
- Security-first verification
- Fraud detection awareness
- Regulatory compliance
- Transaction limitations
Customer Service:
- Empathetic problem-solving
- Clear escalation criteria
- Knowledge base integration
- Omnichannel awareness
Front Desk/Reception:
- Professional greeting frameworks
- Efficient call routing
- Message taking protocols
- After-hours handling
For detailed patterns: See references/industry-best-practices.md
Getting Started Now
- Pick the closest template for your use case
- Copy it and fill in your information
- Run the validator to catch common issues
- Test it in your LiveKit agent
- Collect feedback and iterate based on real usage
- Come back to this skill when you need to refine or troubleshoot
The templates handle 80% of the work. Focus your customization on domain-specific information, tools, and workflows. Then continuously improve based on real-world performance.
