livekit-prompt-builder

name

livekit-prompt-builder

description

Guide for creating effective prompts and instructions for LiveKit voice agents. Use when building conversational AI agents with the LiveKit Agents framework, including (1) Creating new voice agent prompts from scratch, (2) Improving existing agent instructions, (3) Optimizing prompts for text-to-speech output, (4) Integrating tool/function calling capabilities, (5) Building multi-agent systems with handoffs, (6) Ensuring voice-friendly formatting and brevity for natural conversations, (7) Iteratively improving prompts based on testing and feedback, (8) Building industry-specific agents (debt collection, healthcare, banking, customer service, front desk).

LiveKit Voice Agent Prompt Builder

Create production-ready prompts for LiveKit voice agents that work seamlessly with text-to-speech systems.

Quick Start

For most use cases, follow this simple workflow:

1. Choose a template from assets/templates/ that matches your use case
2. Customize the template with your specific information
3. Validate using scripts/validate_prompt.py
4. Test with your LiveKit agent
5. Refine based on real conversations

Workflow

Step 1: Determine Your Agent Type

Select the template that best matches your needs:

General Purpose:

• Basic Assistant (basic-assistant.txt) - Simple conversational agent

  • Use for: General chat, demos, minimal functionality
  • Complexity: Low
  • Tools: Optional

• Tool-Enabled Agent (tool-enabled-agent.txt) - Agent with function calling

  • Use for: Weather, search, data lookup, single-domain assistants
  • Complexity: Medium
  • Tools: 1-5 tools

• Customer Service (customer-service.txt) - Support agent with escalation

  • Use for: Help desks, support lines, troubleshooting
  • Complexity: Medium
  • Tools: Lookup, search, escalation

• Specialist Agent (specialist-agent.txt) - Task-specific agent (reservations, orders, etc.)

  • Use for: Booking systems, order taking, appointments
  • Complexity: Medium-High
  • Tools: Action tools (create, update)

• Multi-Agent Coordinator (multi-agent-coordinator.txt) - Routes to other agents

  • Use for: Main entry point in multi-agent systems
  • Complexity: Medium
  • Tools: Transfer/handoff tools

Industry-Specific:

• Front Desk Receptionist (front-desk-receptionist.txt) - Professional call routing

  • Use for: Office reception, call centers, front desk operations
  • Complexity: Low-Medium
  • Tools: Transfer, routing, message taking

• Debt Collection (debt-collection-agent.txt) - Compliant recovery agent

  • Use for: Debt recovery, collections, payment arrangements
  • Complexity: High
  • Tools: Payment processing, sentiment analysis
  • Note: Includes FDCPA/TCPA/CFPB compliance requirements

Step 2: Customize Your Template

Open your selected template and fill in the placeholders:

Example - Customizing basic-assistant.txt:

Template:

You are [NAME], a friendly voice assistant. You are [TRAIT_1] and [TRAIT_2], with [TRAIT_3].

Customized:

You are Casey, a friendly voice assistant. You are helpful and patient, with a sense of humor.

Key customization areas:

• Identity (name, role, personality)
• Domain context (business info, hours, policies)
• Tool descriptions (when and how to use each tool)
• Workflow steps (information collection sequence)
• Boundaries (what agent can't do)

Step 3: Apply Voice Optimization

Ensure your prompt includes these critical voice optimization rules:

Required elements:

Keep your responses brief - 1-3 sentences. Respond in plain text only - no emojis, asterisks, markdown, or special formatting. Ask one question at a time.

For agents that mention numbers, prices, or contact info:

Spell out numbers ("fifteen" not "15"). Spell phone numbers digit by digit ("5-5-5, 1-2-3-4"). For URLs, omit "https://" and say "dot" for periods.

For agents with tools:

If a tool call fails, explain the issue simply and suggest a fallback. Summarize tool results in conversational language - don't recite technical IDs or raw data.

Step 4: Validate Your Prompt

Run the validation script to check for common issues:

python scripts/validate_prompt.py your_prompt.txt

The validator checks for:

• Missing identity statement
• Lack of voice optimization rules
• TTS antipatterns (emojis, markdown, special characters)
• Excessive length (>500 words)
• Missing tool error handling
• Number formatting guidance

Fix any errors before deploying your prompt.

Step 5: Test with LiveKit

Deploy your prompt in a LiveKit agent and test with real conversations:

Test scenarios:

1. Happy path - User provides information smoothly
2. Interruptions - User changes topic or interrupts
3. Tool failures - Simulate tool errors
4. Edge cases - Ambiguous requests, out-of-scope questions
5. Voice quality - Listen to TTS output for formatting issues

Step 6: Refine

Based on testing, refine your prompt:

Common refinements:

• Add specific guidance for observed failure modes
• Adjust tone or personality descriptors
• Add examples for complex tool usage
• Tighten response length constraints if agent is too verbose
• Add boundaries for out-of-scope requests

Key Concepts

Voice-First Design

Voice agents differ from text chatbots:

Critical differences:

• No visual formatting (users can't see bold, lists, etc.)
• Limited user patience (long responses cause frustration)
• TTS limitations (special characters cause problems)
• No "scroll back" (users can't review previous responses)

Implications for prompts:

• Always specify plain text only
• Enforce brevity (1-3 sentences typical)
• Spell out numbers and abbreviations
• Ask one question at a time
• Avoid complex structures (lists, tables)

The Three Essential Sections

Every voice agent prompt should have:

1. Identity - "You are [name/role]. You are [personality]."
2. Voice Rules - Plain text, brief responses, number formatting
3. Core Functionality - What the agent does, tools it uses, workflows it follows

Additional sections (domain context, guardrails, handoffs) add as needed.

Template vs. Custom Prompts

Use templates when:

• Starting a new agent
• Following common patterns (support, booking, etc.)
• Want quick setup with best practices

Write custom when:

• Unique use case not covered by templates
• Very specialized domain
• Complex multi-step workflows

Even with custom prompts, reference the templates for voice optimization patterns.

Progressive Disclosure: When to Load References

Load reference documentation based on your specific needs:

For all users:

• Start with templates and this SKILL.md

Load prompt-components.md when:

• Building complex prompts from scratch
• Need detailed guidance on specific sections
• Want to understand the "why" behind best practices
• Customizing beyond simple template fill-in

Load voice-optimization.md when:

• Encountering TTS formatting issues
• Agent responses sound awkward or unnatural
• Need deep understanding of voice-specific constraints
• Working with multiple TTS providers
• Debugging number, URL, or email pronunciation

Load tool-integration.md when:

• Integrating function calling/tools
• Building multi-agent systems with handoffs
• Need patterns for error handling or parameter collection
• Tools aren't being used correctly
• Working with RAG or external data sources

Load examples.md when:

• Want to see complete, real-world prompts
• Need inspiration for similar use cases
• Comparing complexity levels
• Learning from production patterns

Load industry-best-practices.md when:

• Building agents for regulated industries (healthcare, banking, debt collection)
• Need compliance guidance (HIPAA, FDCPA, TCPA, etc.)
• Working on industry-specific use cases (front desk, customer service, sales)
• Require specialized tone and approach patterns
• Need security or privacy considerations

Load iterative-improvement.md when:

• Improving existing prompts based on feedback
• Analyzing test results and conversation data
• Conducting A/B tests on prompt variations
• Establishing metrics and measurement frameworks
• Need systematic approach to prompt refinement
• Want to preserve prompt structure while making improvements

Strategy: Start minimal (templates + SKILL.md). Load references only when you hit specific challenges or need deeper guidance.

Multi-Agent Systems

For systems with multiple agents, design each agent separately:

Pattern:

1. Coordinator/Greeter - Uses multi-agent-coordinator.txt

   • Simple routing logic
   • Minimal conversation
   • Quick transfers

2. Specialist Agents - Use specialist-agent.txt

   • Each handles one specific task
   • Collects required information
   • Takes action or transfers to next agent

Key considerations:

• Each agent should have a single, clear responsibility
• Define handoff criteria explicitly
• Decide what context to pass between agents (full chat history vs structured data)
• Test agent transitions for smoothness

Common Patterns

Pattern 1: Simple Conversational Agent

Template: basic-assistant.txt

Use for: Chat, demos, general assistance

Prompt structure:

• Identity + personality
• Voice optimization rules
• Optional: specific topics or capabilities

Example use cases:

• Demo voice assistant
• Companion chatbot
• General information assistant

Pattern 2: Information Retrieval Agent

Template: tool-enabled-agent.txt

Use for: Looking up data, answering questions from external sources

Prompt structure:

• Identity + domain
• Voice optimization
• Tool descriptions (when to use each tool)
• Result summarization guidance

Example use cases:

• Weather agent
• Product lookup
• Knowledge base search
• RAG-enabled assistants

Pattern 3: Action-Taking Agent

Template: specialist-agent.txt

Use for: Booking, ordering, scheduling, transactions

Prompt structure:

• Identity + role
• Voice optimization
• Information collection sequence
• Confirmation before action
• Tool usage for state changes

Example use cases:

• Reservation agent
• Order taking
• Appointment scheduling
• Registration

Pattern 4: Support Agent with Escalation

Template: customer-service.txt

Use for: Help desk, customer support, troubleshooting

Prompt structure:

• Identity + tone (empathetic, patient)
• Voice optimization
• Capabilities and tools
• Clear escalation criteria
• Error handling and fallbacks

Example use cases:

• Customer support
• Technical troubleshooting
• Account assistance
• Order status lookup

Troubleshooting

Issue: Agent uses markdown or special formatting

Symptoms: TTS says "asterisk asterisk hello" or skips characters

Solution: Ensure prompt includes:

Respond in plain text only - no emojis, asterisks, markdown, or special formatting.

Prevention: Run validation script

Issue: Responses too long

Symptoms: Users interrupt frequently, seem impatient

Solution: Add stricter constraints:

Keep responses very brief - 1-2 sentences maximum. Ask one question at a time.

Prevention: Test with real users early

Issue: Numbers sound awkward

Symptoms: "Dollar sign forty-nine point nine nine" or unclear prices/phone numbers

Solution: Add number formatting rules:

Spell out numbers ("forty-nine ninety-nine" for prices). Spell phone numbers digit by digit ("5-5-5, 1-2-3-4").

Issue: Agent doesn't use tools correctly

Symptoms: Tools called at wrong time, missing parameters, no error handling

Solution:

1. Add explicit tool usage instructions in prompt
2. Specify when to use each tool
3. Define parameter collection flow
4. Add error handling guidance

Check: Load tool-integration.md for detailed patterns

Issue: Agent goes off-topic or out-of-scope

Symptoms: Agent tries to handle requests beyond capabilities

Solution: Add boundaries and guardrails:

You can only help with [specific domain]. For questions about [out-of-scope topics], direct users to [alternative resource].

Issue: Multi-agent transfers feel awkward

Symptoms: Abrupt handoffs, lost context, repeated questions

Solution:

1. Add transition messages before transfer
2. Define what context passes between agents
3. Ensure previous agent collects minimum required info

Check: See multi-agent examples in examples.md

Advanced Topics

Emotional Expression

For agents that should convey emotion, use descriptive language:

Good:

When users share good news, respond warmly and enthusiastically. When they express frustration, respond with empathy and patience.

Avoid: Trying to use emojis, asterisks, or formatting to convey emotion.

Trust TTS: Modern TTS systems convey emotion through intonation based on word choice.

Multiple TTS Providers

Different TTS providers have different characteristics:

• OpenAI TTS - Natural, handles most punctuation well
• Cartesia Sonic - Fast, low-latency, clear pronunciation
• Deepgram - Strong with accents and dialects
• ElevenLabs - Highly expressive, personality-driven

Universal best practices work across all providers:

1. Plain text only
2. Brief responses
3. Spell out numbers
4. Avoid special characters

Test your specific provider to identify any quirks.

Handling Sensitive Contexts

For healthcare, finance, or other regulated industries:

Add explicit constraints:

Never:
- Share customer information without verification
- Provide medical/legal/financial advice
- Process transactions without explicit confirmation

Be security-conscious:

For security, you'll need to verify identity before providing account information.

Escalate appropriately:

For [sensitive issue], immediately transfer to [appropriate specialist].

See healthcare and banking examples in examples.md.

Best Practices Summary

1. Start with templates - They encode best practices
2. Keep prompts concise - Under 300 words when possible
3. Always include voice rules - Plain text, brevity, number formatting
4. Test with real TTS - Listen to actual output
5. Validate before deploying - Use the validation script
6. One responsibility per agent - In multi-agent systems
7. Iterate based on real usage - Refine after testing
8. Error handling is critical - Always include fallback guidance
9. Confirm before actions - Especially for state changes
10. Boundaries prevent confusion - Clearly define scope

Resources

Templates:

General Purpose:

• assets/templates/basic-assistant.txt - Simple conversational agent
• assets/templates/tool-enabled-agent.txt - Agent with function calling
• assets/templates/customer-service.txt - Support with escalation
• assets/templates/specialist-agent.txt - Task-specific workflows
• assets/templates/multi-agent-coordinator.txt - Router/greeter agent

Industry-Specific:

• assets/templates/front-desk-receptionist.txt - Professional call routing and reception
• assets/templates/debt-collection-agent.txt - Compliant debt recovery (FDCPA/TCPA)

Reference Documentation:

• references/prompt-components.md - Detailed component guide
• references/voice-optimization.md - TTS formatting deep dive
• references/tool-integration.md - Function calling patterns
• references/examples.md - Complete real-world prompts
• references/industry-best-practices.md - Industry-specific patterns and compliance
• references/iterative-improvement.md - Systematic prompt refinement and testing

Tools:

• scripts/validate_prompt.py - Validation script

External Resources:

• LiveKit Agents Documentation: https://docs.livekit.io/agents/
• LiveKit Prompting Guide: https://docs.livekit.io/agents/build/prompting/
• LiveKit Examples: https://github.com/livekit/agents/tree/main/examples

Example: Complete Workflow

Scenario: Build a pizza ordering agent

Step 1: Choose template

• Task-specific with order taking → specialist-agent.txt

Step 2: Customize

You are Marco, an order specialist at Mario's Pizza. You are friendly, efficient, and helpful.

Keep responses very brief - 1-2 sentences. Plain text only, no formatting. Ask one question at a time.

Our menu:
- Margherita: twelve dollars
- Pepperoni: fourteen dollars
- Veggie Supreme: fifteen dollars

Collect in order:
1. Pizza selection - "What pizza would you like?"
2. Size - "What size: small, medium, or large?"
3. Any modifications - "Any special requests or modifications?"
4. Name - "Name for the order?"
5. Phone - "Phone number?"

Confirm: "Let me confirm: [size] [pizza] for [name] at [phone]. Total is [price]. Is that correct?"

Use create_order(pizza, size, modifications, name, phone) after confirmation.

Spell out prices: "fourteen dollars" not "dollar sign fourteen".
Spell phone numbers: "5-5-5, 1-2-3-4".

Step 3: Validate

python scripts/validate_prompt.py pizza_prompt.txt

Step 4: Deploy and test with LiveKit

Step 5: Refine based on feedback

• If users often ask about delivery time → Add delivery info to domain context
• If tool fails occasionally → Add error handling for out-of-stock items
• If users interrupt with questions → Add guidance for handling interruptions

Iterative Improvement

Voice agent prompts improve through continuous refinement based on real-world usage:

Quick Improvement Workflow

1. Collect feedback - Gather test results, user feedback, conversation metrics
2. Identify patterns - Look for common issues across multiple conversations
3. Make targeted changes - Modify specific sections while preserving structure
4. Test changes - Validate improvements with test scenarios
5. Measure impact - Compare before/after metrics
6. Iterate - Repeat the cycle

Key Principles

• Structure preservation - Make minimal, targeted changes to address specific issues
• Data-driven - Use evidence from tests and conversations, not guesses
• Version control - Track changes and document reasons
• A/B testing - When possible, run old and new versions in parallel
• Regression testing - Ensure changes don't break existing functionality

For detailed guidance: See references/iterative-improvement.md

---

Industry-Specific Guidance

Different industries require specialized approaches:

Debt Collection:

• Empathetic but professional tone
• FDCPA/TCPA/CFPB compliance built-in
• Sentiment-aware escalation
• Payment arrangement workflows

Healthcare:

• HIPAA compliance and privacy protection
• Emergency detection and routing
• Professional, caring tone
• Limited scope (no medical advice)

Banking/Financial:

• Security-first verification
• Fraud detection awareness
• Regulatory compliance
• Transaction limitations

Customer Service:

• Empathetic problem-solving
• Clear escalation criteria
• Knowledge base integration
• Omnichannel awareness

Front Desk/Reception:

• Professional greeting frameworks
• Efficient call routing
• Message taking protocols
• After-hours handling

For detailed patterns: See references/industry-best-practices.md

---

Getting Started Now

1. Pick the closest template for your use case
2. Copy it and fill in your information
3. Run the validator to catch common issues
4. Test it in your LiveKit agent
5. Collect feedback and iterate based on real usage
6. Come back to this skill when you need to refine or troubleshoot

The templates handle 80% of the work. Focus your customization on domain-specific information, tools, and workflows. Then continuously improve based on real-world performance.