| name | copilot-studio-bot-design |
| description | Design Copilot Studio bots — topic vs generative-answers boundaries, knowledge-source hygiene, trigger-phrase design, slot-filling patterns, escalation-to-human criteria, AI Builder vs prompt-flow vs direct Azure OpenAI decisions, and the test-set discipline that catches regressions on every authoring change. Used by `copilot-studio-engineer` (primary). |
Copilot Studio Bot Design Skill
Purpose: Senior-maker playbook for copilot-studio-engineer covering the architecture decisions that determine whether a Copilot Studio bot is a useful coworker or a hallucinating embarrassment in front of customers. The decisions are made on day one — topic vs generative-answers boundaries, knowledge sources, escalation criteria — and the test discipline that keeps them honest as the bot evolves.
When to Use
- Starting a new bot — before the first topic is built, work through the architecture decisions here.
- A deployed bot is misbehaving — hallucinating answers, dropping conversations, escalating too often or not often enough, ignoring intent. This skill covers the diagnostic playbook.
- Adding a new knowledge source or topic to an existing bot — verify it doesn't blow up the existing test set.
- Deciding AI implementation path — Copilot Studio vs AI Builder vs prompt-flow vs direct Azure OpenAI. The trade-off matters for cost, governance, and freshness.
Core Principles
- A bot is a contract about what the org will answer, not a wrapper around an LLM. Treat it like a product surface — versioned, tested, owned.
- Topics for high-stakes, regulated, or specific-format answers. Generative-answers for everything else. The line is "would I want a human to read this exact response before it shipped?" If yes, it's a topic.
- Knowledge-source hygiene is the #1 hallucination prevention. Garbage in, confidently-wrong out. Stale, duplicated, or contradictory sources poison the bot worse than no source.
- Every authoring change re-runs the regression set. A 20-prompt test set is the difference between "we improved the bot" and "we hope we improved the bot."
- Escalation to human is a feature, not a failure. Bots that never escalate are usually hallucinating instead of saying "I don't know."
- Premium connector cost is real — Copilot Studio messages are licensed; AI Builder credits are counted; direct Azure OpenAI is metered. Every architecture choice has a unit-economics consequence.
Playbook
1. Copilot Studio architecture in plain English
A Copilot Studio bot has four kinds of building blocks:
- Topics — authored conversations with explicit trigger phrases, slot-filling, branching, and exit conditions. Deterministic. You wrote it; the bot says exactly what you wrote.
- Generative answers — retrieval-augmented generation (RAG) over the bot's knowledge sources. The bot reads from sources at query time, an LLM composes the answer. Non-deterministic but grounded.
- Knowledge sources — websites, SharePoint, documents, Dataverse, custom data — pointed at the bot, indexed, queryable by generative answers.
- Plugins / actions — calls into Power Automate flows, custom connectors, AI Builder prompts, or external APIs. Executes work; not just speech.
The bot router decides per-utterance: matches an authored topic? Run it. Otherwise hand to generative answers, or escalate.
2. Topic vs generative-answers — the boundary
| Use a topic when… | Use generative answers when… |
|---|
| Regulated content (legal, HR policy, financial disclosure) | Informational FAQ over public-ish content |
| High-blast-radius actions (file a claim, place an order, transfer money) | Synthesizing across many docs to answer a question |
| The answer has a specific format (table, JSON, exact wording) | The answer space is broad and slow-changing |
| Slot-filling required (you need 5 fields to do anything) | The user's question is usually one-shot |
| Compliance demands logged determinism | Variation in phrasing is acceptable |
Test for the line: "if a user gets a wrong answer here, who explains it to compliance / legal / the customer?" If the answer is a real person, it should be a topic.
3. Knowledge-source hygiene
The most common cause of bot hallucination is bad sources, not bad models.
- Curate over collect. Pointing the bot at the whole intranet is a recipe for stale, contradictory answers. Pick canonical sources per topic area.
- Publish-to-bot vs reference. Some sources should be authoritative; others should be reference-only. Mark them accordingly so the bot can cite confidence-weighted.
- Freshness control. Each source has a refresh interval. If a SharePoint page changes daily but indexes weekly, users see stale answers. Set the cadence deliberately.
- De-duplicate and resolve contradictions. If three pages explain the same policy differently, the bot will pick one — often the wrong one. Fix the source content before adding more.
- Test each source in isolation. Add it, run a topic-specific test set, verify no regression.
4. Trigger-phrase design
Trigger phrases are intent classes, not synonyms. Each topic should have 8-15 trigger phrases representing the range of how users would express that intent.
- Bad: "reset password", "reset my password", "password reset" (3 variants of one phrase)
- Good: "reset password", "forgot password", "can't log in", "locked out", "need to change my password", "my password isn't working", "request a new password" (range of intent expressions)
The Copilot Studio NLU model generalises from the range. Narrow training data = brittle triggering.
5. Slot-filling patterns
For any topic that requires inputs:
- Required vs optional — explicitly mark. Optional slots have a default.
- Validation — every slot has a validation expression. "Email" requires regex; "Order Number" requires format check; "Date" requires date parse.
- Re-prompt — when validation fails, the re-prompt should explain why it failed and give an example. "I didn't understand — could you provide your order number in the format ABC-12345?"
- Maximum attempts — after 3 failed attempts, escalate. Looping forever frustrates users and hides bugs.
- Confirmation — before any irreversible action, show the user the collected values and ask "is this right?" Slot-filling without confirmation is how the wrong record gets updated.
6. Guardrails (system prompt, content moderation, off-topic redirection)
- System prompt at the bot level establishes voice, scope, and refusal patterns. "You are a customer service assistant for ACME. You only answer questions related to ACME products and orders. For anything else, politely redirect to live chat."
- Content moderation — Copilot Studio has built-in moderation; verify it's on. For sensitive domains, layer Purview labels and a custom moderation flow.
- Off-topic redirection — when generative answers return low-confidence or the user asks about an obvious off-topic (weather, news, jokes), respond with the redirection pattern, not an attempt.
- Hallucination guardrails — turn on "moderate" or "strict" grounding mode for generative answers; raise the confidence threshold; require citations.
7. Escalation to human — when, how, context
- When: explicit user request ("talk to a person"), N failed slot-fill attempts, low-confidence generative answer in a sensitive topic, any topic explicitly marked "always escalate" (legal, complaints, billing disputes).
- How: Teams channel handoff, Dynamics Customer Service queue, third-party live-chat integration, or email-the-conversation. Pick one and instrument it.
- Context handoff: the human picks up with the full conversation transcript, the detected intent, the values collected in slots, and any actions the bot took. A handoff with no context is worse than no bot — the user has to re-explain everything.
8. AI Builder vs prompt-flow vs direct Azure OpenAI
Decision matrix for "where should the AI step live?":
| Path | Best for | Cost shape | Governance |
|---|
| Copilot Studio generative answers | RAG-style Q&A over curated sources | Per-message licensing | Within Copilot Studio's roof, includes content moderation |
| AI Builder Prompts | Reusable single-shot prompt within a flow or topic | AI Builder credits per call | Authored as a Dataverse component; visible in CoE Kit |
| Custom prompt-flow | Multi-step LLM workflow with branching, RAG, tool use | Compute + LLM tokens | Hosted in Azure ML; separate identity surface |
| Direct Azure OpenAI | Maximum control, custom RAG, fine-tuning, real-time streaming UX | Tokens directly; cheapest at scale | You own everything — moderation, abuse prevention, prompt logging |
Default: stay inside Copilot Studio / AI Builder for governance. Only step out to prompt-flow / Azure OpenAI when there's a concrete requirement (cost at scale, custom RAG with vector store, fine-tuning, streaming UX) that justifies the operational overhead.
Premium connector / capacity note — every option here has cost. Call it out in the recommendation per §3 #8.
9. Regression test sets (the discipline)
Build a test set of 10-30 canonical prompts covering:
- Each topic's trigger-phrase range (2-3 prompts per topic).
- Each generative-answer scope (5-10 known questions with known correct answers).
- Adversarial / out-of-scope prompts (5+ — verify the bot redirects, doesn't hallucinate).
- Escalation triggers (verify escalation fires).
- Slot-filling validation (verify validation rejects bad input).
Re-run the entire set on every authoring change. Manually if necessary; automated via the Copilot Studio testing pane or Power Automate where possible. Save the test transcripts. A regression in test set = block the release.
10. Measurement (containment, escalation, resolution, CSAT)
- Containment rate — % of conversations the bot completes without escalating. Higher isn't always better; a bot that "contains" by hallucinating is worse than escalating.
- Escalation rate — % escalated. Track appropriate vs failure escalations.
- Resolution rate — for transactional bots, did the user actually complete the task?
- CSAT — end-of-conversation feedback. Two scales: was the answer accurate; was the experience good.
Watch trends, not absolutes. A drop after a release usually points to a knowledge source change or trigger phrase regression.
Anti-Patterns to Flag
- One mega-topic with 47 branches instead of decomposed topics
- "Set the bot loose on the intranet" knowledge configuration
- No regression test set ("we'll test it manually each release")
- No escalation path ("the bot will figure it out")
- System prompt copy-pasted from a tutorial, doesn't match the org voice or scope
- Slot-filling without confirmation before irreversible actions
- Trigger phrases that are all variants of one phrasing
- Hard-coded URLs, IDs, or secrets in topic responses (§3 #2 — env vars)
- Premium AI Builder / generative-answer usage without explicit licensing call-out (§3 #8)
Escalation
- Bot will handle PII / PHI / regulated content →
ravenclaude-core security-reviewer on the data flow before any deployment.
- Bot calls flows that touch transactional systems →
flow-engineer + power-automate skill for resilient action wiring; dataverse-architect if it writes to Dataverse.
- Need to ship via solution ALM →
solution-alm-engineer + alm-pipeline-design skill. Bots are solution-aware; treat as code.
- Tenant DLP blocks the bot's required connector →
power-platform-admin + dlp-policy-design skill, not a hallway exemption.
- Stuck on which AI path →
architect in ravenclaude-core for the cost / governance / data-residency conversation.
When in doubt, ship narrower scope with stronger guardrails than wider scope with looser guardrails. A bot that handles 5 things well is a product. A bot that handles 500 things badly is a liability.
