| name | deploying-scalable-agents |
| license | MIT |
| description | Take one working agent prototype go scalable, observable production deployment for Microsoft Foundry. E cover deployment patterns (client-hosted, hosted agents, agent workflows), the agent lifecycle, model routing, response caching, evaluation gates, human-in-the-loop approval, observability with OpenTelemetry, cost optimisation, and smoke-testing deployed agents with the AI Smoke Test action. Based on Lesson 16 of AI Agents for Beginners. USE FOR: deploy one agent go production, scale one agent, Microsoft Foundry hosted agent, Foundry Agent Service, model routing, response caching, evaluation gate, release gate, human approval workflow, agent observability, agent tracing, agent cost optimisation, smoke test one hosted agent, production customer support agent. DO NOT USE FOR: building your first agent (start with Lesson 01), running agents locally on-device (use local-ai-agents / Lesson 17), Azure infrastructure provisioning we no relate to agents, non-Foundry deployment targets. |
Deploying Scalable Agents with Microsoft Foundry
Companion skill for Lesson 16 – Deploying Scalable Agents.
Use am to help learner move agent from prototype go scalable, observable
production deployment. Ground every recommendation inside lesson content and
the runnable notebook; no make you invent Foundry APIs.
Triggers
Activate dis skill when learner wan:
- Deploy agent to Microsoft Foundry as hosted agent and make am versioned/observable.
- Choose between client-hosted, hosted-agent, and agent-workflow deployment patterns.
- Add model routing, response caching, or bounded concurrency to control latency and cost.
- Add evaluation gate so bad agent version no fit ship.
- Add human-in-the-loop approval step for high-risk actions.
- Instrument agent wit OpenTelemetry tracing for production observability.
- Smoke-test deployed agent as quick post-deploy gate.
Core mental model
Production agent na mostly operational skeleton around di model (~80%),
no be di model itself. Map every recommendation to one of dis concerns:
| Concern | Prototype → Production |
|---|
| Hosting | notebook → versioned hosted service |
| Identity | your az login → managed identity + scoped RBAC |
| State | in-memory → externalised thread/memory store |
| Failure | traceback → retries, fallbacks, alerts |
| Cost | "small small cents" → tracked, routed, cached, budgeted |
| Quality | eyeballing → automated evaluation gate |
| Trust | you approve → policy + human-in-the-loop |
Deployment patterns (pick one, or combine)
- Client-hosted — di reasoning loop dey run inside your process. Max control; you dey own scaling/state.
- Hosted agent (Foundry Agent Service) — Foundry dey host the loop, dey store threads, dey enforce RBAC/content safety, dey show di agent for portal. Less control, far less operational wahala.
- Agent workflow — multiple agents/tools join body as graph wit branching, approval nodes, and durable checkpoints.
Lifecycle (di loop wey dey deliver agent)
create → version → evaluate (gate) → deploy hosted → observe online → collect failures → repeat.
Offline evaluation na gate, no be afterthought — version no go ship
unless e clear di threshold. Online observability dey feed real failures back
enter di offline test set.
Scaling and cost levers (priority order)
- Right-size di model — use di smallest model wey fit pass di evaluation gate.
- Route by complexity — small/fast model for simple requests, big model for real reasoning (DIY classifier or Foundry Model Router).
- Cache — serve near-duplicate requests without model call.
- Stateless design + bounded concurrency — externalise state; retry wit backoff.
Key patterns wey dem suppose reproduce
Point learner to these from di notebook
16-python-agent-framework.ipynb:
- Request handler: cache → route by complexity → trace span → run → cache.
- Evaluation gate: score offline test set; return
pass_rate >= threshold and only deploy if true.
- Human approval:
@tool(approval_mode="always_require") for actions like large refunds.
- Tracing: wrap each request inside
tracer.start_as_current_span(...) and set attributes like routed.model, customer.id.
Smoke-testing deployed agent
After deploy, make sure say endpoint really dey answer (green deploy fit still dey
silent). Use AI Smoke Test
action via .github/workflows/smoke-test.yml
with catalog for tests/. Runner dey POST each
prompt to POST {project_endpoint}/agents/{agent_name}/endpoint/protocols/openai/responses
and e dey check the reply text. Identity need Azure AI User role at
Foundry project scope; token audience must be https://ai.azure.com/.
Combine di gates: smoke test (make sure e dey respond, every deploy) → offline
evaluation (good enough to ship before promotion) → online evaluation (how
e dey perform for real life, continuous).
Enterprise controls
- RBAC: give every hosted agent managed identity with least privilege.
- MCP for production: treat every MCP server as untrusted boundary — pin version, scope identity, validate outputs, rate-limit, no ever expose secrets.
Guardrails for the assistant
- Prefer di canonical
FoundryChatClient(...) + provider.as_agent(...) pattern wey dem dey use for di whole course.
- No promise live-Azure results wey you never comfirm; recommend di smoke-test workflow to confirm deployment.
- Keep evaluation and cost advice together: evaluation set di quality floor, routing/caching keep cost near dat floor.
Disclaimer:
Dis document don translate wit AI translation service Co-op Translator. Even tho we dey try make am correct, abeg make you know say automated translation fit get errors or mistakes. Di original document for dia own language na im be di correct source. For important info, make person wey sabi human translation do am. We no go responsible for any misunderstanding or wrong understanding wey fit happen because of dis translation.
