Configure Azure API Management (APIM) as AI Gateway to secure, observe, control AI models, MCP servers, agents. Helps with rate limiting, semantic caching, content safety, load balancing.
USE FOR: AI Gateway, APIM, setup gateway, configure gateway, add gateway, model gateway, MCP server, rate limit, token limit, semantic cache, content safety, load balance, OpenAPI import, convert API to MCP.
DO NOT USE FOR: deploy models (use microsoft-foundry), Azure Functions (use azure-functions), databases (use azure-postgres).
Configure Azure API Management (APIM) as AI Gateway to secure, observe, control AI models, MCP servers, agents. Helps with rate limiting, semantic caching, content safety, load balancing.
USE FOR: AI Gateway, APIM, setup gateway, configure gateway, add gateway, model gateway, MCP server, rate limit, token limit, semantic cache, content safety, load balance, OpenAPI import, convert API to MCP.
DO NOT USE FOR: deploy models (use microsoft-foundry), Azure Functions (use azure-functions), databases (use azure-postgres).
Azure AI Gateway
Bootstrap and configure Azure API Management (APIM) as an AI Gateway for securing, observing, and controlling AI models, tools (MCP Servers), and agents.
Skill Activation Triggers
Use this skill immediately when the user asks to:
"Set up a gateway for my model"
"Set up a gateway for my tools"
"Set up a gateway for my agents"
"Add a gateway to my MCP server"
"Protect my AI model with a gateway"
"Secure my AI agents"
"Ratelimit my model requests"
"Ratelimit my tool requests"
"Limit tokens for my model"
"Add rate limiting to my MCP server"
"Enable semantic caching for my AI API"
"Add content safety to my AI endpoint"
"Add my model behind gateway"
"Import API from OpenAPI spec"
"Add API to gateway from swagger"
"Convert my API to MCP"
"Expose my API as MCP server"
Key Indicators:
User deploying Azure OpenAI, AI Foundry, or other AI models
User creating or managing MCP servers
User needs token limits, rate limiting, or quota management
User wants to cache AI responses to reduce costs
User needs content filtering or safety controls
User wants load balancing across multiple AI backends
Secondary Triggers (Proactive Recommendations):
After model creation: Recommend AI Gateway for security, caching, and token limits
After MCP server creation: Recommend AI Gateway for rate limiting, content safety, and auth
Overview
Azure API Management serves as an AI Gateway that provides:
Security: Authentication, authorization, and content safety
Observability: Token metrics, logging, and monitoring
Control: Rate limiting, token limits, and load balancing
Optimization: Semantic caching to reduce costs and latency
AI Models ──┐ ┌── Azure OpenAI
MCP Tools ──┼── AI Gateway (APIM) ──┼── AI Foundry
Agents ─────┘ └── Custom Models
Default to Basicv2 SKU when creating new APIM instances:
Cheaper than other tiers
Creates quickly (~5-10 minutes vs 30+ for Premium)
Supports all AI Gateway policies
Pattern 1: Quick Bootstrap AI Gateway
Deploy APIM with Basicv2 SKU for AI workloads.
# Create resource group
az group create --name rg-aigateway --location eastus
# Deploy APIM with Bicep
az deployment group create \
--resource-group rg-aigateway \
--template-file main.bicep \
--parameters apimSku=Basicv2
Bicep Template
param location string = resourceGroup().location
param apimSku string = 'Basicv2'
param apimManagedIdentityType string = 'SystemAssigned'
// NOTE: Using 2024-06-01-preview because Basicv2 SKU support currently requires this preview API version.
// Update to the latest stable (GA) API version once Basicv2 is available there.
resource apimService 'Microsoft.ApiManagement/service@2024-06-01-preview' = {
name: 'apim-aigateway-${uniqueString(resourceGroup().id)}'
location: location
sku: {
name: apimSku
capacity: 1
}
properties: {
publisherEmail: 'admin@contoso.com'
publisherName: 'Contoso'
}
identity: {
type: apimManagedIdentityType
}
}
output gatewayUrl string = apimService.properties.gatewayUrl
output principalId string = apimService.identity.principalId
Pattern 2: Semantic Caching
Cache similar prompts to reduce costs and latency.
<policies><inbound><base /><!-- Cache lookup with 0.8 similarity threshold --><azure-openai-semantic-cache-lookupscore-threshold="0.8"embeddings-backend-id="embeddings-backend"embeddings-backend-auth="system-assigned" /><set-backend-servicebackend-id="{backend-id}" /></inbound><outbound><!-- Cache responses for 120 seconds --><azure-openai-semantic-cache-storeduration="120" /><base /></outbound></policies>
Options:
Parameter
Range
Description
score-threshold
0.7-0.95
Higher = stricter matching
duration
60-3600
Cache TTL in seconds
Pattern 3: Token Rate Limiting
Limit tokens per minute to control costs and prevent abuse.
<policies><inbound><base /><set-backend-servicebackend-id="{backend-id}" /><!-- Limit to 500 tokens per minute per subscription --><azure-openai-token-limitcounter-key="@(context.Subscription.Id)"tokens-per-minute="500"estimate-prompt-tokens="false"remaining-tokens-variable-name="remainingTokens" /></inbound></policies>
Options:
Parameter
Values
Description
counter-key
Subscription.Id, Request.IpAddress, custom
Grouping key for limits
tokens-per-minute
100-100000
Token quota
estimate-prompt-tokens
true/false
true = faster but less accurate
Pattern 4: Content Safety
Filter harmful content and detect jailbreak attempts.
Protect MCP servers and tools with request rate limiting.
<policies><inbound><base /><!-- 10 calls per 60 seconds per IP --><rate-limit-by-keycalls="10"renewal-period="60"counter-key="@(context.Request.IpAddress)"remaining-calls-variable-name="remainingCalls" /></inbound><outbound><set-headername="X-Rate-Limit-Remaining"exists-action="override"><value>@(context.Variables.GetValueOrDefault<int>("remainingCalls", 0).ToString())</value></set-header><base /></outbound></policies>
Pattern 6: Managed Identity Authentication
Secure backend access with managed identity instead of API keys.
When user asks to "add my model behind gateway", first discover available models from Azure AI Foundry, then ask which model to add.
Step 1: Discover AI Foundry Projects and Available Models
# Set environment variables
accountName="<ai-foundry-resource-name>"
resourceGroupName="<resource-group>"# List AI Foundry resources (AI Services accounts)
az cognitiveservices account list --query "[?kind=='AIServices'].{name:name, resourceGroup:resourceGroup, location:location}" -o table
# List available models in the AI Foundry resource
az cognitiveservices account list-models \
-n $accountName \
-g $resourceGroupName \
| jq '.[] | { name: .name, format: .format, version: .version, sku: .skus[0].name, capacity: .skus[0].capacity.default }'# List already deployed models
az cognitiveservices account deployment list \
-n $accountName \
-g $resourceGroupName
Step 2: Ask User Which Model to Add
After listing the available models, use the ask_user tool to present the models as choices and let the user select which model to add behind the gateway.
Example choices to present:
Model deployments from the discovered list
Include model name, format (provider), version, and SKU info
Step 3: Deploy the Model (if not already deployed)
# Deploy the selected model to AI Foundry
az cognitiveservices account deployment create \
-n $accountName \
-g $resourceGroupName \
--deployment-name <model-name> \
--model-name <model-name> \
--model-version <version> \
--model-format <format> \
--sku-capacity 1 \
--sku-name <sku>
Step 4: Configure APIM Backend for Selected Model
# Get the AI Foundry inference endpoint
ENDPOINT=$(az cognitiveservices account show \
-n $accountName \
-g $resourceGroupName \
| jq -r '.properties.endpoints["Azure AI Model Inference API"]')
# Create APIM backend for the selected model
az apim backend create \
--resource-group <apim-resource-group> \
--service-name <apim-service-name> \
--backend-id <model-deployment-name>-backend \
--protocol http \
--url "${ENDPOINT}"
Step 5: Create API and Apply Policies
# Import Azure OpenAI API specification
az apim api import \
--resource-group <apim-resource-group> \
--service-name <apim-service-name> \
--path <model-deployment-name> \
--specification-format OpenApiJson \
--specification-url "https://raw.githubusercontent.com/Azure/azure-rest-api-specs/main/specification/cognitiveservices/data-plane/AzureOpenAI/inference/stable/2024-02-01/inference.json"
Step 6: Grant APIM Access to AI Foundry
# Get APIM managed identity principal ID
APIM_PRINCIPAL_ID=$(az apim show \
--name <apim-service-name> \
--resource-group <apim-resource-group> \
--query "identity.principalId" -o tsv)
# Get AI Foundry resource ID
AI_RESOURCE_ID=$(az cognitiveservices account show \
-n $accountName \
-g $resourceGroupName \
--query "id" -o tsv)
# Assign Cognitive Services User role
az role assignment create \
--assignee $APIM_PRINCIPAL_ID \
--role "Cognitive Services User" \
--scope $AI_RESOURCE_ID
Add an API to the gateway from an OpenAPI/Swagger specification, either from a local file or web URL.
Step 1: Import API from Web URL
# Import API from a publicly accessible OpenAPI spec URL
az apim api import \
--resource-group <apim-resource-group> \
--service-name <apim-service-name> \
--api-id <api-id> \
--path <api-path> \
--display-name "<API Display Name>" \
--specification-format OpenApiJson \
--specification-url "https://example.com/openapi.json"
Step 2: Import API from Local File
# Import API from a local OpenAPI spec file (JSON or YAML)
az apim api import \
--resource-group <apim-resource-group> \
--service-name <apim-service-name> \
--api-id <api-id> \
--path <api-path> \
--display-name "<API Display Name>" \
--specification-format OpenApi \
--specification-path "./openapi.yaml"
Step 3: Configure Backend for the API
# Create backend pointing to your API server
az apim backend create \
--resource-group <apim-resource-group> \
--service-name <apim-service-name> \
--backend-id <backend-id> \
--protocol http \
--url "https://your-api-server.com"# Update API to use the backend
az apim api update \
--resource-group <apim-resource-group> \
--service-name <apim-service-name> \
--api-id <api-id> \
--set properties.serviceUrl="https://your-api-server.com"
Convert existing APIM API operations into an MCP (Model Context Protocol) server, enabling AI agents to discover and use your APIs as tools.
Prerequisites
APIM instance with Basicv2 SKU or higher
Existing API imported into APIM
MCP feature enabled on APIM
Step 1: List Existing APIs in APIM
# List all APIs in APIM
az apim api list \
--resource-group <apim-resource-group> \
--service-name <apim-service-name> \
--query "[].{id:name, displayName:displayName, path:path}" \
-o table
Step 2: Ask User Which API to Convert
After listing the APIs, use the ask_user tool to let the user select which API to convert to an MCP server.
Step 3: List API Operations
# List all operations for the selected API
az apim api operation list \
--resource-group <apim-resource-group> \
--service-name <apim-service-name> \
--api-id <api-id> \
--query "[].{operationId:name, displayName:displayName, method:method, urlTemplate:urlTemplate}" \
-o table
Step 4: Ask User Which Operations to Expose as MCP Tools
After listing the operations, use the ask_user tool to present the operations as choices. Let the user select which operations to expose as MCP tools. Users may want to expose all operations or only a subset.
Example choices to present:
All operations (convert entire API)
Individual operations from the discovered list
Include operation name, method, and URL template
Step 5: Enable MCP Server on APIM
# Enable MCP server capability (via ARM/Bicep or Portal)# Note: MCP configuration is done via APIM policies and product configuration
Step 6: Configure MCP Endpoint for API
Create an MCP-compatible endpoint that exposes your API operations as tools:
<policies><inbound><base /><!-- MCP tools/list endpoint handler --><choose><whencondition="@(context.Request.Url.Path.EndsWith("/mcp/tools/list"))"><return-response><set-statuscode="200"reason="OK" /><set-headername="Content-Type"exists-action="override"><value>application/json</value></set-header><set-body>@{
var tools = new JArray();
// Define your API operations as MCP tools
tools.Add(new JObject(
new JProperty("name", "operation_name"),
new JProperty("description", "Description of what this operation does"),
new JProperty("inputSchema", new JObject(
new JProperty("type", "object"),
new JProperty("properties", new JObject(
new JProperty("param1", new JObject(
new JProperty("type", "string"),
new JProperty("description", "Parameter description")
))
))
))
));
return new JObject(new JProperty("tools", tools)).ToString();
}</set-body></return-response></when></choose></inbound></policies>
# Get APIM gateway URL
GATEWAY_URL=$(az apim show \
--name <apim-service-name> \
--resource-group <apim-resource-group> \
--query "gatewayUrl" -o tsv)
# Test MCP tools/list endpoint
curl -X POST "${GATEWAY_URL}/<api-path>/mcp/tools/list" \
-H "Content-Type: application/json" \
-H "Ocp-Apim-Subscription-Key: <subscription-key>" \
-d '{}'
MCP Tool Definition Schema
When converting API operations to MCP tools, use this schema:
{"tools":[{"name":"get_weather","description":"Get current weather for a location","inputSchema":{"type":"object","properties":{"location":{"type":"string","description":"City name or coordinates"}},"required":["location"]}}]}
