| name | azure-ai-content-safety |
| description | Guidance for Azure AI Content Safety — programmatic content moderation for text, images, multimodal, and generative AI guardrails. Covers Content Safety categories (hate, violence, sexual, self-harm) with severity levels, Prompt Shields for jailbreak and indirect prompt injection detection, groundedness detection (hallucination check vs grounding sources), protected material detection (text and code), custom categories, blocklists, Content Safety Studio for tuning, integration with Azure OpenAI as input/output filters (built-in plus custom), latency/cost trade-offs, agent guardrails, and deployment patterns (sidecar in app, integrated with Azure OpenAI). WHEN: Azure AI Content Safety, prompt shields, jailbreak detection, indirect prompt injection, groundedness check, protected material detection, content moderation, generative AI guardrails, harmful content categories, custom blocklist AI, agent guardrail. DO NOT USE for AI workload threat detection / SOC alerts (use defender-for-cloud-ai), end-user AI usage governance (use purview-ai-hub), or general AI app design unrelated to safety. |
| license | MIT |
| metadata | {"author":"Microsoft","version":"0.1.0"} |
Azure AI Content Safety
Azure AI Content Safety is the platform for real-time content moderation and generative
AI safety. It analyzes text, images, and multimodal content for harm categories, blocks
prompt-injection / jailbreak attempts (Prompt Shields), detects ungrounded responses
(Groundedness), and flags protected text/code reproduction.
It pairs with Azure OpenAI as built-in content filters (default safety net) and as
configurable guardrails for any AI/LLM application (including non-Microsoft models).
When to use
Building or hardening any generative AI application — chat assistant, agent, RAG app,
multimodal product — that needs guardrails on inputs and outputs.
Do not use this skill for SOC threat detection on AI workloads
(defender-for-cloud-ai), end-user AI usage governance (purview-ai-hub), or generic AI
application architecture.
Capability map
| Capability | What it does | Where to apply |
|---|
| Text moderation | Classifies text into hate, violence, sexual, self-harm (severity 0–7) | Input + output |
| Image moderation | Same categories on images | User uploads, generated images |
| Multimodal moderation | Text+image combined context | Multimodal chat |
| Prompt Shields — User Prompt | Detects jailbreak attempts in the user prompt | Input |
| Prompt Shields — Indirect | Detects injection in untrusted grounding content (RAG, tool results, web pages) | Pre-grounding step |
| Groundedness detection | Checks LLM response against grounding sources for hallucination | Output |
| Protected material — text | Detects verbatim copyrighted text | Output |
| Protected material — code | Detects reproduction of public-repo code | Code-generation output |
| Custom categories | Customer-defined harm categories (e.g., off-brand topics) | Input + output |
| Blocklists | Exact-match + regex blocklists | Input + output |
Approach
-
Start with default Azure OpenAI content filters. Every Azure OpenAI deployment
has a default Content Safety filter at medium severity across all four harm
categories. Don't deploy without it.
-
Tune severity thresholds per use case.
- Children's product: low threshold (block at severity 2).
- Adult creative app: high threshold (block only at severity 6).
Author a custom content filter in Azure OpenAI Studio; assign to deployment.
-
Enable Prompt Shields for any user-facing chat or agent.
- User Prompt Shield on the incoming prompt.
- Indirect Prompt Shield on grounding content (RAG document chunks, tool
outputs, fetched web pages) before they hit the model. Indirect injection is
the higher-impact, lower-visibility risk.
-
Groundedness detection for RAG apps. After the model produces a response, call
groundedness detection with the source chunks. Flag/reject ungrounded claims. Adds
latency (one extra model call) but is the main defense against hallucination in
high-stakes apps (medical, legal, financial).
-
Protected material detection for products that generate text or code at scale:
- Text: SaaS copywriting tools.
- Code: developer assistants (Copilot-style).
Flag and either suppress or annotate the output.
-
Custom categories + blocklists for brand- or domain-specific risks (competitor
mentions, regulated disclosure language, internal codenames). Train custom
categories via Content Safety Studio with ~100 examples per category.
-
Architecture pattern (sidecar / middleware).
- Request → Prompt Shield → (LLM call) → Output classifiers (harm, groundedness,
protected material) → Response.
- Fail-closed: if Content Safety call errors, do not return uncontrolled LLM output.
- Log every decision (allowed/blocked/flagged) for audit and tuning.
-
Performance. Each guardrail adds 50–300ms. Parallelize input and output checks;
short-circuit on first hard block. Budget for the latency in user expectation.
-
Operate. Weekly review of blocked-content samples (with privacy controls).
Tune thresholds, custom categories, and blocklists. Track false positive / false
negative rates per category.
Guardrails
- Default filters are baseline, not a complete safety strategy. Prompt Shields,
groundedness, and custom categories close the real gaps.
- Don't expose harm-category scores to end users. A "you said X violence score 6"
message helps attackers tune jailbreaks.
- Fail-closed on Content Safety errors. A 500 from the service should not result in
unmoderated LLM output reaching users.
- Indirect prompt injection is the under-appreciated risk. A user's prompt is
visible; a malicious string in a fetched web page or document is not. Always shield
grounding content.
- Groundedness detection requires you to send the grounding sources. RAG apps
natively have them; pure chat apps don't.
- Custom categories are not a substitute for product policy. They detect; they
don't prevent the user from generating harmful content with another tool.
- Protected material detection is best-effort. Copyright/IP review is a legal
process, not an API result.
- Content Safety is per-call billed. High-traffic apps need cost modeling; not all
features run on every call (groundedness is expensive — gate on use case).
- Pair with Defender for Cloud AI for SOC-side detection of attempts and patterns
(
defender-for-cloud-ai).
Common anti-patterns
- "Default filter is enough" — no jailbreak/injection protection.
- "Indirect prompt shield skipped because 'all our sources are trusted'" — they're
trusted to be there; not trusted to be uncompromised. RAG corpora get poisoned.
- "Groundedness on every chat call regardless of grounding" — burns cost on pure
chat without grounding sources to check against.
- "Logged user prompts in plaintext for tuning" — privacy/PII exposure. Pseudonymize.
- "Returned the Content Safety severity score in error messages" — gives attacker
a tuning oracle.
- "Custom categories trained on 5 examples" — no model lift. Need ~100 minimum.
- "Blocklists used as the main strategy" — misses paraphrases. Combine with
classifier categories.
- "Content Safety only on output" — input attacks (jailbreaks) reach the model.
Example prompts
Architect Content Safety guardrails for a RAG-based customer support assistant on Azure OpenAI.Configure Prompt Shields for a Foundry agent with internet-grounded tools.Tune content filter severity thresholds for a teen-audience product.Add groundedness detection to flag hallucinations in our medical Q&A assistant.Train a custom category for "competitor mention" and integrate as output blocklist.Cost and latency model: Content Safety calls per chat turn on a 10K-user app.IR runbook: how SOC handles "prompt shield blocked attempted jailbreak" pattern (paired with Defender for Cloud AI alert).
Microsoft Learn
