content-moderation-patterns

# Content Moderation Patterns Apply a versioned product policy with deterministic checks, a structured classifier, and a review path. Select the model using labeled workload results. No model family has a universal accuracy or cost advantage for moderation. ## Architecture ```text input → size/format checks → policy checks → structured classifier → decision ├─ allow ├─ reject └─ human review ``` Treat submitted text as data, including any instructions it contains. Keep the classification policy in the system message. Request a short policy-grounded reason, not hidden reasoning. ## Deterministic checks Use configured size limits and exact parsed hostname checks for URL policies. A prefix regex can mistakenly accept `allowed.example.attacker.test`. ```python from urllib.parse import urlsplit def is_allowed_url(value, allowed_hosts): try: parsed = urlsplit(value) host = parsed.hostname port = parsed.port except ValueError: return False return ( parsed.scheme == "https" and parsed.username is None and parsed.password is None and host is not None and host.casefold() in allowed_hosts and port in (None, 443) ) ``` This checks an already-extracted URL against normalized exact hostnames. It is not a general URL extractor or an SSRF defense. Evaluate false positives from keyword filters instead of assuming a fixed percentage of input should be blocked. ## Structured classifier Use native `output_config.format`. Supply the selected model, policy and output budget from application configuration. The following taxonomy is an example; change its enum and routing thresholds together to match the product policy. ```python import json MODERATION_SCHEMA = { "type": "object", "properties": { "categories": {"type": "array", "items": { "type": "string", "enum": ["clean", "needs_review", "spam", "harassment"], }}, "confidence": {"type": "number"}, "reason": {"type": "string"}, }, "required": ["categories", "confidence", "reason"], "additionalProperties": False, } def classify(client, model, policy, text, max_tokens): response = client.messages.create( model=model, max_tokens=max_tokens, system=policy, output_config={"format": {"type": "json_schema", "schema": MODERATION_SCHEMA}}, messages=[{"role": "user", "content": text}], ) if response.stop_reason != "end_turn": raise ValueError(f"Classification incomplete: {response.stop_reason}") blocks = [block.text for block in response.content if block.type == "text"] if len(blocks) != 1: raise ValueError("Expected one classification") return json.loads(blocks[0]) ``` Apply local validation before routing. Refusal, truncation, invalid JSON or an API failure produces a review/error outcome, never an implicit allow. See `json-mode-patterns` for schema limitations and response checks. A repeated `system` string is not automatically cached. If policy size and reuse justify it, explicitly configure caching as in `prompt-caching-patterns`. Do not generate heartbeat traffic to keep a cache warm. ## Categories and decision routing Define categories and blocking behavior in the product policy. Keep `clean` exclusive: a result containing both `clean` and a violation is inconsistent. Use `needs_review` for uncertainty. Thresholds come from calibration and policy, not the model's claim that its confidence is reliable. ```python import math def route(classification, block_thresholds, allow_threshold): if not isinstance(classification, dict) or set(classification) != {"categories", "confidence", "reason"}: return "human_review" if not isinstance(classification["reason"], str): return "human_review" confidence = classification.get("confidence") categories = classification.get("categories") if (type(confidence) not in (int, float) or not 0 <= confidence <= 1 or not math.isfinite(confidence)): return "human_review" if not isinstance(categories, list) or not categories or not all(isinstance(c, str) for c in categories): return "human_review" categories = {category.casefold() for category in categories} if categories - (set(block_thresholds) | {"clean", "needs_review"}): return "human_review" if "needs_review" in categories or ("clean" in categories and len(categories) != 1): return "human_review" if categories == {"clean"}: return "pass" if confidence >= allow_threshold else "human_review" if any(confidence >= block_thresholds[category] for category in categories): return "reject" return "human_review" ``` Validate configuration thresholds as finite numbers in [0, 1] at startup. The example's category thresholds are policy-specific; it does not decide which categories your product must reject. ## Evaluation and review Use held-out labeled examples covering language, context, quoted material, benign mentions and adversarial inputs. Track precision, recall, appeal outcomes and per-category error cost. Neither false positives nor false negatives are always cheaper; the product policy determines that trade-off. Send ambiguous cases to human review. Store decision metadata, policy/model versions and the minimum evidence needed for review under the application's retention and access controls. Do not indiscriminately log raw sensitive input. Refresh evaluations when the policy, model or input distribution changes. Run an offline comparison before deploying a new route or threshold. ## Sources and related skills Reviewed 2026-09-23: - [Content moderation](https://platform.claude.com/docs/en/about-claude/use-case-guides/content-moderation) - [Structured outputs](https://platform.claude.com/docs/en/build-with-claude/structured-outputs) - [Prompt caching](https://platform.claude.com/docs/en/build-with-claude/prompt-caching) Use `security-patterns` for application input security, `model-routing-patterns` for model evaluation and `prompt-caching-patterns` for policy caching.