llm-gateway-governance

name	llm-gateway-governance
description	Build a governed LLM gateway that lets internal developers use code agents (Claude Code, Codex) against Amazon Bedrock through a single control point — enforcing identity, per-user virtual keys, model/cost tiering, Bedrock Guardrails, managed web search, and tracing. Supports two developer-auth modes: IAM Identity Center organization-instance SSO (`org-sso`, permission sets + AWSReservedSSO roles) and `cognito-native` (an Amazon Cognito User Pool as the sole identity source, Cognito User Pool Groups as teams) for environments where org-sso is not usable — e.g. an IdC account instance, which cannot host the SAML app that IdC federation would require. Generates AWS CDK (TypeScript): VPC + Aurora Serverless v2 + Bedrock Guardrail + AgentCore Web Search gateway + LiteLLM on ECS Fargate + optional Langfuse + a token service + an internet-facing, SG CIDR-restricted ALB edge (config-driven TLS via certMode: acm/http) + cross-region Mantle (GPT-5.x) VPC peering. Use when the user asks to "build an LLM gateway", "govern Bedrock access for developers", "central proxy for Claude Code / Codex", "LiteLLM on AWS", or "SSO/Cognito virtual keys for LLMs".
license	MIT
metadata	{"version":"1.1","author":"aws-solution-skills","reference-implementation":"llm-gateway-multi-agent"}

LLM Gateway Governance

Purpose

Generate a production-shaped code-agent governance gateway on AWS: a single LiteLLM proxy (on ECS Fargate) that internal developers reach with per-user virtual keys minted from a verified identity, with Bedrock Guardrails, model/cost tiering, managed web search (AgentCore), network isolation, and observability. The solution supports two developer-auth modes: org-sso for IAM Identity Center organization instances using permission-set roles, and cognito-native for environments where org-sso is not usable — an Amazon Cognito User Pool is the sole identity source (no external IdP, no IdC federation) and native Cognito User Pool Groups are the teams. cognito-native is the correct choice for an IdC account instance (a member/standalone account whose IdC cannot host a SAML customer-managed application, so IdC-federated login is impossible at the AWS level) or any account with no usable IdC at all. It lets an org give developers Claude Code / Codex access to Amazon Bedrock while centrally controlling who, which models, how much, what content, and with what audit trail — without requiring LiteLLM Enterprise.

This skill produces AWS CDK (TypeScript), Lambda (Python), the LiteLLM container config, and developer-onboarding scripts, customized to the user's domain (custom domain or not, Langfuse or not, which models, which tiers, which region, web search on/off).

Knowledge sources

Read these before generating. All real knowledge lives in shared/:

• shared/reference/prerequisites.md — check before Phase 1: local tooling (Docker, Node, CDK CLI, AWS CLI), AWS account access/IAM, Bedrock model access, IdC readiness, custom-domain/Route53 needs — incl. the two image-build paths (local Docker default / CodeBuild when Docker can't run locally)
• shared/reference/architecture.md — the 10-stack architecture, request lifecycle, and the "why"
• shared/reference/decision-tree.md — map Discovery answers → config/dev.json + stack choices (region, web search, Mantle)
• shared/reference/aws-services.md — service/model catalog (verify volatile IDs via MCP)
• shared/reference/constraints.md — failure modes & gotchas (bootstrap, edge TLS/certMode, Mantle guardrail, secrets, AgentCore web search, Mantle peering, Marketplace, region)
• shared/reference/sso-setup.md — IAM Identity Center organization instance Discovery + permission-set provisioning + generated config.sso block & AuthStack outputs
• shared/reference/account-instance-setup.md — cognito-native setup for IdC account instances (and any account without usable org-sso): why IdC federation is impossible on an account instance (no SAML customer-managed app), Cognito User Pool as sole identity source, User Pool Groups as teams, cognito:groups claim → LiteLLM team, and cross-platform llmgw-login
• shared/reference/litellm-admin-guide.md — post-deploy operations: Admin UI login, creating teams/users (org-sso permission set → team, cognito-native Cognito User Pool Group → team), checking logs/traces (LiteLLM UI, Langfuse, CloudWatch), applying per-team/per-key budgets, offboarding (revoke virtual keys — IdC/Cognito removal alone does NOT cut off access; see constraints.md → "Virtual-key lifetime ≠ SSO session")
• shared/patterns/cdk-stacks.md — full CDK source for the platform stacks + interfaces + config validation
• shared/patterns/agentcore-websearch.md — AgentCore Web Search gateway stack (Gateway + built-in web-search connector) + LiteLLM wiring (replaces Tavily)
• shared/patterns/mantle-peering.md — Bedrock Mantle in us-east-1 via cross-region VPC peering (MantleNetworkStack + MantlePeeringRoutesStack)

Workflow

Phase 1: Discovery (ask only what you don't know)

0. Prerequisites check — before asking anything else, confirm the operator has what shared/reference/prerequisites.md lists: Docker daemon running, Node/CDK CLI/AWS CLI v2 installed, deploy access to the target account, Bedrock model access requested (gateway region for Claude, us-east-1 for GPT-5.x/Mantle), and IdC enabled (required for the SSO path — see #6). Surface any gap now rather than discovering it mid-deploy. Docker is the ONLY waivable item: if docker info succeeds, use the default local fromAsset() build; if Docker cannot run on this machine at all (e.g. managed Windows laptop — WSL2/Hyper-V install needs admin + a reboot) set litellm.imageBuild.mode='codebuild' (conditional ImageBuildStack builds the image on native ARM in CodeBuild — cdk-stacks.md §4-1, 3-step deploy order). Everything else in the local toolchain is a hard prerequisite — do NOT offer a remote/EC2 deploy host as a workaround: running the skill off the operator's machine breaks operator-locality (the albIngressCidrs Discovery answer is the operator's real egress IP, unknowable from a remote host; the generated onboarding bundle — including the secret-bearing admin-onboarding.html — lands on the remote host and needs yet another transfer channel to reach anyone).

1. Edge TLS (certMode)? CloudFront is removed — the ALB is the edge, always internet-facing, always SG CIDR-restricted. Choose: acm (own a domain + Route53 hosted zone → HTTPS:443 with a public ACM cert; ✅ recommended / PROD) or http (no domain → HTTP:80, no cert; ⛔ the virtual key and prompt/response bodies travel plaintext on the wire — PoC-only, a GATE-1 acknowledgement item). Both modes: ask which source CIDRs may reach the ALB (litellm.albIngressCidrs, required — the SG allowlist is the primary access control). 0.0.0.0/0 with http means the plaintext endpoint is reachable from the whole internet → its own explicit GATE-1 acknowledgement.

2. Models? Which Claude / GPT(Mantle) models? GPT tiers: gpt-5.6-sol (flagship coding/agentic, 1M context) / gpt-5.6-terra (balanced) / gpt-5.6-luna (economy/latency) / gpt-5.5 (proven flagship) / gpt-5.4 (proven economy). ⛔ If any gpt-5.6-* alias is selected, tell the user now that it carries a before developer onboarding (real-deploy incident history: the Codex tool-type 400 — since fixed server-side — plus open Codex-CLI issues like the ≥ 0.147 -namespace collision), and keep / in the model list as the fallback pair — see → the GPT-5.6 gate. ⚠️ () — a real deploy had and profiles but NO / at all; if absent, say so explicitly rather than offering a fallback that cannot be routed. : which teams need their own budget cap + model allowlist? In , each permission set maps 1:1 to a same-named LiteLLM team; in , each Cognito User Pool Group name maps 1:1 to a same-named LiteLLM team. Discovery should capture the team names plus initial / seeds. Onboarding additional teams later is then a console action (IdC permission set / Cognito group + LiteLLM Admin UI) only, never a code change. ("economy/standard" is just one illustrative naming — not a required split; name groups/permission sets/teams after real orgs/teams.)

⛔ GATE 1: summarize requirements + the resulting config/dev.json (incl. awsRegion, authMode, sso or cognitoNative — for cognito-native also the initial user(s) to create post-deploy, agentcore, mantle, litellm.certMode + litellm.albIngressCidrs — with the plaintext acknowledgement if http, and — if any Fable/Mythos-class model is requested — the provider_data_share opt-in acknowledgement from #2b below); await confirmation.

Phase 2: Architecture Design

• Apply shared/reference/decision-tree.md to choose certMode (acm/http), enableLangfuse (acm only — Langfuse UI needs a real domain/cert; http → CloudWatch only), tiers, capacity, region, web search, Mantle peering.
• Verify model IDs + regional availability via AWS Knowledge MCP (aws___search_documentation, aws___get_regional_availability) — never hard-code stale IDs. Resolve each Claude model's actual inference-profile ID with aws bedrock list-inference-profiles; do NOT assume a us. prefix. Recent (2026) models (Opus 4.8, Sonnet 5, Haiku 4.5, Fable 5) exist only as global. GLOBAL profiles — a us. call returns The provided model identifier is invalid. Confirm Web Search + Mantle PrivateLink in us-east-1.
• Produce the stack list (Network → Data → Guardrail → AgentCoreGateway(us-east-1) → LiteLLM [ALB edge per certMode] → Langfuse? [acm only] → Auth(org-sso API Gateway IAM authorizer or cognito-native Cognito User Pools authorizer) → Observability → MantleNetwork(us-east-1) → MantlePeeringRoutes) and a cost estimate. There is no CdnStack — CloudFront is removed; the ALB is the edge.

⛔ GATE 2: present architecture + cost; await confirmation.

Phase 3: Code Generation

• Emit from shared/patterns/: bin/app.ts, lib/*-stack.ts (incl. agentcore-gateway-stack.ts, mantle-network-stack.ts, mantle-peering-routes-stack.ts — and image-build-stack.ts when litellm.imageBuild.mode='codebuild', cdk-stacks.md §4-1), lib/interfaces.ts, lib/config/{constants,schema}.ts (incl. awsRegion, authMode, sso, cognitoNative, agentcore, mantle), lib/nag-suppressions.ts, lambda/token-service/handler.py, lambda/db-init/handler.py, services/litellm/{config.yaml,Dockerfile,entrypoint.sh,callbacks/*}, scripts/*, templates/*.
• Token Lambda auth adapter: generate a clear branch by authMode. org-sso preserves API Gateway IAM + AWSReservedSSO_ ARN parsing. cognito-native uses an API Gateway Cognito User Pools authorizer (which validates the JWT signature/issuer/audience/expiry before the Lambda runs) and reads the verified cognito:groups claim from requestContext.authorizer.claims — no Identity Store lookup, no identitystore:* IAM at all. Never mix the trust anchors.
• Token Lambda tiering (Hard Constraints #13): generate _resolve_team_id so the authorization unit maps 1:1, unbranched, to a same-named LiteLLM team — permission set for org-sso, Cognito User Pool Group name for cognito-native; never emit an if team in {...} per-org branch. Seed TIER_CONFIG only with the team(s) named in Discovery (as a starter models/max_budget for that team's first-ever creation); this is what keeps future onboarding console-only (permission set / Cognito group + LiteLLM Admin UI), not a Lambda redeploy. See shared/patterns/lambda-handlers.md.
• No CdnStack (CloudFront removed). Emit the LiteLLM ALB edge per certMode (Hard Constraints #1/#5): →internet-facing HTTPS:443 + ACM (regional — , ARN or Route53 DNS-issued) + Route53 alias + HTTP→443 redirect; →internet-facing HTTP:80 (no cert). In modes the public ALB's SG ingress is restricted to (no AWS WAF, no Custom Resource, no SSM tunnel). Always keep a separate for the Token Service (SSM URL unchanged). ALB (Hard Constraints #10). Langfuse (acm only) gets its own public ALB.

Phase 4: Validate

• Toolchain alignment first: aws-cdk-lib + cdk-nag must be co-resolved (an old lib pin + open nag range → ERESOLVE), and the aws-cdk CLI must support the library's cloud-assembly schema (schema version mismatch at synth means: upgrade the CLI to at least the version the error names). See constraints.md.
• Pinned-image CLI pre-check: verify every flag the entrypoint passes against the pinned LiteLLM image — docker run --rm --entrypoint litellm <image> --help (seconds; skipping this cost a full deploy-rollback cycle in a real deploy: --forwarded-allow-ips does not exist → exitCode 2 at boot).
• npm install && npm run typecheck && npx cdk synth --all.
• Resolve cdk-nag findings: suppress with written justification (PROD TODO) where they are intentional dev tradeoffs (e.g., IAM5, ELB2, EC23 — no CFR* findings exist since CloudFront is removed), fix genuine issues.
• Verify data.engineVersion exists in the target region; confirm mantle.peerVpcCidr does not overlap network.vpcCidr.

Phase 5: Deploy

• Ensure Docker is running (LiteLLM image builds via fromAsset). ⚠️ On an x86_64 deploy host (Windows, Intel Mac, x86 CI), the generated CDK pins the build to Platform.LINUX_ARM64 to match the ARM64/Graviton Fargate task — but that cross-build needs QEMU emulation (Docker Desktop has it built in; native Linux Docker needs docker run --privileged --rm tonistiigi/binfmt --install arm64 once). If a generated project's ContainerImage.fromAsset() is missing the platform option, the build silently produces an amd64 image and cdk deploy still succeeds — the failure only surfaces afterward as ECS tasks crash-looping with exec format error. See shared/reference/constraints.md → "Docker build architecture mismatch on x86 hosts". imageBuild.mode='codebuild' deploys instead follow the 3-step order (cdk deploy --all alone ships no image — StartBuild is not a CloudFormation resource): deploy ImageBuildStack → aws codebuild start-build + poll to SUCCEEDED + aws ecr describe-images confirms the tag → deploy the remaining stacks (cdk-stacks.md §4-1; skipping step 2 surfaces late as CannotPullContainerError).
• Bootstrap us-east-1 AND the gateway region: cdk bootstrap aws://<acct>/<awsRegion> aws://<acct>/us-east-1 — if leftovers block it, use a custom qualifier (Hard Constraints #2). ⚠️ "no changes" is not proof of a correct bootstrap: the account may be bootstrapped under a different qualifier — check aws cloudformation describe-stacks --stack-name CDKToolkit → the Qualifier parameter, and set @aws-cdk/core:bootstrapQualifier in cdk.json to match (else deploy fails assuming cdk-hnb659fds-* roles that don't exist).
• cdk deploy --all --require-approval never --outputs-file outputs.json. ⚠️ A subset deploy must NOT write the canonical outputs.json — cdk deploy <one-stack> --outputs-file outputs.json OVERWRITES the file with only that stack's outputs, silently deleting every other stack's (real-deploy incident: a missing Cognito host produced https:///oauth2/authorize far downstream). Use --outputs-file /tmp/<stack>-outputs.json for subset deploys, and repopulate the canonical file with . — the running deploy keeps using the image built from the old source; wait for it to finish (or cancel), then start a fresh deploy. A container dying in seconds (exitCode N) → check , not ECS events.

Phase 6: Developer Onboarding — ALWAYS end by generating the two HTML docs

After a successful deploy you MUST generate two self-contained HTML docs with scripts/gen-onboarding.py (from cdk deploy --outputs-file outputs.json), per shared/patterns/developer-onboarding.md §6. This replaces the old inline markdown guide. The base URL is the GatewayUrl output (the ALB domain — CloudFront is removed).

• developer-setup.html — hand to every developer. No admin secrets. Script-first: the doc instructs developers to RUN the generated .sh/.ps1 launchers — never to hand-edit config files (the merged JSON/TOML content appears only as a reference appendix). Filled with deployed values:
  • Step 1 — one-shot setup: run scripts/setup-developer.sh (Windows: scripts/setup-developer.ps1). It merges ~/.claude/settings.json (ANTHROPIC_BASE_URL=<gateway-url>, AWS_REGION=<deploy-region>, all four model aliases incl. ANTHROPIC_DEFAULT_FABLE_MODEL — omitting Fable hides that tier from /model —, apiKeyHelper → the token helper, permissions.deny: ["WebSearch"] — use the AgentCore Web Search MCP instead of the unsupported built-in) and ~/.codex/config.toml ([model_providers.llm-gateway]: base_url=<gateway-url>/v1, wire_api=responses, model=<gpt-alias>, web_search="disabled", .auth.command → the token helper), backing both up to *.llmgw-backup-<timestamp> first (merge, never overwrite).
  • Step 2 — login: org-sso → aws sso login --profile llm-gateway (Token Service accepts only AWSReservedSSO_ principals; a non-SSO caller is rejected 403 by design). cognito-native → llmgw-login / llmgw-login.ps1 (Cognito Hosted UI; aws sso login is NOT used).
  • http only: the base URL is http://<alb-dns> (plaintext — reachable only from the SG albIngressCidrs allowlist; no cert trust, no tunnel).
  • Step 3 — web search MCP (client-side, required): claude mcp add-json at <gateway-url>/mcp/ with headersHelper → gateway_auth.py mcp-headers (rotating virtual key injected dynamically). Tool: websearch-web-search-tool___WebSearch.

Hard Constraints

1. The ALB is the edge — TLS via certMode, CloudFront removed. The ALB is always internet-facing and always SG CIDR-restricted (litellm.albIngressCidrs). acm needs a domain (regional public ACM cert — an existing ARN, or Route53 DNS-issued) → HTTPS:443. http needs no domain → HTTP:80, ⛔ plaintext on the wire, PoC-only (GATE-1 acknowledgement). There is no self-signed mode, no CloudFront, no internal/VPN exposure variant, no SSM tunnel. (See constraints.md.)

2. Bootstrap collisions → bootstrap with a custom --qualifier + @aws-cdk/core:bootstrapQualifier in cdk.json; delete any empty REVIEW_IN_PROGRESS CDKToolkit stack. Bootstrap both us-east-1 and the gateway region. Never delete other apps' bootstrap resources.

3. Bedrock Guardrails are bedrock-runtime only — never attach them to bedrock_mantle/ (GPT) models; cover Mantle with LiteLLM hide-secrets and document the gap.

4. Never hard-code secrets. Master key, DB creds, Langfuse admin password + project keys all go through Secrets Manager. Shared LiteLLM↔Langfuse trace keys must live in a stack created before LiteLLM and be injected with ecs.Secret.fromSecretsManager — never as plaintext environment values.

5. The public ALB is the edge (internet-facing in both modes: acm HTTPS:443, http HTTP:80) — access control is the albIngressCidrs SG allowlist (no AWS WAF; mention WAF only as an optional PROD hardening note for a fully open acm deployment). Always keep a separate internal ALB (:4000) for the Token Service (its SSM URL is unchanged, no hairpin). Langfuse (acm only) has its own public ALB. Never expose the internal ALB or the :4000 listener to the internet. ⚠️ Pass open: false on EVERY addListener() — CDK's default open: true silently appends a 0.0.0.0/0 ingress rule for the listener port to the ALB SG, defeating albIngressCidrs (real-deploy incident; synth-time invisible). After any deploy touching listeners, verify the deployed SG () contains only the intended CIDRs. See .

Generation rules

• Single source of truth in lib/config/constants.ts; runtime-validate config/dev.json (lib/config/schema.ts, incl. awsRegion/authMode/sso/cognitoNative/agentcore/mantle) so deploy fails fast.
• Append-only cross-stack *Exports interfaces; runtime-only wiring via SSM by name; cross-region wiring via crossRegionReferences: true.
• SG ownership rule (cyclic-reference, real-deploy lesson): NetworkStack owns only shared-infra SGs (Aurora/endpoints/Lambda) with CIDR-based ingress; app stacks (LiteLLM/Langfuse) own their service + ALB SGs. Never let a Network-owned SG hold a rule referencing an app-stack SG — and remember addTargets() auto-wires SG rules. See constraints.md + cdk-stacks.md §1.
• Onboarding scripts merge, never overwrite, the user's shared config files (~/.claude/settings.json, ~/.codex/config.toml): backup + JSON load-update-save / TOML block-replace only (constraints.md). The merge logic exists once, in gateway_auth.py setup — .sh/.ps1 are thin launchers; never re-implement it in shell/PowerShell. The merge also removes direct-Bedrock remnants (CLAUDE_CODE_USE_BEDROCK, AWS_BEARER_TOKEN_BEDROCK, top-level model) — leaving them causes 403/"team not allowed to access model" (real Windows incidents, constraints.md). Keep :? error messages apostrophe-free (macOS bash 3.2).
• Windows client rules (constraints.md → "Cross-platform client onboarding"): every .ps1 launcher ends with exit $LASTEXITCODE (PS 5.1 does not propagate native exit codes) and runs the Python-3 preflight — probe py -3 then python with --version, accept only a real Python 3.x banner + exit 0, else an actionable stderr error (winget install Python.Python.3.12) + exit 1; preferring py -3 alone is not enough — on a clean box is absent AND is the Store-alias stub (silent exit 9009 = "script does nothing", real incident); config helper commands use + the copy that installs; (no shell; joined string → os error 123 → 401); user-only file protection needs (chmod is a no-op on Windows).

When to call MCP

Trigger	MCP call
Choosing/confirming model IDs (Claude + GPT-5.x)	aws___search_documentation, aws___get_regional_availability
Resolving a Claude model's inference-profile ID (global. vs us.)	(CLI) aws bedrock list-inference-profiles in the target region
Confirming Fable/Mythos data-retention (provider_data_share) requirement	aws___search_documentation (data-retention + model card)
Confirming AgentCore Web Search is in us-east-1 / CFN resources exist	aws___get_regional_availability (AWS::BedrockAgentCore::Gateway, ...::GatewayTarget)
Confirming bedrock-mantle PrivateLink + GPT-5.x in us-east-1	aws___get_regional_availability; CLI ec2 describe-vpc-endpoint-services
Confirming a service/feature is in the target region	aws___get_regional_availability
Aurora engine version validity	(CLI) rds describe-db-engine-versions in the target region