citadel-hub-deploy

# Citadel Hub Deploy — Layer 1 Governance Hub > **Status:** Public Preview wrapper around the > [Azure-Samples / ai-hub-gateway-solution-accelerator] branch > `citadel-v1` (MIT). The accelerator is the canonical source; this skill > never forks or vendors its Bicep — it pins to a known-good commit, ships > 3 curated AZD env profiles, and wires the deployment into the > awesome-gbb conventions (tenant isolation, MCAPS pilot tagging, > spoke-aware networking). > > **Pinned upstream:** `63f0f812474e713916dc909494d655246783a1d9`; > see [`references/upstream-pin.md`](references/upstream-pin.md). > **Validation boundary:** the current pin is build-validated and was > live-deployed with the lean pilot overlay on 2026-08-19. Core APIM model > discovery and chat succeeded. Positive client-credentials JWT validation > remains unverified because the validation tenant's Conditional Access policy > blocked workload token issuance; see > [`references/live-audit-notes.md`](references/live-audit-notes.md). [Azure-Samples / ai-hub-gateway-solution-accelerator]: https://github.com/Azure-Samples/ai-hub-gateway-solution-accelerator/tree/citadel-v1 --- ## 1. Why this matters Most "Foundry pilot" decks stop at "deploy a Foundry account, run an agent". That works for **one** team and **one** use case. The moment a second team needs the same model, you hit five hard problems: 1. **Cost attribution.** Whose subscription pays for which call? 2. **Quota fairness.** One team's batch run starves another's chat. 3. **Policy uniformity.** PII redaction, content safety, model allow-lists — defined once, enforced everywhere. 4. **Auditability.** Who called what model, when, with which prompt? 5. **Backend abstraction.** Switching a model from PTU → PayAsYouGo shouldn't require every spoke to re-deploy. The Citadel Governance Hub is the AI Apps GBB reference design that solves all five at the platform level. APIM in front of every model backend, Cosmos for usage telemetry, Logic App for billing aggregation, Event Hub for streaming events, and a **per-team Access Contract** (APIM Product) that gives each spoke its own subscription key, scope, and policy bundle. Without this you end up with N spoke projects each negotiating their own Foundry quota and their own PII policy — and your CISO finds out on day 91. ``` ┌──────────────────────────────────────────────────────────┐ │ Layer 1 — Governance Hub (this skill deploys it) │ │ │ │ ┌────────────┐ ┌──────────┐ ┌──────────────────┐ │ │ │ APIM v2 │ ←→ │ Cosmos │ ←→ │ Foundry (×N) │ │ │ │ AI Gateway │ │ Usage DB │ │ Multi-region │ │ │ └────────────┘ └──────────┘ └──────────────────┘ │ │ ↑ ↑ │ │ │ │ │ │ ┌────┴───┐ ┌──────────┴────┐ │ │ │ Event │ │ Logic App │ │ │ │ Hub │ │ Usage │ │ │ └────────┘ │ Aggregation │ │ │ └───────────────┘ │ └──────────────────────────────────────────────────────────┘ ↑ ↑ ↑ │ per-team access contract │ │ │ (APIM Product + sub key) │ │ ┌────────┴──────┐ ┌─────────┴─────┐ ┌──────┴────┐ │ Spoke project │ │ Spoke project │ │ Foundry │ │ team A │ │ team B │ │ Workspace │ │ (use citadel- │ │ (use citadel- │ │ (use │ │ spoke-onbrd) │ │ spoke-onbrd) │ │ spoke…) │ └───────────────┘ └───────────────┘ └───────────┘ ``` --- ## 2. What `citadel-hub-deploy` does (and what it doesn't) ### Does - Captures the upstream `azd` template at a **pinned SHA** (see `references/upstream-pin.md`); never silently rolls forward. - Wraps the deployment in a **tenant-isolated, assertion-gated** workflow (see `azure-tenant-isolation`). - Ships **3 curated AZD environment profiles** in `references/profiles/`: - `pilot-quickstart.env` — Developer SKU, lean optional-service overlay - `enterprise-baseline.env` — Standard v2, production-grade, BYO Log Analytics - `vnet-isolated-spoke-aware.env` — BYO VNet + DNS, pre-wired for `foundry-vnet-deploy` spokes - Documents the current **12-scenario upstream validation sequence** and the four-notebook strongly recommended baseline. - Documents the **post-deploy hand-off** to `citadel-spoke-onboarding` (per-team access contracts) and `foundry-agt` (in-process governance). ### Doesn't - **Doesn't fork or vendor** the upstream Bicep. The deployment uses a detached Git checkout at the exact pinned SHA. `azd init` cannot pin a commit when given `--branch`, so branch-based initialization is forbidden. - **Doesn't onboard spokes.** That's `citadel-spoke-onboarding` — a single `az deployment sub create` against `bicep/infra/citadel-access-contracts/main.bicep`. - **Doesn't add in-process governance.** That's `foundry-agt` — runs inside the agent process, before/after every tool call. - **Doesn't manage post-deploy upgrades.** The `bicep/infra/apim-gateway-upgrade/` flow (StandardV2 → newer SKUs, policy fragment refresh) is upstream-owned. - **Doesn't onboard LLM backends.** That's the `validation/llm-backend-onboarding-runner.ipynb` notebook upstream. | Want to do this | Use this skill instead | |---|---| | Wire your agent project into a deployed hub | `citadel-spoke-onboarding` | | Add per-tool-call governance inside MAF/Foundry agents | `foundry-agt` | | Deploy a single-resource Foundry inside a private VNet (no APIM) | `foundry-vnet-deploy` | | Switch tenants, isolate az/azd config dirs | `azure-tenant-isolation` | | Apply MCAPS pilot tagging conventions (`SecurityControl: Ignore`, `AZURE_TAGS`) | `azd-patterns` | | Get App Insights traces from the deployed hub into your spoke | `foundry-observability` | --- ## 3. When NOT to deploy a Citadel Hub The hub is opinionated: APIM + Foundry control plane + Cosmos + Event Hub + Logic App + private networking, with Redis and API Center optional by profile. That's **~$800-2,500/month baseline cost** in enterprise config (see `guides/citadel-sizing-guide.md` upstream) and 30-45 minutes of APIM provisioning before the first request can flow. Don't deploy a hub when: - **Single-team pilot, single-use-case PoC.** You don't need APIM arbitration if there's only one consumer. Use `microsoft-foundry` + a direct AOAI/AI Services connection. - **Dev-time Foundry exploration.** Engineers spinning up sandbox Foundry workspaces shouldn't pay for a shared APIM. Use `foundry-vnet-deploy` for private networking instead. - **Budget below $1k/mo.** Even the `pilot-quickstart` profile (Developer SKU APIM, no SLA) lands around $200-400/mo with realistic usage, before Foundry model burn. If you can't justify that for governance, you probably shouldn't be running production agents on any platform. - **Pure offline / batch workloads.** No runtime to govern → no gateway needed. `foundry-evals` + direct backend calls suffice. - **You're inside a Landing Zone with a pre-existing hub.** Reuse it via `citadel-spoke-onboarding`. Don't deploy a parallel hub. --- ## 4. Stakeholder TL;DR - **Engineer:** "Clone upstream, detach and verify the pinned SHA, select a profile, then run `azd up` from that checkout. Profile picks the SKU/network shape. 30-45 min wall clock. Don't forget tenant isolation." - **Architect:** "Layer 1 of the 4-layer Citadel platform. APIM is the gateway plane; spokes connect via per-team access contracts (Bicep-driven). Pairs with `foundry-agt` for in-process defence in depth. Telemetry sinks: 3 App Insights workspaces + 1 Log Analytics + Cosmos `usage-db`." - **Compliance:** "PII redaction (Azure AI Language) + Content Safety + JWT-enforceable RBAC + per-team subscription keys with audit trail in Cosmos + private endpoints on every backend service. Documented in `guides/pii-masking-apim.md` and `guides/jwt-client-identity-permissions.md` upstream." - **Seller:** "One repeatable Bicep deployment that checks the platform-team's first 5 boxes (cost attribution, quota fairness, policy uniformity, audit, backend abstraction) plus the unified-ai-api wildcard route lets you onboard AOAI, Foundry, and Gemini behind one developer-friendly endpoint. Demo runs against the deployed hub via `validation/citadel-universal-llm-api-all-models-tests.ipynb`." --- ## 5. Quickstart <!-- <HARD-GATE> STOP. Before running ANY azd or az command in this section: 1. You MUST have selected a profile (pilot-quickstart, enterprise-baseline, or vnet-isolated-spoke-aware). Do NOT deploy without a profile. 2. You MUST have set AZURE_CONFIG_DIR and AZD_CONFIG_DIR per azure-tenant-isolation. A Citadel hub costs $200-1000+/mo — deploying to the wrong subscription is expensive. 3. You MUST assert the exact tenant GUID and subscription GUID immediately before every deploy or mutating post-deploy command. If any of these are not done, STOP and complete them first. </HARD-GATE> --> > **TENANT ISOLATION FIRST.** Per `azure-tenant-isolation`, set both > `AZURE_CONFIG_DIR` and `AZD_CONFIG_DIR` to per-tenant directories > **before** any `az` / `azd` command. Then run the two-layer assertion > (`az account show --query tenantId / id`, plus the selected azd environment) > immediately before every `azd up`, `setup.ps1`, rollback, or destructive > Azure operation. Display names are not identity checks. Without > these, you risk deploying a $1k+/mo hub into the wrong subscription. ### Path A — Pilot Quickstart (lean non-production overlay) Goal: lean non-production hub. Developer SKU APIM is public; data-plane backends stay private. Redis, API Center, Search, Document Intelligence, dashboards, and Foundry network injection are disabled. Keep the upstream default `foundryNetworkInjectionEnabled=false`; enabling it without the full BYO Standard Agent dependency set fails. ```bash set -euo pipefail # 0. Set the path to your awesome-gbb checkout (or `~/.copilot/skills` # user-scope mirror) so the .env profiles below resolve. SKILL_DIR="$HOME/.copilot/skills/citadel-hub-deploy" # or your repo path # 1. Tenant isolation with immutable GUID expectations export AZURE_CONFIG_DIR="$HOME/.azure-tenants/<alias>" export AZD_CONFIG_DIR="$HOME/.azd-tenants/<alias>" EXPECTED_TENANT_ID="<tenant-guid>" EXPECTED_SUBSCRIPTION_ID="<subscription-guid>" assert_azure_target() { local actual_tenant actual_subscription azd_tenant azd_subscription actual_tenant="$(az account show --query tenantId -o tsv)" actual_subscription="$(az account show --query id -o tsv)" azd_tenant="$(azd env get-value AZURE_TENANT_ID --no-prompt)" azd_subscription="$(azd env get-value AZURE_SUBSCRIPTION_ID --no-prompt)" [[ "$actual_tenant" == "$EXPECTED_TENANT_ID" ]] || { echo "Azure CLI tenant mismatch" >&2; return 1; } [[ "$actual_subscription" == "$EXPECTED_SUBSCRIPTION_ID" ]] || { echo "Azure CLI subscription mismatch" >&2; return 1; } [[ "$azd_tenant" == "$EXPECTED_TENANT_ID" ]] || { echo "azd tenant mismatch" >&2; return 1; } [[ "$azd_subscription" == "$EXPECTED_SUBSCRIPTION_ID" ]] || { echo "azd subscription mismatch" >&2; return 1; } } az login --tenant "$EXPECTED_TENANT_ID" azd auth login --tenant-id "$EXPECTED_TENANT_ID" az account set --subscription "$EXPECTED_SUBSCRIPTION_ID" # 2. Materialize and verify the exact pin. Do not replace this with # branch-based azd init: citadel-v1 is mutable. PINNED_SHA="63f0f812474e713916dc909494d655246783a1d9" git clone --filter=blob:none --no-checkout \ https://github.com/Azure-Samples/ai-hub-gateway-solution-accelerator \ my-citadel-hub git -C my-citadel-hub fetch --depth 1 origin "$PINNED_SHA" (cd my-citadel-hub && git checkout --detach "$PINNED_SHA") test "$(git -C my-citadel-hub rev-parse HEAD)" = "$PINNED_SHA" || exit 1 cd my-citadel-hub azd env new citadel-pilot-01 azd env set AZURE_TENANT_ID "$EXPECTED_TENANT_ID" azd env set AZURE_SUBSCRIPTION_ID "$EXPECTED_SUBSCRIPTION_ID" # 3. Apply the pilot-quickstart profile (env-var bundle from the skill) while IFS='=' read -r k v; do [[ -z "$k" || "$k" == \#* ]] && continue azd env set "$k" "$v" done < "$SKILL_DIR/references/profiles/pilot-quickstart.env" # 4. Review the structured aiFoundryInstances and aiFoundryModelsConfig # arrays in bicep/infra/main.bicepparam. ENV profiles cannot safely # override arrays. Reduce them directly if quota or model scope requires. # 5. Deploy assert_azure_target azd up ``` Expected wall clock: **30-45 min** (APIM provisioning dominates). Expected baseline cost: ~$200-400/mo with light usage. Complete the common Entra step below after the deployment. ### Path B — Enterprise Baseline (production-grade, public APIM) Goal: Standard v2 APIM, all backend services on private endpoints, BYO Log Analytics for the central observability landing zone. ```bash # Repeat Path A steps 0-2 with env name citadel-prod-01. You must be inside # the verified detached checkout before continuing. # Set BYO Log Analytics first azd env set USE_EXISTING_LOG_ANALYTICS true azd env set EXISTING_LOG_ANALYTICS_NAME "log-central-prod" azd env set EXISTING_LOG_ANALYTICS_RG "rg-observability-prod" azd env set EXISTING_LOG_ANALYTICS_SUBSCRIPTION_ID "<central-sub-id>" # Apply the enterprise-baseline profile while IFS='=' read -r k v; do [[ -z "$k" || "$k" == \#* ]] && continue azd env set "$k" "$v" done < "$SKILL_DIR/references/profiles/enterprise-baseline.env" assert_azure_target azd up ``` To make APIM ingress private-only: set `APIM_V2_PUBLIC_NETWORK_ACCESS=false` after applying the profile. Event Hub public access must still remain `Enabled` for APIM v2 provisioning. ### Path C — VNet-Isolated, Spoke-Aware (peers to your landing zone) Goal: Deploy the hub into an existing hub-spoke topology, BYO VNet, BYO Private DNS Zones (typical landing zone with central DNS), pre-wired for spokes deployed via `foundry-vnet-deploy`. ```bash # Repeat Path A steps 0-2 with env name citadel-prod-01. # Pre-requisites: # - VNet vnet-citadel-hub already exists in rg-network-prod # with subnets snet-apim, snet-private-endpoint, snet-functionapp # - Private DNS zones already exist in rg-dns-prod (one zone per privatelink.* type) # Set BYO networking first azd env set USE_EXISTING_VNET true azd env set VNET_NAME "vnet-citadel-hub" azd env set EXISTING_VNET_RG "rg-network-prod" azd env set EXISTING_DNS_ZONE_OPENAI "/subscriptions/<dns-sub-id>/resourceGroups/rg-dns-prod/providers/Microsoft.Network/privateDnsZones/privatelink.openai.azure.com" # … repeat EXISTING_DNS_ZONE_* for the other 12 zones — see # `$SKILL_DIR/references/profiles/vnet-isolated-spoke-aware.env` for the # full list of EXISTING_DNS_ZONE_* env vars. # Apply the vnet-isolated-spoke-aware profile while IFS='=' read -r k v; do [[ -z "$k" || "$k" == \#* ]] && continue azd env set "$k" "$v" done < "$SKILL_DIR/references/profiles/vnet-isolated-spoke-aware.env" assert_azure_target azd up ``` Then deploy your spoke separately with `foundry-vnet-deploy`, peer the spoke VNet to the hub VNet, and link the `privatelink.azure-api.net` zone to the spoke VNet so spoke agents resolve the hub APIM private FQDN. ### Required Entra setup for all profiles All profiles enable Entra JWT policy, but all also keep Key Vault public network access disabled. Run the pinned `bicep/infra/entra-id-setup/setup.ps1` only after `azd up`, from an administrative host that uses the same isolated az/azd environment and has private DNS plus network reachability to the Key Vault private endpoint (for example, a workstation connected by the approved VPN or a peered management host). Do not weaken the profile automatically just to run the script. The signed-in operator needs: - Microsoft Graph `Application.ReadWrite.All` permission or the Entra **Application Developer** role to create/update the app registration, service principal, and policy-compliant client secret. - **Key Vault Secrets Officer** on the deployed vault data plane. - **API Management Service Contributor** on the deployed APIM service (or its resource group) to read/create/update JWT named values. > **Credential-policy workaround required.** The pinned `setup.ps1` calls