pilotswarm-azure-lessons

Stars8

Forks6

UpdatedJuly 3, 2026 at 23:11

Workarounds for common Azure issues: RBAC with Conditional Access, PostgreSQL region restrictions, Key Vault + CSI Driver setup.

Installation

Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.

Source

affandar

affandar/PilotSwarm

View GitHub Repository View Creator Repositories

Download

Related occupationsSOC

Based on SOC occupation classification

Network and Computer Systems AdministratorsComputer and Mathematical Occupations·SOC 15-1244

SKILL.md

readonly

name	pilotswarm-azure-lessons
description	Workarounds for common Azure issues: RBAC with Conditional Access, PostgreSQL region restrictions, Key Vault + CSI Driver setup.

PilotSwarm Azure Lessons Learned

Verified workarounds for Azure issues encountered during PilotSwarm deployments.

RBAC with Corporate Conditional Access

In enterprise tenants with conditional access policies, az role assignment create may fail with AADSTS530084 because the command calls the Graph API to resolve principal/role IDs, and conditional access token protection policies block that call.

Workaround: Use az rest to call the ARM RBAC API directly, bypassing Graph entirely:

ASSIGNMENT_ID=$(uuidgen | tr '[:upper:]' '[:lower:]')
az rest --method PUT \
  --url "https://management.azure.com<SCOPE>/providers/Microsoft.Authorization/roleAssignments/${ASSIGNMENT_ID}?api-version=2022-04-01" \
  --body "{
    \"properties\": {
      \"roleDefinitionId\": \"<SUBSCRIPTION_SCOPE>/providers/Microsoft.Authorization/roleDefinitions/<ROLE_GUID>\",
      \"principalId\": \"<PRINCIPAL_OBJECT_ID>\",
      \"principalType\": \"ServicePrincipal\"
    }
  }"

Common role definition GUIDs:

Key Vault Secrets User: 4633458b-17de-408a-b874-0445c86b69e6
Key Vault Secrets Officer: b86a8fe4-44ce-4948-aee5-eccb2c155cd7
Storage Blob Data Contributor: ba92f5b4-2d11-453d-a403-e96b0029c9fe

Note: az role assignment create may report an error even when the assignment actually succeeds. A subsequent attempt returning RoleAssignmentExists confirms it worked.

Azure Key Vault with Secrets Store CSI Driver

When using AKV + Secrets Store CSI in AKS:

Create the vault with --enable-rbac-authorization true
Assign Key Vault Secrets Officer to yourself (the operator) for storing secrets
Assign Key Vault Secrets User to the workload identity managed identity for reading secrets
Use a SecretProviderClass manifest with objectType: secret entries
Mount secrets as env vars via secretObjects in the SecretProviderClass
The CSI driver requires a volume mount even if you only use env vars

Docker Image Platform Mismatch

Building Docker images on macOS (Apple Silicon) produces ARM64 images by default. AKS nodes run linux/amd64. Always use --platform linux/amd64 when building:

docker build --platform linux/amd64 -f deploy/Dockerfile.worker -t <tag> .

Without this, pods will fail with no match for platform in manifest on image pull.

Local Package Parity in Docker Builds

When package.json uses file: links to a peer package (e.g., "pilotswarm-sdk": "file:../pilotswarm/packages/sdk"), the Docker build context cannot follow symlinks outside the build root.

If package.docker.json references "^0.1.6" (npm), the deployed image gets the published version — which may be behind the local source.

Fix: Copy the peer packages into the build context and use file: references in package.docker.json:

# Before docker build
cp -r ../pilotswarm/packages/sdk pilotswarm-sdk-local
cp -r ../pilotswarm/packages/app pilotswarm-app-local

"pilotswarm-sdk": "file:pilotswarm-sdk-local",
"pilotswarm": "file:pilotswarm-app-local"

Add the local copies to .gitignore.

More from this repository

same repository

agent-repair-loop

affandar/PilotSwarm

The order of operations for changing a live agent safely — diagnose, patch, publish, converge, verify, roll back. Use whenever you are about to modify an agent package that something is already running on.

2026-08-038

pilotswarm-aks-deploy

affandar/PilotSwarm

Deploy PilotSwarm to AKS using the repo's canonical scripts and checks. Use when refreshing secrets, building/pushing the worker image, rolling out to AKS, or verifying provider/model changes in the live cluster.

2026-08-038

pilotswarm-aks-reset

affandar/PilotSwarm

Reset remote PilotSwarm state for AKS safely. Use when wiping the PilotSwarm database/blob state, clearing stale orchestration history, or recovering from namespace drift and replay/nondeterminism issues.

2026-08-038

pilotswarm-new-env-deploy

affandar/PilotSwarm

Use when bringing up a fresh, isolated PilotSwarm environment (`mysandbox`, `myenv2`, etc.) via the npm Bicep/GitOps orchestrator at `deploy/scripts/deploy.mjs`. Covers `new-env` scaffolding, EDGE_MODE × TLS_SOURCE selection, the `all` aggregate, per-service redeploys with `--steps`, force-redeploy semantics, verification, and teardown. Strictly separate from the legacy bash path operated by `scripts/deploy-aks.sh`.

2026-08-038

pilotswarm-vpn-client-profile

affandar/PilotSwarm

Use after deploying a PilotSwarm stamp with VPN_GATEWAY_ENABLED=true when an operator needs the Azure VPN client profile (azurevpnconfig.xml). Wraps deploy/scripts/auth/Get-VpnClientProfile.ps1 — downloads the gateway-issued profile zip via 'az network vnet-gateway vpn-client generate' and extracts it under the gitignored deploy/envs/local/<env>/vpn-client/ folder. The XML is the same for every user (no per-user credentials), and end users still authenticate with their own Entra ID at connect time.

2026-08-038

cost-latency-analysis

affandar/PilotSwarm

How to compute model latency and estimated $ cost from PilotSwarm observability data. Read this before reporting that a model is "slow" or "expensive" — most apparent slowness is orchestration overhead, not model inference, and most cost numbers are guesses unless they reference a real published price card.

2026-08-038

name	pilotswarm-azure-lessons
description	Workarounds for common Azure issues: RBAC with Conditional Access, PostgreSQL region restrictions, Key Vault + CSI Driver setup.

PilotSwarm Azure Lessons Learned

Verified workarounds for Azure issues encountered during PilotSwarm deployments.

RBAC with Corporate Conditional Access

Workaround: Use az rest to call the ARM RBAC API directly, bypassing Graph entirely:

ASSIGNMENT_ID=$(uuidgen | tr '[:upper:]' '[:lower:]')
az rest --method PUT \
  --url "https://management.azure.com<SCOPE>/providers/Microsoft.Authorization/roleAssignments/${ASSIGNMENT_ID}?api-version=2022-04-01" \
  --body "{
    \"properties\": {
      \"roleDefinitionId\": \"<SUBSCRIPTION_SCOPE>/providers/Microsoft.Authorization/roleDefinitions/<ROLE_GUID>\",
      \"principalId\": \"<PRINCIPAL_OBJECT_ID>\",
      \"principalType\": \"ServicePrincipal\"
    }
  }"

Common role definition GUIDs:

Key Vault Secrets User: 4633458b-17de-408a-b874-0445c86b69e6
Key Vault Secrets Officer: b86a8fe4-44ce-4948-aee5-eccb2c155cd7
Storage Blob Data Contributor: ba92f5b4-2d11-453d-a403-e96b0029c9fe

Note: az role assignment create may report an error even when the assignment actually succeeds. A subsequent attempt returning RoleAssignmentExists confirms it worked.

Azure Key Vault with Secrets Store CSI Driver

When using AKV + Secrets Store CSI in AKS:

Create the vault with --enable-rbac-authorization true
Assign Key Vault Secrets Officer to yourself (the operator) for storing secrets
Assign Key Vault Secrets User to the workload identity managed identity for reading secrets
Use a SecretProviderClass manifest with objectType: secret entries
Mount secrets as env vars via secretObjects in the SecretProviderClass
The CSI driver requires a volume mount even if you only use env vars

Docker Image Platform Mismatch

Building Docker images on macOS (Apple Silicon) produces ARM64 images by default. AKS nodes run linux/amd64. Always use --platform linux/amd64 when building:

docker build --platform linux/amd64 -f deploy/Dockerfile.worker -t <tag> .

Without this, pods will fail with no match for platform in manifest on image pull.

Local Package Parity in Docker Builds

When package.json uses file: links to a peer package (e.g., "pilotswarm-sdk": "file:../pilotswarm/packages/sdk"), the Docker build context cannot follow symlinks outside the build root.

If package.docker.json references "^0.1.6" (npm), the deployed image gets the published version — which may be behind the local source.

Fix: Copy the peer packages into the build context and use file: references in package.docker.json:

# Before docker build
cp -r ../pilotswarm/packages/sdk pilotswarm-sdk-local
cp -r ../pilotswarm/packages/app pilotswarm-app-local

"pilotswarm-sdk": "file:pilotswarm-sdk-local",
"pilotswarm": "file:pilotswarm-app-local"

Add the local copies to .gitignore.