agentops-workflow

# AgentOps Workflow Help the user wire AgentOps into the release path so every candidate has a clear gate and proof pack. The default starting point is a PR eval gate. Full DEV/QA/PROD workflows are useful only after Azure auth, environments, and a real deployment owner are configured. **Pick the platform up front.** AgentOps supports two: - `--platform github` (default) - writes `.github/workflows/*.yml` using GitHub Actions. Auth via OIDC + GitHub Environments. - `--platform azure-devops` - writes `.azuredevops/pipelines/*.yml` using Azure DevOps Pipelines. Auth via a Service Connection + a variable group named `agentops`. The conceptual workflows are identical: one PR gate and optional deploy stages (dev/qa/prod). The PR and production templates already run `agentops doctor --evidence-pack` so reviewers get `evidence.json` and `evidence.md` in artifacts and, for GitHub Actions, in the run summary with the Doctor finding summary. A separate scheduled Doctor workflow is optional for periodic health checks, not the default release path. For a new repository or tutorial, start with the PR gate only: `agentops workflow generate --kinds pr`. Generate DEV/QA/PROD deploy workflows only after environments, Azure auth, and real build/deploy commands are configured. For copied accelerators or unfamiliar repos (for example GPT-RAG, Live Voice Practice, AI Landing Zone/Bicep-based apps), run `agentops workflow analyze` first and use the findings as the implementation plan before generating or editing workflows. AgentOps reuses **azd** for app/infrastructure deployment when the repo already has an azd project, and stays **Foundry-native** for prompt-agent candidate workflows. Do not invent a parallel deployment system. AgentOps should gate quality and record proof; `azd provision`, `azd deploy`, azd hooks, Foundry Toolkit, the `microsoft-foundry` skill, and project tooling manage lifecycle actions. For Foundry prompt-agent configs (`agent: name:version`), the generated eval gate should use **AgentOps cloud eval in Foundry**: a temporary cloud config plus `agentops eval run`, not the legacy official Action/task. Foundry still executes the managed eval; AgentOps enforces thresholds, writes `results.json` / `report.md`, and makes PR failures explainable in the summary. ## Fast path - generated GitHub setup Use this path when the user already generated GitHub workflows or asks to get the PR gate running. Stay local-first and deterministic; do not start by discovering the whole Azure subscription. 1. Inspect the repo before cloud discovery: - `agentops init show --dir .` without `--reveal-secrets`. - `agentops.yaml`. - `.agentops/.env`, plus `.azure/config.json` and active `.azure/<env>/.env` when the repo uses azd. - `azd env get-values` when `azure.yaml` exists and azd is available. - `.github/workflows/agentops-*.yml`. 2. Read the generated workflows to determine exactly which GitHub environments and variables are needed. For prompt-agent PR gates, `pr` uses `environment: sandbox`; deploy workflows use `dev`, `qa`, or `production`. 3. Treat these as GitHub Actions environments for OIDC and variables. `sandbox` points at the Foundry authoring project. `dev` points at the first shared post-merge project. 4. Proceed only when these values are known or deliberately chosen: - GitHub `owner/repo`. - workflow environment names from `jobs.*.environment`. - `AZURE_CLIENT_ID`, `AZURE_TENANT_ID`, `AZURE_SUBSCRIPTION_ID`. - `AZURE_AI_FOUNDRY_PROJECT_ENDPOINT`. - `AZURE_OPENAI_DEPLOYMENT`. - optional `APPLICATIONINSIGHTS_CONNECTION_STRING`. 5. Prefer existing values and exact checks: - `git remote get-url origin` and `gh repo view --json nameWithOwner`. - `git branch -vv` to confirm the local trunk branch tracks `origin/main` when the tutorial uses trunk-based `main`. - `gh variable list --env <env>` and `gh secret list --env <env>`. - `agentops init show`, local `.agentops/.env` or `.azure/<env>/.env`, and `azd env get-values` values before `az account show`. - `az account show` only as a proposal for tenant/subscription; confirm before writing it to GitHub variables. 6. For GitHub OIDC, treat `AZURE_TENANT_ID` as the tenant that owns the app registration / federated credential, not merely the tenant associated with the subscription or a `managedByTenants` entry. Before writing `AZURE_TENANT_ID`, verify the chosen tenant can see the app registration and the exact federated credential: - `az ad app show --id <AZURE_CLIENT_ID>` in the active tenant, or an equivalent Microsoft Graph query scoped to the proposed tenant. - `az ad app federated-credential list --id <AZURE_CLIENT_ID>` and confirm the `subject`, `issuer`, and `audiences`. If the app is visible in one tenant but the Azure subscription is associated with another tenant, use the app/federated-credential tenant for `AZURE_TENANT_ID`; the subscription id remains `AZURE_SUBSCRIPTION_ID`. Do not copy a `managedByTenants[*].tenantId` value into GitHub variables unless the app and federated credential are verified there too. 7. When creating or connecting the GitHub remote for the prompt-agent tutorial, make sure the local trunk branch tracks the remote trunk before telling the user to continue: - If `main` is newly pushed, use `git push -u origin main`. - If `origin/main` already exists, use `git branch --set-upstream-to=origin/main main`. - Verify with `git branch -vv`; `main` must show `[origin/main]`. Without this, a later `git pull` on `main` can fetch but not update the local branch. 8. Copy CI variables from local AgentOps/azd configuration into the GitHub environment used by the workflow. Reuse local values for `AZURE_AI_FOUNDRY_PROJECT_ENDPOINT`, `AZURE_OPENAI_ENDPOINT`, `AZURE_OPENAI_DEPLOYMENT`, and optional `APPLICATIONINSIGHTS_CONNECTION_STRING` instead of asking the user to type them again. Explain `AZURE_OPENAI_DEPLOYMENT` only if it is missing: it is the Azure OpenAI deployment used as the evaluator/judge model, not the user's agent. 9. For prompt-agent tutorials that use Foundry trace sampling / trace-to-dataset, verify observability RBAC before telling the user step 18 is ready: - Resolve the dev Foundry project managed identity principal id. - Resolve the connected Application Insights resource. - Grant or verify **Reader** on that Application Insights resource to the dev Foundry project managed identity. - If the App Insights component is workspace-based, also grant or verify **Reader** on the backing Log Analytics workspace. This is separate from GitHub OIDC and separate from the signed-in user's portal access. Operate dashboards can still render while trace-to-dataset fails if the project identity cannot read App Insights. 10. Do not enumerate subscriptions, Foundry projects, Azure OpenAI resources, or model deployments to guess missing values. If `AZURE_SUBSCRIPTION_ID`, `AZURE_TENANT_ID`, `AZURE_AI_FOUNDRY_PROJECT_ENDPOINT`, or `AZURE_OPENAI_DEPLOYMENT` is absent from AgentOps/azd/local env, ask the user to choose or provide it. Only run a scoped Azure query after the user confirms the subscription and the exact missing value. 11. For GitHub OIDC, derive the federated credential subject from **both** the generated workflow and the repository's actual subject claim prefix. If the job has `environment: dev`, the default subject is `repo:<owner>/<repo>:environment:dev`, but accounts with an immutable-ID subject claim prefix send numeric account and repository IDs instead: `repo:<owner>@<accountId>/<repo>@<repoId>:environment:dev`. Entra compares `subject` literally, with no wildcards and no normalization, so the wrong format fails with `AADSTS700213`. Always read the prefix first: - `gh api repos/<owner>/<repo>/actions/oidc/customization/sub` Use the `sub_claim_prefix` field, not the booleans beside it. `use_default` can be `true` and `use_immutable_subject` can be `false` while the prefix still carries IDs, and the prefix is what lands in the token. The account ID is stable per account, but the repository ID changes per repository, so re-read it for every repo. Create **two** federated credentials per environment on the same app registration, one per format. They cost nothing, work on both kinds of account, and survive a later change to the account's subject configuration: - `<prefix>:environment:<env>` where `<prefix>` is the value read above. - `repo:<owner>/<repo>:environment:<env>` (the default format). Skip the second credential only when `sub_claim_prefix` already equals `repo:<owner>/<repo>`. Do not assume branch or `pull_request` subjects without reading the workflow. 12. Before triggering a Foundry prompt-agent workflow, make sure the OIDC app / service principal has **two** RBAC assignments. Both are required; the eval step fails silently (every metric returns `null`) if only one is in place. 1. **Foundry User** on the Foundry project (or the Foundry resource scope if that is the team's standard). Role id `53ca6127-db72-4b80-b1b0-d745d6d5456d` (formerly Azure AI User). Without this the candidate-staging step fails on `Microsoft.CognitiveServices/accounts/AIServices/agents/read`. 2. **Cognitive Services OpenAI User** on the underlying Azure AI Services account that hosts the evaluator model deployment (typically the parent account of the Foundry project). Role id `5e0bd9bd-7b93-4f28-af87-19fc36ad61bd`. Without this the Foundry `azure_ai_evaluator` graders fail with a 401 `PermissionDenied` on `Microsoft.CognitiveServices/accounts/OpenAI/deployments/chat/completions/action` and every metric comes back `null` in the cloud eval report. AgentOps now lifts that error into `results.json` and the orchestrator's "0 usable metric scores" warning so the cause is visible in CI logs, but the workflow still fails the gate. Grant this role **before** the first run. Azure **Reader** is not enough for either step. 13. If either RBAC assignment is missing, do not run the workflow yet. Show the exact GitHub OIDC client ID / service principal, desired role, target scope (project for Foundry User, AI Services account for Cognitive Services OpenAI User), then ask the user to approve the role assignment or get an Azure/Foundry admin to grant it. After assignment, read it back or ask the user to confirm before dispatching the workflow. When the user approves and you know the scopes, use the role ids to avoid rename drift: - `az ad sp show --id <AZURE_CLIENT_ID> --query id -o tsv` - `az role assignment list --assignee <sp-object-id> --scope <foundry-scope> --include-inherited` - `az role assignment create --assignee-object-id <sp-object-id> --assignee-principal-type ServicePrincipal --role 53ca6127-db72-4b80-b1b0-d745d6d5456d --scope <foundry-scope>` - `az role assignment create --assignee-object-id <sp-object-id> --assignee-principal-type ServicePrincipal --role 5e0bd9bd-7b93-4f28-af87-19fc36ad61bd --scope <ai-services-account-scope>` The AI Services account scope looks like `/subscriptions/<sub>/resourceGroups/<rg>/providers/Microsoft.CognitiveServices/accounts/<ai-account-name>` and can be derived from `az cognitiveservices account list --resource-group <foundry-project-rg> --query "[?kind=='AIServices'].id" -o tsv`. 14. Ask before creating or updating GitHub repos, GitHub environments, variables/secrets, Entra app registrations/service principals, federated credentials, managed identities, or Azure RBAC assignments. 15. When creating federated credentials from PowerShell, avoid fragile interpolation. Do **not** write `"repo:$repo:environment:$envName"` because `$repo:` can be parsed as a scoped variable. Use `"repo:${repo}:environment:${envName}"` or `("repo:{0}:environment:{1}" -f $repo, $envName)`, then build JSON from a PowerShell object with `ConvertTo-Json`. 16. After creating or updating a federated credential, read it back and verify before triggering a workflow: - `az ad app federated-credential list --id <AZURE_CLIENT_ID> --query "[].{name:name, subject:subject, issuer:issuer, audiences:audiences}" -o table` - `subject` exactly matches a subject the workflow can present. Both the default and the `sub_claim_prefix` form must be present unless the prefix already equals `repo:<owner>/<repo>`. - `issuer` is `https://token.actions.githubusercontent.com`. - `audiences` includes `api://AzureADTokenExchange`. If any value differs, fix the credential before running GitHub Actions. 17. After setting GitHub environment variables, read them back and verify `AZURE_TENANT_ID` still matches the app/federated-credential tenant before triggering a run. Map `azure/login` failures to the right cause instead of guessing: - `AADSTS700213: No matching federated identity record found for presented assertion subject` means the subject is not byte-identical to what GitHub sent. Copy the subject quoted in the error, diff it against the credential list, and add the missing credential. On an immutable-ID account this is the missing `sub_claim_prefix` credential from rule 11. - `AADSTS53003` is usually tenant/app misalignment, not Conditional Access. Re-check that `AZURE_TENANT_ID` is the tenant owning the app registration. 18. Do not dispatch `gh workflow run` as a surprise validation step. First show that the GitHub environment, variables/secrets, federated credential, and Foundry RBAC are ready, then ask the user before triggering workflows. 19. Avoid broad discovery unless local config is missing. Do **not** run broad `az resource list`, `az graph query`, SDK inspection, or web search to find the Foundry project when `agentops init show`, `.agentops/.env`, or `.azure/<env>/.env` already has `AZURE_AI_FOUNDRY_PROJECT_ENDPOINT`. If the endpoint is missing, say exactly what is missing and ask the user before scanning the subscription. ## Branch model assumed ``` feature/* ── PR ──▶ develop [agentops-pr] gate │ └── merge ─▶ develop [agentops-deploy-dev] build + eval + deploy DEV release/* ── push [agentops-deploy-qa] build + eval + deploy QA release/* ── PR ──▶ main [agentops-pr] gate │ └── merge ─▶ main [agentops-deploy-prod] safety eval + build + deploy PROD ``` If the user is on trunk-based development, omit `qa` and `release/**` and have them generate `--kinds pr,dev,prod`. The PR workflow uses the eval step as the hard merge gate. Doctor also runs there and writes release evidence; by default the Doctor step blocks the PR on critical findings such as regression detection (`--severity-fail critical`, the default behavior of `agentops workflow generate --doctor-gate critical`). This catches metric drops (for example groundedness going from 5.0 to 4.0) that would still pass the configured eval thresholds. To restore the pre-1.x advisory behavior — Doctor writes release evidence but does not block the PR — generate with `--doctor-gate none`. DEV/QA/PROD deploy workflows always keep Doctor as a critical release gate; the `--doctor-gate` flag only controls the PR template. ## Step 0 - Prerequisites 1. `pip install "agentops-accelerator @ git+https://github.com/Azure/agentops.git@main"` if `agentops` is missing. 2. `agentops eval analyze` has been reviewed, `agentops.yaml` exists at the project root, and `agentops eval run` works locally. 3. The user's repo follows GitFlow (or is willing to). If not, ask which branches map to dev/qa/prod and adjust the triggers after generation. ## Step 1 - Generate the workflows First analyze the repo shape: ```bash agentops workflow analyze agentops workflow analyze --format markdown --out agentops-workflow-plan.md ``` Use the analysis to decide whether `--deploy-mode auto` is enough or whether you need to adapt placeholders/project-specific deployment. The analyzer is local-only and looks for `azure.yaml`, Bicep, AgentOps prompt-agent config, landing-zone manifests, private-network signals, Docker/Container Apps signals, and existing CI folders. Treat README matches as hints only; structural files drive the recommendation. **GitHub Actions (default):** ```bash agentops workflow generate --kinds pr # or full scaffold: agentops workflow generate --kinds pr,dev,qa,prod --force ``` **Azure DevOps Pipelines:** ```bash agentops workflow generate --platform azure-devops --kinds pr # or full scaffold: agentops workflow generate --platform azure-devops --kinds pr,dev,qa,prod --force ``` The full scaffold writes: | Kind | GitHub Actions path | Azure DevOps path | Trigger | Environment | |---|---|---|---|---| | `pr` | `.github/workflows/agentops-pr.yml` | `.azuredevops/pipelines/agentops-pr.yml` | PRs to `develop`, `release/**`, `main` | `sandbox` for prompt-agent PR candidates, `dev` for generic PR gates | | `dev` | `.github/workflows/agentops-deploy-dev.yml` | `.azuredevops/pipelines/agentops-deploy-dev.yml` | push to `develop` | `dev` | | `qa` | `.github/workflows/agentops-deploy-qa.yml` | `.azuredevops/pipelines/agentops-deploy-qa.yml` | push to `release/**` | `qa` | | `prod` | `.github/workflows/agentops-deploy-prod.yml` | `.azuredevops/pipelines/agentops-deploy-prod.yml` | push to `main` | `production` | | `doctor` | `.github/workflows/agentops-doctor.yml` | `.azuredevops/pipelines/agentops-doctor.yml` | daily cron (06:00 UTC) | `dev` | PR and PROD workflows upload release evidence. Explain that this is a projection of existing eval/Doctor/Foundry/monitoring signals, not a separate exit-code contract. Generate the optional scheduled Doctor workflow only when the team explicitly wants periodic health-check artifacts outside PR/release events. Useful flags: - `--platform github | azure-devops` - pick the CI/CD platform. - `--force` - overwrite existing workflow files. - `--kinds pr,dev,qa,prod` - generate a subset. Prefer `--kinds pr` until deploy environments are configured. - `--kinds doctor` - optional scheduled Doctor-only workflow for periodic checks. Do not use it as a substitute for the PR gate. - `--deploy-mode auto|placeholder|azd|prompt-agent` - `auto` uses azd templates when `azure.yaml` exists, otherwise uses prompt-agent templates when `agentops.yaml` targets a Foundry prompt agent; `azd` forces `azd provision` / `azd deploy`; `prompt-agent` stages/evaluates a Foundry prompt candidate; `placeholder` keeps the generic stack-agnostic scaffold.