foundry-agentops

# Foundry AgentOps adoption Use native Azure AgentOps to collect reviewable operational evidence for **one already deployed agent**. This is an instruction/runbook skill, not a wrapper executable, deployment engine, or replacement evidence schema. ## Modes and ownership — decide first - **Standalone:** analyze existing workflow shape first; generation requires separate user approval of platform, files, triggers, and gates. Never generate workflows merely because Doctor reports them missing. - **Threadlight:** detect `specs/manifest.json` or existing Threadlight conventions in the selected project. Provide AgentOps configuration and prerequisites only: **NO `workflow generate`**, competing pipeline files, adapter, or new manifest. `threadlight-cicd` owns pipelines; final readiness scoring stays with Threadlight. If ownership is unclear, stop before writing. - **Citadel:** leave gateway/access contracts, isolation, routing, and rate limits unchanged. Use the existing approved endpoint; never bypass its access path. Do not mutate APIM, networking, RBAC, branch protection, or environment approvals. - **Canonical specialists:** `foundry-evals` owns deep evaluators/datasets; `foundry-observability` owns instrumentation; `foundry-agt` owns AGT authoring and runtime governance. Missing prerequisites are handoffs, not permission to provision resources here. This skill issues **no production-ready certification**. ## 1. Select exactly one agent root Before bootstrap or native analysis, inspect only candidate markers, in order: 1. The user's **explicit agent root** wins; validate it, do not silently substitute. 2. Otherwise, a **unique `azure.yaml` service with `host: azure.ai.agent`**: resolve its `project` path relative to that manifest (default `.`). 3. Otherwise, a **unique root containing `.foundry/agent-metadata*.yaml`**. 4. Otherwise, an existing **`agentops.yaml` root**, only if unique. Multiple candidates at any tier require the user's choice; do not fall through to a lower tier to hide ambiguity. Multiple agents inside a chosen root also require an explicit target. With no candidate, request a root; do not initialize the repository root by default. `agentops.yaml` is the **only opt-in marker**: deployment metadata and `.agentops/` alone are discovery hints, not consent. Record the selected absolute root, target identity/version, and deployment environment. **After selection, never scan or modify sibling roots.** Keep all config, dataset, baseline, and output paths inside this root; reject symlinks or relative paths escaping it. If a native recursive scan would include another agent root, stop and obtain a narrower one-agent workspace instead. ### PRE-EXECUTION credential isolation **Before any CLI/native call, including context discovery**, the owner must provide a process-local `AZURE_CONFIG_DIR`. Set it in the parent before spawning Copilot, SDK helpers or native commands; inherit the same value in **every Bash** call. For **all three routes below**, also provide a private **fresh empty** `AZD_CONFIG_DIR`, including approved cached-CLI mode: **no login or token cache** may be copied into or created in that directory. Keep it credential-empty for the entire run; do not mutate/repoint it or deliberately run azd commands (including login/token acquisition) against it. Never inspect/copy the global `~/.azure`/`~/.azd` caches, switch accounts, log in, or change identity/RBAC here. Follow [azure-tenant-isolation's two-layer guard](../azure-tenant-isolation/SKILL.md#mandatory-rules); isolation and an approved subscription scope are separate requirements. At v0.14.0, [get_shared_credential](https://github.com/Azure/agentops/blob/v0.14.0/src/agentops/utils/azure_credentials.py) first probes `az account show` and returns **AzureCliCredential before DefaultAzureCredential** whenever that cache is available. Doctor sources use this factory, while prompt creation, eval invocation and preflight can construct DefaultAzureCredential directly. Approved `AZURE_*` environment credentials do **not** override the factory's CLI-first branch. Even `AZURE_TOKEN_CREDENTIALS` only controls DefaultAzureCredential, not that earlier branch. There is a second boundary: the factory defaults to **`exclude_developer_cli_credential=False`** and passes that keyword explicitly. Azure Identity 1.25.3 applies explicit exclusion keywords **after** the selector, so a native default chain retains **AzureDeveloperCliCredential alongside the selected source**. Doctor's [Azure resources](https://github.com/Azure/agentops/blob/v0.14.0/src/agentops/agent/sources/azure_resources.py#L431) and [RBAC helpers](https://github.com/Azure/agentops/blob/v0.14.0/src/agentops/agent/checks/_rbac_authorization.py#L38) use that default (also RBAC line 91). A configured source is **not a claim that only one credential object is constructed**. The selector plus cache isolation constrain **usable credentials**, not identical chain membership. Choose exactly one owner-approved route, shared by creation/eval/Doctor/cleanup: | Route | Isolated Azure CLI state | `AZURE_TOKEN_CREDENTIALS` | |---|---|---| | Approved cached CLI user, or CI service-principal CLI login | Owner-provisioned directory containing **only** the approved cached identity; active tenant/subscription already approved | `AzureCliCredential` | | Environment service principal | Fresh empty directory; never populate it during this run | `EnvironmentCredential` | | Federated workload identity | Fresh empty directory; never populate it during this run | `WorkloadIdentityCredential` | **A cached CLI user has no service-principal client ID requirement.** Leave `AZURE_CLIENT_ID` unset for that route; never substitute the CLI public application ID or put `N/A` in a credential variable. The user identity and approved tenant/subscription belong in the restricted approval record. CI's cached service-principal route still requires its real `AZURE_CLIENT_ID` alongside the runner's tenant/subscription exports. Environment and workload routes require the real principal's client ID and complete credentials below. Do not confuse a selector (`AzureCliCredential`) with the type of principal it authenticates. The fixture's `cached-user` / `ci-cli` arguments distinguish these two cases without inventing native settings. Use **`azure-identity~=1.25.3`** in the native and helper environments and verify the resolved version. Individual credential-name selection is supported since 1.24.0; AgentOps's broader dependency range alone does not guarantee it. This is a supported Azure Identity selector, **not** an AgentOps opt-out. Reject an unset selector, `prod`, `dev`, or an unapproved source: an empty CLI directory alone still allows managed identity, PowerShell, shared token cache, VS Code/broker or azd fallback through an unrestricted DefaultAzureCredential. Do not patch/wrap the native factory, use test-only probe controls, or invent exclusion environment variables. The retained developer-CLI leg may make a noninteractive azd attempt if reached; this is not permission to log in or use azd as another credential route. Azure Identity's subprocess inherits the same `AZD_CONFIG_DIR`, never the global default when that variable is preserved. With no login/token cache there, this leg must fail unavailable rather than supply an identity. Missing, changed or populated isolation means **STOP**. If an operation requires intentional azd use or authentication, hand it to the deployment/workflow owner in a separate approved execution; do not populate this directory to make it pass. Reading existing workspace `.azure/<env>` deployment facts is distinct from azd login. The owner supplies complete credentials for the chosen source, not just IDs: environment authentication needs the approved tenant/client and secret or certificate settings; federation needs the approved tenant/client, authority and readable `AZURE_FEDERATED_TOKEN_FILE`. Validate the selected source and already-granted scope/permissions in the restricted approval record. Constructor selection is **not authorization**, token validity or endpoint reachability. If the approved route is unavailable, **STOP**; the inert developer-CLI attempt is not an authorized alternative. No fallback to another populated cache/source, login, account switch or permission repair. Keep the config paths and selector unchanged for the process lifetime: native probe/credential results are cached. Restart with owner-approved context rather than changing it mid-run. For manual non-CI cached runs, read-only tenant and `allowed_subscriptions` membership checks may run **only inside the approved isolated cache**; mismatch means STOP, never select an arbitrary account. For empty-cache routes, do not expect CLI sign-in: approval establishes the permitted principal/scope. For CI, the **runner-provisioned context is authoritative**: preserve show-don't-assert guidance, do not add subscription equality/cache-availability assertions or rediscover credentials. The workflow owner must establish the isolated context before authentication and inherit it into primary/retry/helper processes. Follow the [manual reproduction sequence](references/day2-runbook.md#manual-equivalent-and-outer-executor-diagnosis) for a separately authorized user-cache run. It does not prove the unchanged CI service-principal/workload-identity route was executed. Verify actual native and direct SDK **constructors only (no token calls)** offline with a simulated CLI-probe result. Inspect the nested `credentials` list, not just the outer type, for **both** exclusion values and **all three** selectors: | Simulated CLI state | Native factory `True` | Native factory `False` / omitted | |---|---|---| | Unavailable (each selector) | Default chain: selected credential | Default chain: selected credential, AzureDeveloperCliCredential | | Available (approved CLI route) | AzureCliCredential | AzureCliCredential | For direct DefaultAzureCredential, explicit `True` or no exclusion keyword produces the selected credential alone; explicit `False` adds the developer-CLI leg. Verify subprocess environment inheritance offline with an inert process stand-in, including an unavailable-azd response; never run azd or acquire tokens in these tests. These are construction/boundary checks, **not authorization** or proof of real empty-cache authentication failure. A live principal/authorization check remains the operator's responsibility. ### Resolve deployment context without drift Resolve azd context first: `AZURE_ENV_NAME`, then `.azure/config.json`'s `defaultEnvironment`, then the unique `.azure/<env>/.env`. Missing/ambiguous selected environments require resolution, not fallback to another deployment. Use only the selected service's outputs. Overlay its `.foundry/agent-metadata*.yaml` deployment facts where needed; this is a **manual context reconciliation**, not an invented AgentOps config loader. Conflicting tenant, subscription, project endpoint, target, or version means STOP. Reconcile existing `agentops.yaml` and process environment against those facts. Native dotenv loading prefers active azd, then `.agentops/.env`, then root `.env`; it does **not** overwrite process variables. `AGENTOPS_AGENT`/`--agent` can override the configured target. Do not allow stale overrides or a silent switch between environments. Map the confirmed project endpoint explicitly to `project_endpoint` and, for Doctor, `AZURE_AI_FOUNDRY_PROJECT_ENDPOINT`; do not assume other tools' `FOUNDRY_PROJECT_ENDPOINT` alias is consumed automatically. Use already-authorized credentials; tenant/network/RBAC setup belongs elsewhere. ### PRE-EXECUTION telemetry approval Before eval or Doctor, follow the [telemetry approval gate](references/day2-runbook.md#pre-execution-telemetry-approval): verify the actual tenant-local exporter destination (including Foundry auto-discovery/fallbacks) and obtain explicit owner approval for payload capture, access and retention in the effective environment. Endpoint inference/read access is not export authorization. Unknown or unauthorized means **STOP** and handoff to `foundry-observability`; do not change approved configuration or architecture. `GITHUB_STEP_SUMMARY` suppression/private logs are **not a network-export opt-out**. `AZURE_EXPERIMENTAL_ENABLE_GENAI_TRACING=false` does not suppress native full input/expected span attributes or Doctor finding text. No verified blanket native opt-out is claimed at this pin. Resolve the **actual linked App Insights component and LAW**, including effective table retention, before execution. Native Doctor's minimum lookback is **1 day**; the four component-scoped aggregates have no per-agent/run filter. Historical 24h totals are **not fresh per-agent ingestion**. Required sources stay enabled; if the actual shared scope is unacceptable, STOP rather than inventing narrower queries or patching native collection. Retention is a separate gate: a **7-day local raw retention** cap does not alter service retention (an observed **90-day** table value is illustrative only). Native Responses 0.14.0 omits `store=false` and does not persist the returned **response ID**. Agent deletion does **not** prove stored responses deleted. Use only synthetic, retention-approved smoke data; STOP before inference if zero storage or verified immediate deletion is required. No conversation searches, fake cleanup or interception/monkeypatch to recover IDs. The [runbook](references/day2-runbook.md#effective-scope-retention-and-responses-storage) records the native limits and owner handoffs. ## 2. Pin, opt in, and configure the native workspace Support is **Python >=3.11**, **`agentops-accelerator==0.14.0`**, tag `v0.14.0`, SHA `fb5c93eee489c71ef4084fa209adae24f762e3d7`. See the [upstream pin](references/upstream-pin.md). This is **not the `agentops` SaaS distribution**; the executable/import namespace happens to be `agentops`. Use a dedicated virtual environment in the selected workspace and install `python -m pip install "agentops-accelerator==0.14.0"` with that environment's Python. Check `importlib.metadata.version("agentops-accelerator") == "0.14.0"`. An unknown/different installed version means **stop and re-pin/revalidate**, not use latest help as a compatibility promise. All command examples below use `agentops` as shorthand for that verified **absolute executable path**. Bash calls are stateless: replace the shorthand each time and `cd` to the selected absolute root **in the same call**. Persist paths/context without secrets; do not rely on activation or prior shell variables. The credential-isolation gate above applies to every call, even local analysis; reject missing or changed config paths/selector rather than inheriting defaults. After explicit opt-in, `agentops init --no-prompt` scaffolds root `agentops.yaml`, seed data/traces and support files under `.agentops/`. Review proposed paths first. Never overwrite existing config or datasets without approval; do not use `--force`, a reconfiguration wizard, or init flags to bypass that rule. Existing workspaces usually need only a reviewed config edit, not reinitialization. `.agentops/` holds cache/generated evidence, not competing deployment metadata. Preserve curated ignore rules. Ensure secrets (`.agentops/.env`, root `.env`, `.azure/`) and generated `results/`, `official-eval/`, `.resolved/`, Doctor `agent/`, and `release/` output are ignored. Review raw `traces/` separately. Do not blanket-ignore `.agentops/`: approved sanitized `data/`, optional `.agentops/agent.yaml`, and baseline policy may need version control. Native init's ignore file does not cover every evidence path; inspect the resulting diff. The tagged flat schema derives target kind from `agent`; do not add `target:` or nested `agent: {kind: ...}` structures: | Existing target | Native `agent` value / settings | |---|---| | Foundry prompt | `"<agent-name>:<version>"`; confirmed `project_endpoint` | | Foundry hosted | Actual deployed Foundry URL; `protocol: responses` or `invocations` matching its contract | | Generic HTTP/JSON | Approved HTTPS URL; `protocol: http-json`; mappings matching the actual request/response contract | | Direct deployment | `"model:<deployment>"`; confirmed `project_endpoint`, not a public model catalog ID | For HTTP, use schema fields `request_field`, `response_field`, `tool_calls_field`, `response_fields`, and `headers` only as the real API requires. Secrets belong in an environment variable named by `auth_header_env`, never literal headers in YAML. There is no top-level `extra_fields` in this tag's config model. Minimal structural example, **not a sufficient release gate**: ```yaml version: 1 project_endpoint: "https://<account>.services.ai.azure.com/api/projects/<project>" agent: "<agent-name>:<version>" dataset: .agentops/data/smoke.jsonl execution: local ``` Provide representative, approved JSONL rows with `input`, normally `expected`, and required context/tool fields. Agree explicit `evaluators` and metric `thresholds` before running (for example, a lexical check can use `evaluators: [F1ScoreEvaluator]` and `thresholds: {f1_score: ">=0.8"}`; neither that evaluator nor that value is a universal quality policy). Confirm judge deployment/credential prerequisites for AI-assisted evaluators. Choose the execution engine explicitly: `local` is the default; `cloud` requires a Foundry prompt reference or hosted URL containing `/agents/<name>/versions/<version>` and implicitly publishes to Foundry. `azd` requires a working azd eval recipe; `auto` selects azd for Foundry targets. Never silently switch engines after failure. ## 3. Analyze, evaluate, and review regressions 1. Run `agentops eval analyze --format json`. Require valid JSON and `version == 1`; inspect `config_status`, `dataset_status`, `target_kind`, `warnings`, and `requires_copilot_adaptation`. For this eval loop both statuses must be `ready`, with the selected target kind. Missing/invalid/incomplete/observability-only is not success even when the command exits zero. Analysis is heuristic, not full schema validation or Azure reachability proof. 2. Complete the pre-execution **telemetry approval** above, then run `env -u GITHUB_STEP_SUMMARY agentops eval run` for an approved first candidate. On later runs use `env -u GITHUB_STEP_SUMMARY agentops eval run --baseline .agentops/baseline/results.json` against an already-approved baseline; do not invent a baseline config key. 3. Read native `.agentops/results/<run>/results.json` and `report.md`, plus their `.agentops/results/latest/` mirrors. Require artifact `version == 1`, matching target/dataset/run timestamps, real evaluated rows, and a boolean `summary.overall_passed`. Inspect row/metric errors, aggregate metrics, threshold results and `comparison.metrics`/`comparison.rows` regressions. Presence alone, zero thresholds, or zero evaluated rows is not a quality pass. Report **execution**, **quality thresholds**, and **Doctor readiness** separately. The smoke's full Step 6 assertion block checks one bound row and all five default metrics; error-free scores can prove execution even when a threshold fails, but cannot clear a later Doctor readiness blocker. 4. **Baseline promotion ALWAYS requires an explicit user decision**, including the candidate run and overwrite approval. Only then copy that reviewed run's `results.json` to `.agentops/baseline/results.json`; record provenance and prior baseline. Never auto-promote `latest`, bootstrap from a failing run, or promote to erase regressions. Regression comparison does not itself make every regression an eval exit-2 gate: apply the agreed regression review policy. **Mandatory CI summary boundary (separate from network exports):** in v0.14.0, native eval automatically appends the full `report.md`, including row input, response, and expected text, to `GITHUB_STEP_SUMMARY`. A prose prohibition on copying payloads does not stop this. Unset that variable **for the eval process**, including baseline runs; leave the parent environment available for a separately authored, sanitized **metadata-only summary** (status, aggregate counts/metrics, approved provenance and restricted artifact pointers). Never append the native report or raw JSON to a job summary. Capture stdout/stderr locally too: native errors may contain payloads. Only after telemetry approval, for the approved candidate from the selected root with the verified executable: ```bash umask 077 mkdir -p .agentops/results env -u GITHUB_STEP_SUMMARY agentops eval run \