agentops-config

# AgentOps Config Generate `agentops.yaml` at the project root. This file references the agent candidate and dataset used to answer "can we ship it?" The flat schema has only a handful of fields - most projects need just `version`, `agent`, and `dataset`. This skill configures AgentOps release gates. It does **not** create or deploy Foundry agents. If the user needs to create/deploy a Prompt Agent or Hosted Agent first, hand off to Foundry Toolkit / the `microsoft-foundry` skill / azd, then return here once there is a `name:version` or URL. ## Step 0 - Prerequisites 1. `pip install "agentops-accelerator @ git+https://github.com/Azure/agentops.git@main"` if `agentops` is missing. 2. Run `agentops eval analyze` first. If it reports missing or ambiguous target/dataset/scenario signals, use this skill to adapt the config. 3. If `agentops.yaml` does not exist, run `agentops init` first. The init wizard already collects the agent reference and dataset path, so `agentops-config` is most useful when the user wants to **tweak** an existing config (add thresholds, switch to a different agent target, add HTTP auth headers, etc.) rather than create one from scratch. ## Step 1 - Detect the agent target Search the codebase for the strongest signal and pick one: | Signal | `agent:` value | |---|---| | Foundry Prompt Agent ID `name:N` | `"<name>:<N>"` | | Foundry Hosted Agent URL `https://...services.ai.azure.com/...agents/...` | the full URL | | Any other HTTP endpoint your agent serves (FastAPI, Express, ACA, AKS) | the full URL | | Direct model use (`openai.chat.completions.create(model=...)`) with no orchestration | `"model:<deployment-name>"` | Look in: `README.md`, `main.py`/`server.py`/`app.ts`, `.agentops/.env`, `.env`/`.env.local`, `.azure/<env>/.env`, `infra/`, IaC outputs. If nothing is found, ask the user once. ## Step 2 - Detect the dataset If a JSONL with rows that include `input` already exists in the repo, use its path. Otherwise leave the default `.agentops/data/smoke.jsonl` and hand off to the `agentops-dataset` skill before the first run. ## Step 3 - Write agentops.yaml Minimal example: ```yaml version: 1 agent: "my-rag:3" dataset: .agentops/data/smoke.jsonl ``` HTTP/JSON example: ```yaml version: 1 agent: "https://my-aca-app.eastus2.azurecontainerapps.io/chat" dataset: .agentops/data/smoke.jsonl request_field: message # default is "message" response_field: text # dot-path; default is "text" auth_header_env: MY_API_TOKEN ``` Optional extras (only add when the user asks for them): ```yaml thresholds: coherence: ">=3" groundedness: ">=3" avg_latency_seconds: "<=30" # Prompt-agent only: auto-bootstrap empty Foundry projects on first deploy. # When the deploy workflow runs against a Foundry project that does not yet # contain the agent named in `agent:`, AgentOps reads this block plus # `prompt_file` and creates the first version automatically. Recommended # for multi-environment prompt-agent workflows (sandbox → dev → qa → prod) # so operators do not have to manually recreate the seed agent in every # Foundry project. prompt_agent_bootstrap: model: gpt-4o-mini # required - same deployment name in every env description: "Helps plan short trips." # model_parameters: # optional - temperature, top_p, etc. # temperature: 0.2 # tools: [] # optional - tool definitions # Publish results to the Foundry Evaluations panel. # - execution: local + publish: true → Classic Foundry (uploads metrics) # - execution: cloud → New Foundry (server-side run; # publish is implicit, cloud always publishes) execution: local publish: true # project_endpoint: "https://<resource>.services.ai.azure.com/api/projects/<p>" # Cloud dataset submission policy. The local JSONL remains the source of truth; # cloud runs sync it to Foundry Data/Datasets by default. dataset_sync: mode: auto # auto | foundry | inline # name: agentops-smoke # version: content-hash evaluators: # rare - AgentOps auto-selects from agent + dataset - name: similarity threshold: ">=4" ``` Governance evidence (optional): when the repo already contains ASSERT, ACS, or red-team evidence artifacts, wire them into Doctor/release evidence without executing the external tools: ```yaml assert_path: .assert/evaluation-policy.yaml acs_path: acs.yaml redteam_path: .agentops/governance/redteam-plan.md ``` If the user needs help drafting these files, switch to the `agentops-governance` skill. AgentOps validates and hashes these artifacts; it does not execute ASSERT, apply ACS controls, or run red-team campaigns. ## Step 4 - Validate Run `agentops eval run` once. If the config is malformed AgentOps prints a clear error pointing at the offending key. Adjust and re-run. ## Guardrails - Do **not** add legacy keys (`bundle`, `target`, `execution`, `output`, `backend`). The 1.0 schema rejects them. - Do **not** fabricate agent IDs, endpoint URLs, or model deployment names. Ask the user when uncertain. - Keep the file small. Auto-selection covers most metrics. - Keep local JSONL canonical. For cloud runs, prefer `dataset_sync.mode: auto` so AgentOps keeps Foundry Data/Datasets in sync; use `inline` only for quick experiments or environments without dataset upload permission.