agentops-config

name	agentops-config
description	Generate or update agentops.yaml (flat 1.0 schema) for AgentOps release-readiness gates. Trigger on "configure agentops", "agentops.yaml", "set up evaluation", "what should I evaluate". Infer the agent target and dataset from the codebase; ask only when nothing can be found.

AgentOps Config

Generate agentops.yaml at the project root. This file references the agent candidate and dataset used to answer "can we ship it?" The flat schema has only a handful of fields - most projects need just version, agent, and dataset.

This skill configures AgentOps release gates. It does not create or deploy Foundry agents. If the user needs to create/deploy a Prompt Agent or Hosted Agent first, hand off to Foundry Toolkit / the microsoft-foundry skill / azd, then return here once there is a name:version or URL.

Step 0 - Prerequisites

1. pip install "agentops-accelerator @ git+https://github.com/Azure/agentops.git@main" if agentops is missing.
2. Run agentops eval analyze first. If it reports missing or ambiguous target/dataset/scenario signals, use this skill to adapt the config.
3. If agentops.yaml does not exist, run agentops init first. The init wizard already collects the agent reference and dataset path, so agentops-config is most useful when the user wants to tweak an existing config (add thresholds, switch to a different agent target, add HTTP auth headers, etc.) rather than create one from scratch.

Step 1 - Detect the agent target

Search the codebase for the strongest signal and pick one:

Signal	agent: value
Foundry Prompt Agent ID name:N	"<name>:<N>"
Foundry Hosted Agent URL https://...services.ai.azure.com/...agents/...	the full URL
Any other HTTP endpoint your agent serves (FastAPI, Express, ACA, AKS)	the full URL
Direct model use (openai.chat.completions.create(model=...)) with no orchestration	"model:<deployment-name>"

Look in: README.md, main.py/server.py/app.ts, .agentops/.env, .env/.env.local, .azure/<env>/.env, infra/, IaC outputs. If nothing is found, ask the user once.

Step 2 - Detect the dataset

If a JSONL with rows that include input already exists in the repo, use its path. Otherwise leave the default .agentops/data/smoke.jsonl and hand off to the agentops-dataset skill before the first run.

Step 3 - Write agentops.yaml

Minimal example:

version: 1
agent: "my-rag:3"
dataset: .agentops/data/smoke.jsonl

HTTP/JSON example:

version: 1
agent: "https://my-aca-app.eastus2.azurecontainerapps.io/chat"
dataset: .agentops/data/smoke.jsonl
request_field: message      # default is "message"
response_field: text         # dot-path; default is "text"
auth_header_env: MY_API_TOKEN

Optional extras (only add when the user asks for them):

thresholds:
  coherence: ">=3"
  groundedness: ">=3"
  avg_latency_seconds: "<=30"

# Prompt-agent only: auto-bootstrap empty Foundry projects on first deploy.
# When the deploy workflow runs against a Foundry project that does not yet
# contain the agent named in `agent:`, AgentOps reads this block plus
# `prompt_file` and creates the first version automatically. Recommended
# for multi-environment prompt-agent workflows (sandbox → dev → qa → prod)
# so operators do not have to manually recreate the seed agent in every
# Foundry project.
prompt_agent_bootstrap:
  model: gpt-4o-mini        # required - same deployment name in every env
  description: "Helps plan short trips."
  # model_parameters:        # optional - temperature, top_p, etc.
  #   temperature: 0.2
  # tools: []                # optional - tool definitions

# Publish results to the Foundry Evaluations panel.
# - execution: local + publish: true  → Classic Foundry (uploads metrics)
# - execution: cloud                  → New Foundry (server-side run;
#                                       publish is implicit, cloud always publishes)
execution: local
publish: true
# project_endpoint: "https://<resource>.services.ai.azure.com/api/projects/<p>"

# Cloud dataset submission policy. The local JSONL remains the source of truth;
# cloud runs sync it to Foundry Data/Datasets by default.
dataset_sync:
  mode: auto            # auto | foundry | inline
  
  

           
    
     

Governance evidence (optional): when the repo already contains ASSERT, ACS, or red-team evidence artifacts, wire them into Doctor/release evidence without executing the external tools:

assert_path: .assert/evaluation-policy.yaml
acs_path: acs.yaml
redteam_path: .agentops/governance/redteam-plan.md

If the user needs help drafting these files, switch to the agentops-governance skill. AgentOps validates and hashes these artifacts; it does not execute ASSERT, apply ACS controls, or run red-team campaigns.

Step 4 - Validate

Run agentops eval run once. If the config is malformed AgentOps prints a clear error pointing at the offending key. Adjust and re-run.

Guardrails

• Do not add legacy keys (bundle, target, execution, output, backend). The 1.0 schema rejects them.
• Do not fabricate agent IDs, endpoint URLs, or model deployment names. Ask the user when uncertain.
• Keep the file small. Auto-selection covers most metrics.
• Keep local JSONL canonical. For cloud runs, prefer dataset_sync.mode: auto so AgentOps keeps Foundry Data/Datasets in sync; use inline only for quick experiments or environments without dataset upload permission.