agentops-dataset

# AgentOps Dataset Generate a small, realistic JSONL dataset for the agent under evaluation. Default location: `.agentops/data/smoke.jsonl` (referenced from `agentops.yaml`). These rows are repo-side release-gate inputs: keep them reviewable and deterministic, not a full replacement for Foundry dataset management. ## Step 0 - Prerequisites 1. `pip install "agentops-accelerator @ git+https://github.com/Azure/agentops.git@main"` if `agentops` is missing. 2. Run `agentops eval analyze` first. If it reports missing dataset columns or recommends `agentops-dataset`, use this skill before the first eval run. 3. If `agentops.yaml` does not exist, run `agentops init` first (the init wizard will prompt for the agent reference, project endpoint, and dataset path, then create a starter `.agentops/data/smoke.jsonl`). ## Step 1 - Pick the columns Read `agentops.yaml` (and the agent code) to figure out the agent type, then choose the row schema: | Agent type | Required columns | Optional columns | |---|---|---| | Direct model / Q&A | `input`, `expected` | - | | RAG | `input`, `expected`, `context` | - | | Conversational | `input`, `expected` | - | | Tool-using agent | `input`, `expected`, `tool_calls` | `tool_definitions` | `input` is always the user prompt. `expected` is the gold answer. `context` is the retrieved passage(s). `tool_calls` is a list of `{name, arguments}` describing the expected tool invocations. ## Step 2 - Ground the rows in the codebase - Read the README, system prompt, tool definitions, and any sample fixtures. - Generate **5–10 rows** that exercise the agent's actual capabilities. - If the domain is unclear, generate a tiny generic draft and clearly flag it as a placeholder. ## Step 3 - Write the JSONL One JSON object per line, no trailing commas, UTF-8: ```json {"input": "What is the refund policy?", "expected": "Refunds within 30 days...", "context": "Refund policy: ..."} ``` Save to the path referenced by `dataset:` in `agentops.yaml` (default `.agentops/data/smoke.jsonl`). This file is the AgentOps source of truth. In Foundry cloud evaluation, AgentOps syncs it to a stable Foundry dataset version by default and reuses the same Foundry dataset version while the JSONL content is unchanged. If the user forces `dataset_sync.mode: inline`, Foundry may show generated `eval-data-*` backing assets in the project Data/Datasets page. ## Step 4 - Sanity-check Run a quick eval and confirm rows are picked up: ```bash agentops eval run ``` Open `.agentops/results/latest/report.md` and confirm the row count matches. ## Guardrails - Do not invent customer data, real names, or sensitive content. - Keep rows short - datasets are meant to be quick gates, not full QA suites. - If the user already has a domain dataset, prefer pointing `agentops.yaml` at that file rather than generating new rows. - If the user asks why Foundry shows `eval-data-*`, explain that those are cloud-eval backing assets from inline compatibility mode; normal cloud runs should use the stable `agentops-*` Foundry dataset.