| name | agentops-report |
| description | Read, regenerate, and explain AgentOps release-gate reports. Trigger on "show report", "explain scores", "regenerate report", "what do these metrics mean", "where is the proof". Operates on results.json and report.md produced by `agentops eval run`. |
AgentOps Report
Help the user understand a finished AgentOps run and the evidence it provides
for the release decision. Reports explain the repo-side gate; they do not
replace Foundry Evaluations, Traces, or Monitor drilldown.
Step 0 - Locate the run
Latest run: .agentops/results/latest/. Each run produces:
results.json - machine-readable metrics, per-row scores, thresholds.report.md - human-readable summary suitable for PR comments.cloud_evaluation.json (when Foundry visibility is enabled) - deep-link
to the Foundry Evaluations panel. mode: classic when execution: local
and publish: true upload metrics to Classic Foundry; mode: cloud when
execution: cloud runs server-side via the OpenAI Evals API.
Step 1 - Regenerate report.md if needed
agentops report generate
agentops report generate --in <results.json> --out <report.md>
report generate always reads the flat 1.0 results schema and emits
Markdown. There is no HTML format.
Step 2 - Explain the metrics
Common metrics and their meaning:
| Metric | Range | Higher is better? | Notes |
|---|
similarity | 1-5 | yes | LLM-judged similarity to expected. |
coherence | 1-5 | yes | Answer is internally consistent. |
fluency | 1-5 | yes | Natural language quality. |
groundedness | 1-5 | yes | Answer is supported by context (RAG). |
relevance | 1-5 | yes | Answer is on-topic for input. |
f1_score | 0-1 | yes | Token overlap with expected. |
tool_call_accuracy | 0-1 | yes | Predicted tool calls match tool_calls. |
intent_resolution | 0-1 | yes | User intent was resolved. |
task_completion | 0-1 | yes | Multi-step task finished. |
avg_latency_seconds | seconds | no | Wall-clock latency per row. |
Pass/fail rows are derived from thresholds: in agentops.yaml. The
exit code of the original run reflects the gate:
0 → all thresholds passed2 → one or more thresholds failed1 → runtime error
Step 3 - Help the user act on results
- For low scores on a specific metric, point at the lowest-scoring rows
in
results.json (row_metrics[] and item_evaluations[]) and
suggest concrete prompt or retrieval changes.
- For latency regressions, look at
run_metrics.avg_latency_seconds and
per-row latency.
- To compare a new run against a previous one, re-run with
agentops eval run --baseline <previous-results.json> and explain the
generated Comparison vs Baseline section.
- For production evidence, open
.agentops/release/latest/evidence.md. If it
includes a governance section, explain ASSERT/ACS/red-team entries as
read-only proof references (path, hash, status, ACS checkpoint coverage), not
as proof that AgentOps executed those frameworks.
Guardrails
- Never invent metric values. If a metric is absent, say so.
- Do not edit
results.json by hand - re-run the eval.
- Do not paste raw red-team payloads into the response; summarize only metadata
and reviewer/status information.
