run-evals-and-graph

name	run-evals-and-graph
description	Run the eval app, gather the JSON snapshots EvalResultStore writes, and render charts (per-run scores for stochastic evals; mean+spread summary). Use when the user wants visual eval output for a writeup, a slide, or a comparison.

You're producing visual eval output. Upstream AgentEval already supports JSON / JUnit / Markdown / TRX / CSV exports (see docs/export.md upstream), but it does not ship a chart renderer. This skill adds that layer using the snapshots EvalResultStore already persists.

If a future upstream release adds native charting, swap the local script for the upstream command — but right now this is the simplest path that works.

Inputs

EvalResultStore writes one JSON file per snapshot to:

.AgentEval/ECS2026MAF_Evals/{key}.json

Single-run evals (Eval01, Eval02, Eval03) write EvalSnapshot records. Stochastic evals (Eval04, Eval05) write StochasticSnapshot records — those have a Scores array, MeanScore, MinScore, MaxScore, Spread worth charting.

Schema lives in AgentEval/samples/ECS2026MAF.Eval/EvalResultStore.cs.

Step 1 — Run the evals

Interactive run:

dotnet run --project AgentEval/samples/ECS2026MAF.Eval

Press the menu key for the eval(s) you care about. For stochastic comparisons, run 4 and 5 so both snapshots exist.

If the eval bodies are still placeholders, this skill has nothing useful to chart. Replace at least one body via the create-eval-test skill first.

If you want a non-interactive runner, that's a separate piece of work — see add-console-prompt for the menu pattern; you'd add a top-level args[0] switch in Program.cs that calls RunAsync directly.

Step 2 — Render charts

The bundled render.py reads .AgentEval/ECS2026MAF_Evals/*.json and writes PNG + SVG to .AgentEval/charts/. Run from the repo root:

python .agent/skills/run-evals-and-graph/render.py

PowerShell on Windows is identical:

python .agent\skills\run-evals-and-graph\render.py

The script needs matplotlib. Install once:

python -m pip install --user matplotlib

What it produces:

• For each StochasticSnapshot — a per-run bar chart with the pass/fail line drawn, mean/min/max annotations.
• A side-by-side mean/spread summary across all stochastic snapshots in the directory (useful for the "agent vs workflow" comparison).
• For each EvalSnapshot — a small criteria-met-vs-missed bar.

Outputs land at .AgentEval/charts/{snapshot-key}.png and .AgentEval/charts/{snapshot-key}.svg.

Step 3 — Sanity check before sharing

The eval snapshots never contain the API key (only score numbers and labels), so charts are safe to paste into slides. But:

• Don't include the endpoint URL or deployment name in chart titles if you're sharing externally — those leak which subscription/region you're using.
• Don't share dotnet user-secrets list output as a chart, ever. That's a different shape of artefact and it's never appropriate for sharing.

Step 4 — Cleanup

Charts are gitignored (.AgentEval/charts/). If you want to keep one, copy it out of .AgentEval/charts/ into wherever you store talk assets — don't try to commit it.

Don'ts

• Don't add a charting NuGet to the .NET project. Keep it as an out-of-band Python tool — the .NET app stays minimal.
• Don't extend EvalResultStore with chart-rendering responsibility. It's a serialiser; the renderer is downstream.
• Don't pull pandas or seaborn for what matplotlib can do directly. Slim is the goal.