agentdojo-report

name	agentdojo-report
description	Generate a Marp slide deck (MD + HTML) summarizing an AgentDojo run — headline metrics, case taxonomy, failure buckets, cache health, and optional baseline diff. Use when the user asks for a slide/report of an experiment result.

agentdojo-report

Produce a presentable slide deck for any runs/<dir>/. Reuses agentdojo-analyze, agentdojo-failures, and (for DAMI runs) agentdojo-cache-doctor to gather numbers, renders a Marp markdown template, then converts to a standalone HTML deck via marp-cli --html (no browser required).

Why HTML, not PPTX: marp-cli's PPTX/PDF paths require a real Chrome install (libnspr4/libnss3/libcups/etc.) that this host doesn't have. pandoc's direct MD→PPTX works but produces decks that don't open cleanly. --html is the only robust path — self-contained file, renders in any browser, preserves Marp theming.

Inputs

• logdir (required) — run dir to report on.
• baseline (optional) — second run dir for a diff. Great for "DAMI v1 pre-fix vs post-fix" or "ChatInject no_defense vs tool_filter".
• title (optional) — slide title. Default: <attack> on <model_id>.
• out (optional) — stem path. Default: reports/<logdir_basename>_<YYYYMMDD>. Two files are produced: <stem>.md and <stem>.html.

Procedure

1. Gather metrics (reuse existing skills rather than recounting):
   • Run scripts/watch_run.py --logdir <logdir> --once → read _live_metrics.json.
   • Same for baseline if provided.
   • Note: prefer calling the helper skills when the user is working interactively; when running autonomously, inline the watcher call directly.
2. Gather failure taxonomy — invoke the same python walk as agentdojo-failures and keep top 2 quotes per bucket (≤180 chars each).
3. Cache health (only if attack starts with dami_attack_v1) — run the cache-doctor summary (no --quarantine).
4. Render Marp markdown using the template below. Fill all {{…}} placeholders.
5. Write the MD to reports/:
   • mkdir -p reports first.
   • If <stem>.md already exists, bump to <stem>_v2.md, _v3.md, etc.
6. Convert to PPTX:

   npx @marp-team/marp-cli@latest <stem>.md --pptx --allow-local-files -o <stem>.pptx
   

   First run may download Chromium (~slow). Warn the user. Subsequent runs are fast.
7. Verify & report — check <stem>.pptx exists and size > 10 KB. Print both paths to the user.

Marp template

---
marp: true
theme: default
paginate: true
size: 16:9
---

# {{title}}
**Run:** `{{logdir}}`
**Model:** {{model}}  **Attack:** {{attack}}  **Defense:** {{defense}}
**Traces:** {{total}}  **Generated:** {{YYYY-MM-DD}}

---

## Headline — 4-Case Breakdown

| Case          | {{run_label}} | {{baseline_label}} | Δ |
|---------------|--------------:|-------------------:|--:|
| **Both ⚠️** (u=T, s=T) | **{{both_pct}}** | {{b_both_pct}} | {{d_both}} |
| AtkOnly (u=F, s=T)     | {{ao_pct}}   | {{b_ao_pct}}   | {{d_ao}}   |
| DefOnly ✅ (u=T, s=F)   | {{do_pct}}   | {{b_do_pct}}   | {{d_do}}   |
| Failed (u=F, s=F)       | {{fail_pct}} | {{b_fail_pct}} | {{d_fail}} |

**Stealth KPI = Both%** — this is what a stealth attack must improve. ASR/Utility are derived.

---

## Derived Metrics

| Metric  | {{run_label}} | {{baseline_label_or_blank}} | Δ |
|---------|--------------:|----------------------------:|--:|
| ASR     = (Both+AtkOnly)/n | {{asr}} | {{b_asr}} | {{d_asr}} |
| Utility = (Both+DefOnly)/n | {{util}} | {{b_util}} | {{d_util}} |
| DSR     = 1 − ASR          | {{dsr}} | {{b_dsr}} | {{d_dsr}} |
| Errored                    | {{err}} | {{b_err}} | — |

**Shape check**: Both / (Both + AtkOnly) = {{stealth_share}} — > 0.45 means stealth-dominant, < 0.25 means brute-force.

---

## Per-Suite

| Suite     |    N | Utility |  ASR |
|-----------|-----:|--------:|-----:|
| workspace | {{ws_n}} | {{ws_u}} | {{ws_a}} |
| banking   | {{bk_n}} | {{bk_u}} | {{bk_a}} |
| travel    | {{tr_n}} | {{tr_u}} | {{tr_a}} |
| slack     | {{sl_n}} | {{sl_u}} | {{sl_a}} |

---

## Why the Attack Failed

| Bucket        | Count |
|---------------|------:|
| ignored       | {{ign}} |
| refused       | {{ref}} |
| wrong_tool    | {{wt}}  |
| utility_drop  | {{ud}}  |

> **[ignored]** {{ignored_quote_1}}
> **[refused]** {{refused_quote_1}}

---

{{cache_slide_if_dami}}

---

## What Changed
{{diff_narrative_or_skip}}

---

## Next Steps
- {{next_1}}
- {{next_2}}

The {{cache_slide_if_dami}} block, when present, looks like:

## DAMI v1 Cache Health

- total: {{cache_total}}
- valid (body ≥ 20): {{cache_valid}}
- empty: {{cache_empty}}
- body length: min={{cmin}}, median={{cmed}}, max={{cmax}}
- prev_empty archive: {{cache_quar}} files

Tool usage

• Bash for the watcher call, cache inspection python, and the npx marp-cli conversion.
• Read for _live_metrics.json.
• Write for the generated MD.
• Never call marp-cli without --allow-local-files — otherwise local images (if any future template uses them) won't render.

Guardrails

• Write only under reports/. Never overwrite an existing <stem>.md or <stem>.pptx — bump _v2, _v3.
• Truncate all quotes to ≤180 chars (ellipsis) so no raw trace content leaks into the deck.
• If marp-cli conversion fails (e.g. chromium download blocked), leave the MD intact and tell the user: "MD generated at X; pptx failed, run npx @marp-team/marp-cli@latest X --pptx manually after chromium is available."
• Do not embed private tokens/paths — only the logdir name and metric numbers.
• If baseline is supplied but the two runs have different total counts, flag it on the "What Changed" slide — comparisons across different trace sets are misleading.

Example output to the user

Report generated:
  reports/qwen3-32b-dami_attack_v1-no_defense_20260413.md    (18 slides)
  reports/qwen3-32b-dami_attack_v1-no_defense_20260413.pptx  (142 KB)

Headlines vs baseline (DAMI v1 pre-fix):
  ASR   0.013 → 0.183  (+0.170)
  Util  0.676 → 0.612  (−0.064)
  Cache  5 valid / 0 empty  (was 231 / 718)

Next: open the pptx or rerun agentdojo-analyze for per-suite drill.