agentdojo-run

name	agentdojo-run
description	Launch an AgentDojo benchmark run (model × attack × defense × suites) with automatic watcher, smoke/full modes, and DAMI v1 safety guards. Use when the user wants to kick off or rerun an experiment.

agentdojo-run

Launch an AgentDojo benchmark. Handles shell-script reuse/creation, live metrics watcher, DAMI-specific preflight, and end-of-run summary.

Inputs to parse from the user request

Field	Default	Notes
model	qwen3:32b (local ollama)	Maps to --model LOCAL --model-id <id>. Other models use their enum value.
attack	required	e.g. dami_attack_v1, chat_inject_qwen3, important_instructions.
defense	none (omit flag)	One of tool_filter, melon, ipiguard, spotlighting_with_delimiting, repeat_user_prompt, transformers_pi_detector, necessity_judge, task_shield.
suites	workspace travel banking slack	Subset allowed.
mode	full	smoke = 3 user tasks × 2 injection tasks on one suite.
logdir	runs/<model>-<attack>-<defense> (auto)	Override only if user asks.
gpu/port	GPU1/11446 default, GPU3/11436 alt	See CLAUDE.md infra table.

Procedure

1. Check for existing shell script. Grep shell_scripts/ for a matching run_*_<attack>_*.sh. If found, reuse it. If not, generate one from the run_qwen3_dami_attack_v1_no_defense.sh template, substituting model/attack/defense/suite.
2. DAMI v1 preflight (only if attack == dami_attack_v1):
   • Call agentdojo-cache-doctor first. If empty entries exist, ask user whether to quarantine before proceeding.
   • Ensure DAMI_STRICT=1 is exported in the generated script.
   • Verify the script's PROBE block uses max_tokens:1024 (not 32).
3. Guard existing run dir. If runs/<logdir>/ already has traces, move them to runs/_archive/$(date +%Y-%m-%d)_<name>/ before starting. Never overwrite silently. Only pass --force-rerun if the user explicitly requested it.
4. Start watcher in background:

   /opt/conda/envs/agentdojo/bin/python scripts/watch_run.py \
     --logdir runs/<logdir> --interval 5 --stop-when-idle 120 \
     > logs/<name>_watcher.log 2>&1
   

   Use run_in_background=true Bash.
5. Start benchmark in background, logging to logs/<name>.log. Same run_in_background=true.
6. Monitor: read runs/<logdir>/_live_metrics.json every few minutes. Report running ASR/utility to user at natural pause points.
7. On completion (task-notification arrives, or watcher logs exiting):
   • Read final _live_metrics.json.
   • Grep the run log for [dami_v1], RuntimeError, Traceback, FATAL.
   • Summarize: total traces, ASR, DSR, utility, per-suite breakdown, any warnings.
   • If agentdojo-analyze is available and a baseline exists (e.g. the archived previous run), suggest a diff.

Smoke mode

When mode == smoke, bypass the full script and use run_qwen3_dami_attack_v1_no_defense_smoke.sh as the template: single suite, -ut user_task_0 -ut user_task_1 -ut user_task_2 -it injection_task_0 -it injection_task_1, --force-rerun, logdir suffix -smoke. Expect ~5 minutes on qwen3:32b.

Guardrails

• Never rm -rf runs/... — always move to _archive/.
• Never skip the DAMI v1 probe — a failed probe must abort.
• Never run more than one benchmark against the same ollama port concurrently — it serializes and results become unreliable.
• If the user asks for multiple (model × attack × defense) combinations, run them sequentially unless they explicitly ask for parallel on different GPUs.
• Keep the user informed during long runs by reading _live_metrics.json at ~5 min intervals; don't poll faster or you'll churn cache.

Output to user at end of turn

A short block like:

DAMI v1 rerun complete (no defense):
  total=949  ASR=0.183  DSR=0.817  utility=0.612
  per-suite: workspace 0.21 / banking 0.17 / travel 0.19 / slack 0.15
  warnings: 12 empty attacker responses (all retried successfully)
Next step: run agentdojo-failures on this dir to drill into the ~82% failures.