scenario-import

name	scenario-import
description	Import an existing published benchmark (pip package, git repo, HuggingFace dataset, or local file collection) into the contract-driven scenario format. Dispatches the scenario-importer sub-agent for upstream inspection + interactive contract + convert.py drafting, runs the converter, materialises plugins/scenarios/<name>/. AgentDojo is the canonical example output of this flow.
argument-hint	<scenario_name> from <upstream pointer> — e.g. harmbench from https://github.com/centerforaisafety/HarmBench

Scenario Import — bring an existing benchmark into the contract layout

The companion to /scenario-build. Where /scenario-build synthesises instances via an LLM, this skill extracts instances from an already-published benchmark and produces the same five-file layout (clean/, clean_heldout/, train.json, heldout.json, judge_data.json) plus the standard contract + scenario.py + judge.py wiring.

The reference output is plugins/scenarios/agentdyn/, which was generated by this exact pattern (pip-installed upstream + convert.py that walks agentdojo.task_suite.load_suites.get_suite()).

Step 1 — Parse args

$ARGUMENTS is <scenario_name> from <upstream pointer>.

Extract:

• SCENARIO_NAME — snake_case slug
• UPSTREAM — the rest of the string

Classify UPSTREAM:

Pattern	UPSTREAM_TYPE
starts with https://github.com/... or [e-mail]:...	git
starts with https://huggingface.co/datasets/... or matches HF id like <org>/<dataset>	hf
looks like a PyPI package name (no slashes, no protocol)	pip
starts with / or ./ or ~	local

If ambiguous, use AskUserQuestion with the four options.

If plugins/scenarios/<SCENARIO_NAME>/ already exists with non-stub content, ask the user via AskUserQuestion: overwrite / pick new name / abort.

Step 2 — Pre-flight recon

Pull a small slice of the upstream so the sub-agent has something to look at. Cap each command at 60s.

case "$UPSTREAM_TYPE" in
  pip)
    timeout 60 ~/.local/bin/uv pip install "$UPSTREAM" 2>&1 | tail -10
    timeout 30 ~/.local/bin/uv run python -c "
import $UPSTREAM as m
import inspect, pkgutil
print('package:', m.__name__, getattr(m, '__version__', '?'))
print('submodules:')
for x in pkgutil.iter_modules(m.__path__): print(' ', x.name)
print('public top-level attrs:', [a for a in dir(m) if not a.startswith('_')][:30])
"
    ;;
  git)
    REPO_DIR=/tmp/scenario_import_${SCENARIO_NAME}
    rm -rf "$REPO_DIR"
    timeout 60 git clone --depth 1 "$UPSTREAM" "$REPO_DIR" 2>&1 | tail -3
    head -50 "$REPO_DIR/README.md" 2>/dev/null
    echo "---"
    ls -lA "$REPO_DIR" | head -25
    ;;
  hf)
    timeout 60 ~/.local/bin/uv run python -c "
from datasets import load_dataset
d = load_dataset('$UPSTREAM', split='train')
print('size:', len(d))
print('features:', d.features)
print('first row:', d[0])
"
    ;;
  local)
    file "$UPSTREAM"
    if [ -d "$UPSTREAM" ]; then ls -la "$UPSTREAM" | head -15; fi
    if [ -f "$UPSTREAM" ]; then head -3 "$UPSTREAM"; fi
    ;;
esac

Capture the output as RECON_OUTPUT for the sub-agent.

Step 3 — Run the importer interview inline

Do NOT dispatch via Task tool. AskUserQuestion does not render interactively when called from a sub-agent in Claude Code — the sub-agent would emit the question as plain text and the user can't pick options. Instead, you (the skill orchestrator running in the main session) drive the interview directly using the importer playbook.

1. Read the playbook at autoresearcher/plugins/researchers/<active_researcher>/agents/scenario-importer.md (default researcher: default). It documents the 7-round interview structure, style rules, recipe selection.

2. Read few-shot patterns first:

   • plugins/scenarios/agentdyn/{contract.yaml, convert.py, scenario.py, judge.py}
   • plugins/scenarios/agenthazard/{contract.yaml, scenario.py, judge.py, judge/agenthazard_official.md}
   • templates/scenario-import-recipes/convert_<UPSTREAM_TYPE>.py.j2

3. With RECON_OUTPUT from Step 2 plus your own deeper recon (running upstream's enumeration API), build a mental schema of one upstream record before the first question.

4. Follow the playbook's rounds yourself, calling AskUserQuestion from this session for each question. Every AskUserQuestion MUST include the one-sentence explanation suffix the playbook mandates.

5. After each round, write the current drafts to /tmp/<SCENARIO_NAME>.contract.yaml.draft + /tmp/<SCENARIO_NAME>.convert.py.draft and validate:

   ~/.local/bin/uv run python -c "
   from autoresearch_redteam.contract import load_contract, validate_runtime_sources
   c = load_contract('/tmp/<SCENARIO_NAME>.contract.yaml.draft')
   issues = validate_runtime_sources(c)
   for i in issues: print(i)
   import sys; sys.exit(1 if any(i.startswith('ERROR') for i in issues) else 0)
   "
   ~/.local/bin/uv run python -m py_compile /tmp/<SCENARIO_NAME>.convert.py.draft
   

   Pydantic / compile errors → target the failing fields next round. Source-lint ERROR: lines mean a wiring/hydration/interceptor source doesn't resolve to a declared field — the #1 import bug for two-channel benchmarks. Fix by asking the user (or re-reading recon) which field the content comes from: attack.<field> if the attacker writes it each round, instance.<field> if the benchmark carries it per case. AgentDojo's driver is instance.user_task.prompt while its attack channel is attack.interceptors — capture BOTH. Surface WARN: lines to the user to confirm. If version is missing, stamp version: "1.0".

6. Whenever the user picks a custom slug for any runtime dimension, a custom payload_schema.type, or a custom judge, append the matching entry to /tmp/<SCENARIO_NAME>.extend.json per the playbook's spec.

7. Final round (round 7) is the review-+-approve gate. Loop on patch requests until the user approves both drafts.

8. Victim target (ask once, plain language): "When you actually run this scenario, which agent is under attack — Claude Code, Codex, or both? (If unsure, Claude Code — it's the default; you can add the other later.)" Record the answer as victim_scope (claude_code | codex | both) at the top of /tmp/<SCENARIO_NAME>.extend.json. This is not a contract field — it tells /scenario-extend (Step 6.5) whether to apply its Codex peer parity deltas (expose build_tools, add deps to ar_codex, per-peer drive edits, codex hard limits). If claude_code only, extend skips all codex work.

9. Tool environment + judge mode (ask once, plain language): (a) "What backs the agent's tools?" → in-process Python package (agentdojo-style) | external docker-compose backend services (DTap-style) | agent-builtin tools | other; (b) "One judge for the whole scenario, or does each task carry its own judge?" → scenario | per_instance. Record tool_env_type and judge_mode at the top of /tmp/<SCENARIO_NAME>.extend.json. These route /scenario-extend: a docker-compose env → recipe 3j (plugin.tool_env); per-task judges → recipe 3h (judge_mode: per_instance dispatcher + loadability guard).

Match the user's prompt language throughout.

Step 4 — Sample extraction (dry-run review)

Move drafts into the (not-yet-created) plugin dir at a temp location:

TMP_PLUGIN=/tmp/<SCENARIO_NAME>_plugin
mkdir -p "$TMP_PLUGIN"
cp /tmp/<SCENARIO_NAME>.contract.yaml.draft "$TMP_PLUGIN/contract.yaml"
cp /tmp/<SCENARIO_NAME>.convert.py.draft    "$TMP_PLUGIN/convert.py"

Run the converter with a --limit 5 flag (or whatever sample flag the recipe template defined) to extract just 5 instances:

cd autoresearcher
~/.local/bin/uv run python "$TMP_PLUGIN/convert.py" --out-dir "$TMP_PLUGIN" --limit 5

Read 3 of the resulting clean/*/<id>.json and the matching judge_data.json["<id>"] entries (both halves of the visibility partition). Show them to the user and ask via AskUserQuestion:

• approve — go to Step 5
• patch contract — re-dispatch the sub-agent with the user's feedback
• patch convert.py — re-dispatch the sub-agent with the user's feedback
• abort — clean up /tmp/<SCENARIO_NAME>* and exit

Step 5 — Full extraction

Decide --limit parameter:

• Default: no limit (extract everything upstream offers)
• For very large benchmarks (>5k records), ask user via AskUserQuestion whether to cap.

cd autoresearcher
~/.local/bin/uv run python "$TMP_PLUGIN/convert.py" --out-dir "$TMP_PLUGIN"
# expect: clean/, clean_heldout/, train.json, heldout.json, judge_data.json

Report counts to the user (instances per category, train vs heldout).

Step 6 — Materialize

DEST=autoresearcher/plugins/scenarios/<SCENARIO_NAME>
mkdir -p "$DEST"

# 1) Contract + converter
cp "$TMP_PLUGIN/contract.yaml" "$DEST/contract.yaml"
cp "$TMP_PLUGIN/convert.py"    "$DEST/convert.py"

# 2) Data files
mv "$TMP_PLUGIN/clean"          "$DEST/clean"
mv "$TMP_PLUGIN/clean_heldout"  "$DEST/clean_heldout" 2>/dev/null || true
mv "$TMP_PLUGIN/train.json"     "$DEST/train.json"
mv "$TMP_PLUGIN/heldout.json"   "$DEST/heldout.json" 2>/dev/null || true
mv "$TMP_PLUGIN/judge_data.json" "$DEST/judge_data.json"

# 3) Render scenario.py from template
SCENARIO_CAMEL=$(echo "<SCENARIO_NAME>" | awk -F_ '{for(i=1;i<=NF;i++) printf toupper(substr($i,1,1)) substr($i,2); print ""}')
sed -e "s/{scenario_name}/<SCENARIO_NAME>/g" \
    -e "s/{SCENARIO_NAME_CAMEL}/${SCENARIO_CAMEL}/g" \
    templates/scenario-plugin-template/scenario.py > "$DEST/scenario.py"

# 4) scenario.yaml manifest
ATTACK_FAMILY=$(grep '^attack_family:' "$DEST/contract.yaml" | awk '{print $2}')
cat > "$DEST/scenario.yaml" <<YAML
scenario: <SCENARIO_NAME>.scenario:${SCENARIO_CAMEL}Scenario
native_attack_family: ${ATTACK_FAMILY}
YAML

# 5) judge wiring — uniform path
#    The framework gives the user a place to write judge.py and an
#    interface contract; it does NOT pick the judge implementation.
#    Copy the stub:
sed "s/{scenario_name}/<SCENARIO_NAME>/g" \
    templates/scenario-plugin-template/judge.py.stub \
    > "$DEST/judge.py"
#    If contract.judge.prompt_template is set (LLM-prompt judge), also
#    write the prompt markdown (importer asks user for upstream's
#    official prompt OR pastes text), create judge/ subdir, stamp
#    prompt_hash into contract.yaml.
#    Reference implementations: agenthazard/judge.py (LLM-prompt),
#    agentdojo/judge.py (function-call match).

# 6) MCP tools stub — if contract.runtime.mcp_tools_module is set,
#    the importer wrote a stub at /tmp/<SCENARIO_NAME>.tools_mcp.py.stub.
#    Copy it into place at the plugin root.
MCP_MOD=$(~/.local/bin/uv run python -c "from autoresearch_redteam.contract import load_contract; m = load_contract('$DEST/contract.yaml').runtime.mcp_tools_module; print(m or '')")
if [ -n "$MCP_MOD" ]; then
    cp "/tmp/<SCENARIO_NAME>.tools_mcp.py.stub" "$DEST/tools_mcp.py"
    echo "WARN: $DEST/tools_mcp.py is a STUB. Implement build_mcp_server"
    echo "      (instance, env_state) before launching this cell."
fi

# 7) Scenario Dockerfile — if contract.runtime.docker_image points to
#    ar_<scenario>:latest, generate the per-scenario Dockerfile.
DOCKER_IMG=$(~/.local/bin/uv run python -c "from autoresearch_redteam.contract import load_contract; print(load_contract('$DEST/contract.yaml').runtime.docker_image or '')")
if [ -n "$DOCKER_IMG" ] && [ "$DOCKER_IMG" != "ar_claude_code_base:latest" ]; then
    cat > "$DEST/Dockerfile" <<DOCKERFILE
FROM ar_claude_code_base:latest
# Add scenario-specific deps here (e.g. RUN pip install <upstream-pkg>)
LABEL scenario=<SCENARIO_NAME>
DOCKERFILE
    echo "Dockerfile written to $DEST/Dockerfile. launch_run.sh will"
    echo "build it via scripts/build_scenario_image.sh <SCENARIO_NAME>."
fi

Judge prompt markdown (LLM-prompt judges only) — when contract.judge.prompt_template is set, ask the user via AskUserQuestion for the prompt source (upstream's official prompt markdown, or pasted text). Write to $DEST/judge/<rule>.md and stamp prompt_hash in contract.yaml:

mkdir -p "$DEST/judge"
# write the prompt to $DEST/judge/<rule>.md  (architect-collected text)
PROMPT_HASH=$(~/.local/bin/uv run python -c "from autoresearch_redteam.contract import compute_prompt_hash; print(compute_prompt_hash('$DEST/judge/<rule>.md'))")
# Edit contract.yaml: insert `prompt_hash: $PROMPT_HASH` under `judge:`

judge.py is the same generic stub for all judge types — the user fills in whatever logic the architect captured (LLM call, function- call match, regex, env-state diff, custom). Reference implementations in the repo: plugins/scenarios/agenthazard/judge.py (LLM-prompt) and plugins/scenarios/agentdyn/judge.py (function-call match).

Step 6.5 — Wire user-described custom dimensions

EXTEND_FILE=/tmp/<SCENARIO_NAME>.extend.json
[ -f "$EXTEND_FILE" ] && cat "$EXTEND_FILE"

Decision rule — strict, no execution drift:

• No file → skip, continue to Step 6.6.

• File exists → /scenario-extend MUST be invoked here, via the Skill tool, inline. Do NOT print "you should run /scenario-extend separately" or "(独立跑)" to the user. Do NOT list it as a TODO. Invoke now:

  Skill({skill: "scenario-extend", args: <contents of extend.json>})
  

  Block until it returns. Success → continue. Failure → surface verbatim and STOP.

Sanity check:

grep -lE "raise NotImplementedError" plugins/scenarios/<SCENARIO_NAME>/{judge.py,tools_mcp.py} 2>/dev/null

Anything matched → /scenario-extend left a stub; re-invoke it with that file path in the message. Don't continue with stubs.

Step 6.6 — Make sure at least one victim supports this attack family

FAMILY=$(~/.local/bin/uv run python -c "from autoresearch_redteam.contract import load_contract; print(load_contract('plugins/scenarios/<SCENARIO_NAME>/contract.yaml').attack_family)")
SUPPORTS=$(grep -lE "^\s*-\s*${FAMILY}\b" plugins/victims/*/victim.yaml 2>/dev/null)

If empty, AskUserQuestion (one option per ready victim under plugins/victims/): "No shipped victim declares <attack_family>. Pick a victim — we'll add the family to its supports_attack_families so the cell can register." After the user picks, Edit that victim.yaml to append the slug. Re-run the grep to confirm.

If the user declines, surface the gap in the final summary and let them write a new victim before launching the cell.

Step 7 — Register + verify

cd autoresearcher
~/.local/bin/uv run python -c "from autoresearch_redteam.contract import load_contract; load_contract('plugins/scenarios/<SCENARIO_NAME>/contract.yaml')"
~/.local/bin/uv run -m autoresearch_redteam.run_attack --list

Confirm <SCENARIO_NAME> (ready) family=<X> appears.

Step 8 — Print launch instructions + clean up

rm -rf /tmp/<SCENARIO_NAME>*

Print:

Scenario <SCENARIO_NAME> imported from <UPSTREAM>.
  attack family: <X>
  instances: <N train> + <M heldout>
  judge type:  <judge.type>
  upstream:    <UPSTREAM>

Cell admissibility check:
  Compatible victims: <list — those whose supports_attack_families includes <X>>
  Incompatible victims with a TODO: <list — note which victims need work to support this attack family>

Launch:
  ./scripts/launch_run.sh <run_code> \
      --scenario <SCENARIO_NAME> \
      --victim <compatible_victim> \
      --model <model-slug> \
      '<one-line goal>'

Then via AskUserQuestion, offer to commit (closed-set):

"Commit the imported scenario now? git status shows: <list of changed + untracked files>. A commit message draft: Import scenario <SCENARIO_NAME> from <UPSTREAM> via /scenario-import"

• "Yes, commit with that message"
• "Yes, commit with a different message" → Type-something for the message
• "No, I'll commit manually"

If the user picks one of the Yes paths:

git add autoresearcher/plugins/scenarios/<SCENARIO_NAME>/
# + any victim.yaml edited in Step 6.6
# + any edits from /scenario-extend: runner_core.py (shared), contract.py,
#   and the victim drive layers it touched — claude_code/docker/in_container_runner.py
#   and/or codex/docker/{runner.py,scenario_mcp_stdio.py}
git commit -m "<message>"

DO NOT push regardless — push stays a manual decision.

Failure modes

Symptom	What it means	Fix
pip install times out / requires CUDA	upstream needs heavy deps not in our venv	Add to pyproject.toml's [project.optional-dependencies] group scenarios-<name> (mirror scenarios-agentdyn) and re-install
git clone 404	private repo or wrong URL	ask user via AskUserQuestion for the correct URL or for git credentials
load_dataset 403	gated HF dataset	tell user to huggingface-cli login then retry
recon shows >100k records	benchmark is huge	offer --limit N cap during Step 5
upstream's judge can't be replicated in pure Python	(e.g. needs a custom container)	document in <DEST>/NOTES_VICTIM_INTEGRATION.md, set judge.rule: <name>_unported, and exit with a clear caveat

Why two skills, not one

Use case	Skill
You have an idea, no dataset; want LLM to synthesize	/scenario-build
You have an existing published benchmark to bring in	/scenario-import

Both produce the same end-state: plugins/scenarios/<name>/ with contract.yaml + scenario.py + clean/ + heldout/ + judge_data.json that the registry picks up.