autoqec-run

name	autoqec-run
description	Run the AutoQEC research loop on a given env YAML. Orchestrates the autoqec-ideator / autoqec-coder / autoqec-analyst subagents via the Agent tool, invokes the Runner CLI for training + evaluation, and writes history.jsonl + pareto.json. Use when the user asks "run AutoQEC on <env>", "start a research round", or provides an EnvSpec YAML.

/autoqec-run

One-shot orchestration recipe for Claude Code chat. The orchestrator (you) drives the loop; three autoqec-{ideator,coder,analyst} subagents live behind the Agent tool.

All shell commands below are written as python -c "..." one-liners so the recipe is identical on PowerShell (Windows) and bash / zsh (macOS / Linux). The helpers resolve every path relative to the repo root via _REPO_ROOT = Path(__file__).resolve().parents[...] — no cwd assumptions.

When to use

• The user says "run AutoQEC", "start a research round", or provides an env YAML path.
• The user wants to see an AI-generated predecoder candidate on an existing env (surface_d5_depol, bb72_depol, etc.).

Required inputs

• env_yaml — path to an EnvSpec YAML (e.g. autoqec/envs/builtin/surface_d5_depol.yaml).
• rounds — default 1. Each round is Ideator → Coder → Runner → Analyst and takes ~2–5 min in dev profile.
• profile — dev (2048 train / 2048 val shots, 3 epochs) or prod (8192 train / 8192 val shots, 10 epochs). Val shots matter more than train shots here because Δ_LER stderr scales as 1/√n_val — undersized val makes every round statistically indistinguishable from zero. Default dev.

Default output_mode

Always default to output_mode: soft_priors unless the user or the Ideator's hypothesis explicitly asks for hard_flip. soft_priors is the only mode with a well-posed supervised training target (DEM-error labels) and a working eval path on both MWPM (per-sample DEM reweighting) and OSD (priors passed to bposd_decoder). hard_flip is still supported but treats the raw syndrome as a weak surrogate target during training — it will not reliably move delta_ler without a problem-specific loss. If you find yourself emitting a DSL with output_mode: hard_flip without an explicit Ideator rationale, stop and check.

Working scratch directory

Use Python tempfile.gettempdir() when you need to stash intermediate subagent responses. Never hard-code /tmp/....

python -c "import tempfile,os; print(os.path.join(tempfile.gettempdir(), 'autoqec-run'))"

Create it once per run; the steps below refer to it as <SCRATCH>.

The loop (per round)

Do these steps in order. If any step fails, halt and report the failure to the user — do not silently skip.

1. Prepare the run + round directory

Pick a run id (UTC YYYYMMDD-HHMMSS) if the user did not supply one. All subsequent paths live under runs/<run_id>/. Initialise memory and the orchestrator trace via the Python helpers — do NOT write history.jsonl / pareto.json / log.md / orchestrator_trace.md by hand:

python -c "
from autoqec.orchestration.memory import RunMemory
from autoqec.orchestration.trace import init_trace
RunMemory('runs/<run_id>')
init_trace('runs/<run_id>', env_yaml='<env_yaml>', rounds=<N_TOTAL>, profile='<profile>')
print('runs/<run_id>')
"

init_trace is called once per run (before round 1). If orchestrator_trace.md already exists (resuming a killed run), it appends a _resumed at ..._ divider instead of truncating. Every later step in this SKILL appends sections to that file via append_section / append_note so you get a chat-level narrative next to history.jsonl.

2. Build the Ideator prompt

python scripts/run_single_round.py --env-yaml <env_yaml> --run-dir runs/<run_id> --round-idx <N>

Capture stdout into a JSON variable (the whole plan dict). The ideator_prompt key inside is the string to hand to the subagent.

3. Dispatch the Ideator

Use the Agent tool with subagent_type="autoqec-ideator" and pass the ideator_prompt verbatim. The subagent returns exactly one fenced

```bash
python -c "
import sys, json
from autoqec.agents.dispatch import parse_response
from autoqec.orchestration.trace import append_note, append_section
raw = sys.stdin.read()
append_section('runs/<run_id>', <N>, 'ideator prompt', <IDEATOR_PROMPT>)
append_note('runs/<run_id>', <N>, 'dispatched autoqec-ideator')
append_section('runs/<run_id>', <N>, 'ideator raw', raw)
parsed = parse_response('ideator', raw)
append_section('runs/<run_id>', <N>, 'ideator parsed', parsed)
print(json.dumps(parsed))
"

Pipe the subagent's raw response on stdin. Required keys (validated automatically by parse_response): hypothesis, expected_delta_ler, expected_cost_s, rationale. Optional: dsl_hint.

If parse_response raises ValueError: Invalid ideator response payload:, the response is malformed — re-dispatch once with the pydantic error appended to the prompt, then give up if it fails again.

4. Build the Coder prompt

python -c "
import json, sys
from pathlib import Path
from autoqec.orchestration.memory import RunMemory
from autoqec.orchestration.loop import build_coder_prompt
hypothesis = json.loads(sys.stdin.read())
mem = RunMemory('runs/<run_id>')
schema_md = Path('autoqec/decoders/dsl_schema.py').read_text(encoding='utf-8')
print(build_coder_prompt(hypothesis=hypothesis, mem=mem, dsl_schema_md=schema_md))
"

Pipe the parsed Ideator JSON on stdin.

5. Dispatch the Coder

Agent tool with subagent_type="autoqec-coder". Required response keys (validated by parse_response): tier, dsl_config, rationale. tier is "1" or "2".

Double-check dsl_config against PredecoderDSL and trace:

python -c "
import json, sys
from autoqec.agents.dispatch import parse_response
from autoqec.decoders.dsl_schema import PredecoderDSL
from autoqec.orchestration.trace import append_note, append_section
raw = sys.stdin.read()
append_section('runs/<run_id>', <N>, 'coder prompt', <CODER_PROMPT>)
append_note('runs/<run_id>', <N>, 'dispatched autoqec-coder')
append_section('runs/<run_id>', <N>, 'coder raw', raw)
parsed = parse_response('coder', raw)
PredecoderDSL(**parsed['dsl_config'])   # raises on schema drift
append_section('runs/<run_id>', <N>, 'coder parsed', parsed)
print(json.dumps(parsed))
"

If validation fails, re-dispatch once with the pydantic error as feedback. Give up after one retry.

Fork-from decision (§15.2)

After the Ideator responds, read its fork_from field:

fork_from value	Action
"baseline"	Call create_round_worktree(repo_root, run_id, N, slug, fork_from="main").
"exp/<run_id>/<M>-<slug>" (string)	Call create_round_worktree(..., fork_from=<that branch name>).
list of ≥2 branch names	Compose round: call create_compose_worktree(repo_root, run_id, N, slug, parents=<list>). If it returns status="compose_conflict": write a history.jsonl row via record_round with status=compose_conflict, round_attempt_id=<UUID>, conflicting_files=<list>, and skip to the next round (no Runner invocation).

Mint round_attempt_id as str(uuid.uuid4()) before calling the Ideator prompt is assembled, and pass it through every subsequent call (Coder prompt, Runner subprocess, Analyst prompt, record_round).

Runner invocation (worktree path)

Use subprocess_runner.run_round_in_subprocess(cfg, env, round_attempt_id=<uuid>) with:

• cfg.code_cwd = <worktree_dir> (from the worktree plan dict)
• cfg.branch = <branch> (from the plan)
• cfg.fork_from = <Ideator's fork_from value>
• cfg.compose_mode = <Ideator's compose_mode> (only for compose rounds)

On success (metrics.status == "ok"), the subprocess has already committed the pointer file in the worktree. Proceed to Analyst dispatch.

On status=compose_conflict or any non-ok status, call cleanup_round_worktree (removes the checkout; branch persists unless it was a compose_conflict, in which case create_compose_worktree already deleted the branch).

6. Write round_<N>/config.yaml and invoke the Runner

python -c "
import json, sys, yaml
from pathlib import Path
from autoqec.orchestration.trace import append_note, append_section
parsed = json.loads(sys.stdin.read())
out = Path('runs/<run_id>/round_<N>')
out.mkdir(parents=True, exist_ok=True)
cfg_path = out / 'config.yaml'
with cfg_path.open('w', encoding='utf-8') as f:
    yaml.safe_dump(parsed['dsl_config'], f, sort_keys=False)
append_section('runs/<run_id>', <N>, 'config.yaml', cfg_path.read_text(encoding='utf-8'), fence='yaml')
append_note('runs/<run_id>', <N>, 'invoking Runner (profile=<profile>)')
"

python -m cli.autoqec run-round <env_yaml> runs/<run_id>/round_<N>/config.yaml runs/<run_id>/round_<N> --profile <profile>

The CLI prints the RoundMetrics JSON on stdout. Save it — you also need it for the Analyst. Trace the metrics dict before moving on:

python -c "
import json, sys
from autoqec.orchestration.trace import append_section
metrics = json.loads(sys.stdin.read())
append_section('runs/<run_id>', <N>, 'runner metrics', metrics)
"

If metrics.status != "ok", skip the Analyst and the verifier for this round. Do not call build_analyst_prompt, do not dispatch the autoqec-analyst subagent, and do not run independent_verify. Call record_round() at step 10 with only round_metrics=<metrics> (omit verify_verdict and verify_report); the recorder synthesises a fallback summary from metrics.status / status_reason.

7. Build the Analyst prompt (only if metrics.status == "ok")

python -c "
from autoqec.orchestration.memory import RunMemory
from autoqec.orchestration.loop import build_analyst_prompt
mem = RunMemory('runs/<run_id>')
print(build_analyst_prompt(mem=mem, round_dir='runs/<run_id>/round_<N>', prev_summary='<prev or empty>'))
"

8. Dispatch the Analyst

Agent tool with subagent_type="autoqec-analyst". Required response keys (validated): summary_1line, verdict ("candidate" or "ignore"), next_hypothesis_seed.

After receiving the raw response, validate and trace:

python -c "
import json, sys
from autoqec.agents.dispatch import parse_response
from autoqec.orchestration.trace import append_note, append_section
raw = sys.stdin.read()
append_section('runs/<run_id>', <N>, 'analyst prompt', <ANALYST_PROMPT>)
append_note('runs/<run_id>', <N>, 'dispatched autoqec-analyst')
append_section('runs/<run_id>', <N>, 'analyst raw', raw)
parsed = parse_response('analyst', raw)
append_section('runs/<run_id>', <N>, 'analyst parsed', parsed)
print(json.dumps(parsed))
"

9. Run the independent verifier (automatic)

If metrics.status == "ok" AND the Analyst verdict is candidate:

from pathlib import Path
from autoqec.envs.schema import load_env_yaml
from autoqec.eval.independent_eval import independent_verify
from autoqec.orchestration.trace import append_note, append_section

append_note("runs/<run_id>", <N>, "running independent verifier")
env = load_env_yaml("<ENV_YAML_PATH>")
sp = env.noise.seed_policy
holdout = list(range(sp.holdout[0], sp.holdout[1] + 1))[:50]
report = independent_verify(
    checkpoint=Path("<ROUND_DIR>/checkpoint.pt"),
    env_spec=env,
    holdout_seeds=holdout,
)
Path("<ROUND_DIR>/verification_report.json").write_text(
    report.model_dump_json(indent=2), encoding="utf-8",
)
verify_verdict = report.verdict       # pass into record_round
verify_report = report.model_dump()   # ditto
append_section("runs/<run_id>", <N>, "verifier report", verify_report)

Skip cases: metrics.status != "ok", compose_conflict, Analyst verdict == ignore.

On verifier crash: catch, set verify_verdict="FAILED", verify_report=None, include the exception in the Analyst summary.

10. Record the round

python -c "
import json, sys
from autoqec.orchestration.memory import RunMemory
from autoqec.orchestration.round_recorder import record_round
from autoqec.orchestration.trace import append_note
payload = json.loads(sys.stdin.read())
mem = RunMemory('runs/<run_id>')
row = record_round(
    mem=mem,
    round_metrics=payload['round_metrics'],
    verify_verdict=payload.get('verify_verdict'),  # 'VERIFIED' | 'FAILED' | None
    verify_report=payload.get('verify_report'),    # dict or None
)
metrics = payload['round_metrics']
append_note(
    'runs/<run_id>', metrics.get('round'),
    f\"round {metrics.get('round')} recorded (status={metrics.get('status')}, \"
    f\"verify={payload.get('verify_verdict') or 'skipped'})\",
)
print(json.dumps(row['summary_1line']))
"

Pipe one JSON object on stdin with keys: round_metrics (the full RoundMetrics dict including branch, commit_sha, fork_from, round_attempt_id, status, status_reason, etc.), optional verify_verdict ("VERIFIED" / "FAILED" / None), and optional verify_report (the VerifyReport dict from Step 9, or None). The recorder synthesises a fallback summary_1line on the runner-failure path.

This writes the round to history.jsonl, appends to log.md, and refreshes pareto.json (full non-dominated archive) alongside pareto_preview.json (top-5 projection for humans).

11. Loop or stop

If the user asked for more than one round, go back to step 2 with round_idx+=1. After the last round, close the trace and print a short summary:

python -c "
from autoqec.orchestration.trace import append_note
append_note('runs/<run_id>', None, 'run complete — <N_TOTAL> round(s)')
"

Then print to chat:

Run complete. Path: runs/<run_id>/
  log.md                     one-liner per round
  history.jsonl              <N> rounds
  pareto.json                non-dominated archive (VERIFIED)
  orchestrator_trace.md      full subagent + tool narrative

Failure handling

• Malformed subagent response — parse_response raises with the pydantic error in the message. Retry once with that error appended to the prompt; give up after a second failure.
• Runner compile_error — call record_round(mem, round_metrics=<metrics>) without verify_verdict; surface the status_reason in the chat; continue.
• Runner killed_by_safety — same as compile_error; the recorder's fallback summary shows which safety fired.
• 3 consecutive ignored rounds — halt and ask the user to inspect; likely a DSL schema drift or Runner regression.

Compose conflict handling (§15.6.3)

If a compose round returns status="compose_conflict":

• record_round writes the compose_conflict row BEFORE any cleanup
• The synthetic branch is already deleted by create_compose_worktree
• Ideator on the next round will see the FAILED_compose node in fork_graph and must not re-propose the same fork_from_canonical set
• Do NOT run the Step 9 verifier for compose_conflict rounds — there's no checkpoint

Tool-use rules

• The orchestrator (you) may call Agent, Bash, Read, Write, Edit.
• autoqec-ideator subagent: Read, Grep, Glob.
• autoqec-coder subagent: Read, Write, Edit, Grep, Glob.
• autoqec-analyst subagent: Read, Grep.

Costs + caveats

• Torch must be installed in the Python env used for cli.autoqec run-round. The rest of the loop is torch-free.
• Dev profile rounds finish in 2–5 minutes on CPU (dominated by the per-sample MWPM rebuild during eval); prod rounds take 10–30 min and benefit from GPU for the training pass.
• Verification runs automatically in this skill for successful rounds with Analyst verdict candidate (Step 9). /verify-decoder remains available for post-hoc re-audits with different holdout seeds.