stress-test-router

name	stress-test-router
description	Stress-tests the router against real-world open-source KiCad boards (downloaded, normalized, stripped of routing), measuring routing completion rates and DRC violations per board. Aggregates results and files GitHub issues for new router/parser findings after user approval. Use to regression-test the router at scale or to hunt for robustness issues.

Stress-Test Router on Real-World Boards

Run the tests/stress/ harness end-to-end: prepare the board corpus, route every board following the plan-pcb-routing skill workflow, aggregate completion/DRC statistics, and turn novel findings into GitHub issues.

All corpus artifacts live OUTSIDE the repo in $STRESS_DIR (default ~/Documents/kicad_stress_test). Never commit boards to the repo.

Step 1: Prepare the corpus (skip parts that already exist)

Check $STRESS_DIR/boards_unrouted/*.kicad_pcb first — if the corpus exists and parses (run tests/stress/validate_boards.py), skip to Step 2.

cd tests/stress
python3 fetch_boards.py            # downloads from GitHub (needs `gh` auth)

KIPY=/Applications/KiCad/KiCad.app/Contents/Frameworks/Python.framework/Versions/Current/bin/python3
$KIPY normalize_boards.py          # pcbnew round-trip -> current format
for f in "$STRESS_DIR"/boards/*.kicad_pcb; do   # ONE board per process
  $KIPY strip_routing.py "$(basename "${f%.kicad_pcb}")"
done
python3 validate_boards.py         # all boards must parse with sane stats

Platform note: on Linux/Windows find the KiCad-bundled python equivalent, or any python with a working pcbnew module of KiCad 9+.

To extend the corpus, add (owner/repo, note) entries to REPOS in fetch_boards.py and a fragment->name mapping in normalize_boards.py. Only KiCad 6+ sources survive; older ones are rescued by the pcbnew round-trip.

Step 2: Run boards (queue manager)

Drive the whole corpus with the queue manager — it keeps headless claude -p board workers in flight until every board has a results JSON, deriving all state from disk (safe to stop and restart):

bash tests/stress/run_queue.sh [max_concurrency=8] [model=sonnet]
bash tests/stress/stress_status.sh        # monitor: DONE/RUNNING/TODO + free slots

run_queue.sh auto-launches tests/stress/queue_watchdog.sh, which caps per-board attempts (default 3, QUEUE_MAX_LAUNCH) and stubs a FAILED results JSON for any board whose worker keeps dying without writing one — otherwise the queue relaunches it forever and never terminates. The watchdog also runs standalone next to a manually-driven queue.

Each worker (run_board.sh <board> <set> [model]) routes one board per RUNBOOK.md (note RUNBOOK rule 1a': never cp/mv/alias board files in the run dir -- only tool --outputs may create them, or the redo manifest's replay chain is silently severed) and writes $STRESS_DIR/results[_set2]/<board>.json plus a FINDINGS.md. It also captures the agent transcript (transcript.jsonl) and auto-derives agent_narrative.md — a compact routing decision trail (the agent's narration paired with the actual route/diff/plane/fanout commands) via tests/stress/extract_narrative.py. The headless workers run claude -p --dangerously-skip-permissions, which the harness blocks by default — authorize it once with a Bash allow-rule for bash tests/stress/run_board.sh:* / bash tests/stress/run_queue.sh:* in .claude/settings.local.json (gitignored, so a checkout never inherits it), or approve when prompted.

Hard operational limits (baked into the scripts; violating these has crashed the machine before):

• Concurrency is LOAD-BASED; the arg is a hard ceiling (default 8). A fixed worker count cannot know whether the heavy python route steps happen to be coinciding — a board worker is mostly thinking (LLM latency), so the same N workers sit at load 3 or load 20 depending on what each is doing at that instant. A new board launches only while the 1-minute load average is under QUEUE_LOAD_MAX (default ncore-2, leaving two cores for other work), and at most one launch per minute (QUEUE_LAUNCH_INTERVAL). Four things that gate gets right, each a way to get it wrong:
  • Load, not swap or memory pressure. A busy Mac's normal state is pressure level 2 ("warn") with GBs of swap in use — macOS compressing idle claude processes, not thrashing. Gating on either wedges the queue forever. Only pressure level 4 (critical) blocks.
  • A floor (QUEUE_CONC_MIN, default 2) launches regardless of load, so other work on the machine can't stall the wave indefinitely.
  • Rate-limit, because load lags. The 1-min average is an EWMA: a launch doesn't show up in it for up to a minute. Measured at target 8: three consecutive admissions each read load 7.4–7.5 (all under the bar) and load then settled at 9.3 and hit 11.3. Poll faster than the interval or the limit doesn't govern the cadence.
  • Don't target half the machine. ncore/2 collapsed the tail to the floor: one heavy route step alone holds load near 4. The other guard is per-step memory, not the worker count — run_limited.sh kills any single step that exceeds 12 GB.
• Every tool command runs through tests/stress/run_limited.sh (12 GB RSS watchdog — raised from 4 GB in #422; a legitimate fine-grid run on a big sparse board can still peak several GB, so 4 GB killed real work). An OOM kill is a finding, not noise. Size a machine from 12 GB, not 4 — this is the figure people provision from, and 4 GB under-provisions by 3x.
• Give every routing command an explicit long timeout. The runbook tells the worker to block for up to the 3-hour per-command cap, but a driving harness's default is often ~120 s — two orders of magnitude smaller. In the sets-21-27 wave that killed at least one route attempt on 21 of 99 boards (steps that legitimately take 141-316 s), and the orphan guard then reaped the reparented child. Worse, the killed step's log is usually empty (stdout fully buffered, nothing flushed before SIGTERM), so it looks like a hang rather than a timeout — pass if you want a diagnosable partial log. See #599.

Manual fallback (no queue script)

To drive by hand instead, spawn one general-purpose subagent per board without a fresh results JSON, keeping ~4 in flight and refilling off stress_status.sh:

Read tests/stress/RUNBOOK.md (in the tools repo — the single source of truth) and execute it for BOARD= (). Follow the runbook exactly: analyze per the plan-pcb-routing skill, route with the repo's tools, verify, and write $STRESS_DIR/results[_set2]/.json. Never modify the tools repo.

A subagent must never end its turn while a routing process is still running (the run gets orphaned — RUNBOOK rule 12).

When driving by hand (the subagent path doesn't write transcript.jsonl), generate the narrative afterward from the sub-agent's transcript:

python3 tests/stress/extract_narrative.py \
  ~/.claude/projects/<project-slug>/subagents/agent-<id>.jsonl \
  -o "$STRESS_DIR/runs[_setN]/<board>/agent_narrative.md" --board "<board> (set N)"

Step 3: Aggregate

When all boards have results JSONs, build a summary table sorted by completion rate: board, layers, routable nets, completion %, multipoint pads connected/total, DRC baseline/final/delta, connectivity verdict, orphan stubs, wall time, issue count. Flag:

• completion < 100% — which nets, what failure mode
• DRC delta > 0 — violation types introduced by the router
• crashes / hangs / OOM kills — always report, with tracebacks
• per-board issues lists — deduplicate into distinct findings

Step 4: File GitHub issues (with approval)

For each distinct finding:

1. Search for an existing issue first (gh issue list --search "<keywords>" --state all), then act by state:
   • Open, same root cause → don't re-file; add a comment with the new evidence (affected boards + numbers + run date).
   • Closed and the bug RECURRED → reopen it (gh issue reopen <n>) and add an updated comment with the fresh evidence. A refound closed issue means the fix regressed or was insufficient — reopen, never file a duplicate.
   • Closed but only an adjacent new bug (the original fix still holds) → file a NEW issue (or comment on the closed one) and say so explicitly.
   • No match → draft a new issue (step 2).
2. Draft: title, affected boards, reproduction command (exact tool invocation against the corpus board), observed vs expected, relevant log excerpt, and severity (router-correctness > parser-robustness > route-quality > workflow-friction).
3. Present all drafts to the user and get approval BEFORE creating any issue. File only approved ones with gh issue create, and always apply a label (--label): bug for correctness/robustness/DRC defects, enhancement for new features or route-quality improvements (documentation / question when apt). When commenting on or reopening an existing issue, add a label too if it has none.

Known findings already on record (do not re-file — search/comment, or reopen if closed-and-refound, per the state rule above). Now FIXED (if refound, REOPEN with a repro/evidence comment — do NOT file a new one): power-type copper layers dropped (#76), Edge.Cuts regex cross-match (#77), KiCad 6/7 fp_text-reference collapse (#78), oval/slot drills read as SMD (#106), multipoint route_multipoint_main UnboundLocalError on free-end-less nets. Still OPEN (add evidence, don't duplicate): multipoint orphan dead-end stubs (#84), router success-vs-connectivity mismatch (#8), fine-pitch pads boxed in by sub-clearance copper / misleading "no rippable blockers found" (#95), no incremental output so a killed run loses work (#100), thermal-via exposed-pad falsely reported disconnected (#108), board-global fine-grid OOM on large 4+ layer boards (#109).

Reporting

End with: the summary table, the list of new issues filed (numbers/links), duplicates skipped, and any corpus-preparation problems. Keep per-board detail in the results JSONs, not the chat.