hil

# Hardware-in-the-Loop (HIL) Testing Run TinyUSB HIL tests on real boards. **Run `hostname` first** — it tells you which host you are on, which determines the default config and whether remote mode is possible. Rule of thumb: only `ci` and `tusb` are infra rigs; **any other hostname is a dev PC** and uses `local.json`. | Host | Local config | Remote (SSH → ci.lan)? | |-----------------------------------|--------------------------------------|--------------------------------------------------------| | `ci` (the rig) | `test/hil/tinyusb.json` (large pool) | no — boards are already local | | `tusb` (hifiphile's external rig) | `test/hil/hfp.json` | no outbound SSH to dev PCs/ci; SSH-reachable FROM both | | anything else (a dev PC) | `test/hil/local.json` | yes (large pool, `test/hil/tinyusb.json`) | Default to **local**. Use **remote** only when on a dev PC and the request or task scope names `remote`/`ci.lan`. Never attempt remote on `ci`. `tusb` (ssh alias `hifiphile`) is an external rig (hosted by maintainer hifiphile), exercised by the GitHub CI `hil-tinyusb (hfp.json)` matrix job — **never run HIL against it unless the user explicitly asks.** ## Board locks — the CI runner keeps running The `ci` rig also hosts a GitHub Actions runner that flashes boards and runs HIL as part of CI. Hardware access is arbitrated **per board** with kernel flocks in `/tmp/tinyusb-hil-locks/` — do NOT stop the runner service. - `hil_test.py` self-locks each board for its flash+test (holder reason `hil_test.py`). A locked board fails immediately (`<board> Failed: board locked: {holder info}`) without flashing — in CI, re-run the failed job later; if your `hold` is refused with reason `hil_test.py`, a CI job is mid-test — wait a few minutes and retry rather than forcing. - For usbtest bring-up or case diagnosis, load the `usbtest` skill before a manual case run; its `run_case.py` takes and releases the lock itself. This skill still owns board locks and rig selection. - For hardware work outside `hil_test.py` (JLink/GDB, manual flashing, `usbtest.py`, serial poking), hold the lock first and release it when done — release is mandatory cleanup; the auto-release on holder death is a backstop, not the plan: ```bash python3 test/hil/helper/hil_lock.py hold BOARD [BOARD...] --reason "why" # ... hardware work ... python3 test/hil/helper/hil_lock.py release BOARD [BOARD...] ``` - A hold refused by another holder: report holder and reason (`hil_lock.py status`), never kill the holder. A holder reason of `hil_test.py` is a CI job mid-test — wait a few minutes and retry once when the task allows, otherwise return the holder to the caller. - A manual session on a dev-bench board with no entry in this host's HIL config locks it by an agreed board name, with no config: `hil_lock.py hold BOARD --reason "..."` only reserves that name, so verify the probe serial and board identity yourself. A named-board lock needs no config; config-driven tools (`hil_test.py`, `hil_pool_check.py` without an explicit config, `hil_lock.py hold --all`) still need this host's config. - Never pre-hold boards you are about to run `hil_test.py` on — it self-locks and would treat your own hold as a conflict. - Rig-wide operations (uhubctl power cycling, `usb_recover.sh root-cycle`, pci-rebind, controller resets — bus renumbering) affect every board: `hil_lock.py hold --all --config <this host's config> --reason "..."` first — `--all` defaults to `tinyusb.json`, so on `tusb` it would reserve 27 boards that do not exist there and none of the three that do. Even a single root-port bounce needs `--all`: nothing maps a sysfs busport to a board name, and `hil_lock.py hold` accepts any string, so a "just the siblings" hold reserves nothing while reporting success. If `--all` cannot be taken, wait: a partial hold is worse than none, because it reads as protection. - `hil_lock.py status` lists holders. Locks auto-release when the holder process dies (kernel flock); `/tmp` clears on reboot. - A board whose usbtest battery reported the device wedged (a HUNG case the in-run recovery could not clear, or a device it could no longer identify afterwards) gets a `board-wedged` cell and the rest of its examples are skipped for that run. No admission marker blocks the next run. Before a board's first flash, a run without `--skip-flash` looks for a D-state `testusb` on that board's usbfs node, which a cancelled or killed run leaves behind, and resets the board through its convoy-safe recovery flasher (`flasher_recover`, else `flasher`); a board it cannot free is reported `board-wedged` without being flashed. A stray on a board outside the run is not swept. Interactively, follow `usb-kernel-recover` from its triage before running it again; a delegated run follows Reporting. - Forcing past a lock: `HIL_NO_BOARD_LOCK=1 python3 test/hil/hil_test.py ...` bypasses the guard without killing the holder. Only when the request or task scope explicitly names forcing that board — it risks colliding with whatever holds it; a refused hold alone never adds that scope. ## Pool check (board/probe health) Board/probe health scanning (`test/hil/helper/hil_pool_check.py`) has its own skill: **hil-pool-check**. Use it before a HIL campaign, after rig maintenance/reboot, or when boards fail to flash. ## PR-scoped selection `tools/change_impact.py` maps a diff to affected boards/tests (used by CI on PRs; fail-open to the full matrix). Manual use: ```bash SEL=$(python3 tools/change_impact.py --base master test/hil/tinyusb.json) FULL=$(printf '%s' "$SEL" | python3 -c "import json,sys; print(json.load(sys.stdin)['full'])") ARGS=$(printf '%s' "$SEL" | python3 -c "import json,sys; print(json.load(sys.stdin)['args']['tinyusb.json'])") if [ "$FULL" = "True" ] || [ -n "$ARGS" ]; then python3 test/hil/hil_test.py $ARGS test/hil/tinyusb.json # $ARGS empty when full: run everything else echo "diff affects nothing on this rig - skip HIL" fi ``` Read `full`, never `args` alone: `args` is empty for BOTH `full: true` (run the whole matrix — a broad or unclassified change) and "nothing selected" (skip). Skip only when `full` is false AND `args` is empty. Unit suites (no hardware) live in `test/hil/test/test_*.py`; the `hil-test` pre-commit hook runs every `test_hil*.py`, `change-impact-test` `test_change_impact.py`, `test_ci_metrics.py` and `test_hil_util.BottomLayer`. `test_change_impact.py` covers only selection, `test_ci_metrics.py` only the code-size plumbing; the bounded reads and the build and pool guards live in `test_hil_bounded.py` and `test_hil_util.py`; `test_hil_report.py` covers the report document, `test_hil_rtt.py` the RTT console and `test_hil_pool_check.py` the pool check's verdicts and output. Run them all when changing `test/hil`: `for f in test/hil/test/test_*.py; do python3 "$f"; done` (about a minute, half of it `test_hil_bounded.py`'s deliberate hang/timeout simulation). ## Stuck runs What bounds a stuck run is `HIL_POOL_TIMEOUT` plus the job's `timeout-minutes`: a worker the kernel will not let die (D state) holds the job until that ceiling. The `hil-pool-check` skill finds which boards a wedge left unusable. See the `usb-kernel-recover` skill for what a real wedge looks like and how to clear it, and the `usb-kernel-debug` skill to explain WHY the kernel rejected a device (dmesg analysis). ## Prerequisites Examples must be built for the target board(s) — see [Build and Validate](../../../CLAUDE.md#build-and-validate). Build them with the `build` skill's `--shared --variants <config>`, naming the HIL config the run uses: it writes the `cmake-build/cmake-build-<variant>/` folders `hil_test.py` flashes from by default, each with its variant's `flags` and `defines`, and refuses a folder still configured with options neither the variant nor the command line supplies. A local `hil_test.py --build` runs that same build for the selected boards first. A **remote** run stages the same folders; see Remote execution below. (This applies to `hil_test.py`; `hil_pool_check.py` never builds: it flashes CI artifacts from its own cache, see **hil-pool-check**.) A bare `--shared` takes nothing from the roster and builds only `cmake-build-<board>/`, so a self-named variant is built without its flags (`raspberry_pi_pico`'s `CFG_TUH_RPI_PIO_USB=1`) and any other variant stays unbuilt. An unbuilt variant's tests report `Skip (no binary)` without failing the run (`stm32f723disco-DMA` goes untested). On a run of one test per variant (a single-test `-bt`, or `-t` tests the board's `only` or capabilities leave at one) without `--skip-flash`, a later unbuilt variant instead fails `same-PID boundary ... not cleared (no board_test binary)`, naming `board_test`, not the missing build. A board whose flasher probe has no VCOM (or whose BSP has no UART) uses RTT as its console — "No serial device found for /dev/serial/by-id/…" on every host test is the symptom. Config: `"logger": "rtt"` (jlink flashers only) plus a self-named variant carrying the define — `"variant": [{"name": "<board>", "defines": ["LOGGER=rtt"]}]` — and prebuilt example sets must carry the same `-DLOGGER=rtt`. Caveat: the cdc/msc-fixture host tests don't speak RTT yet, so such a board cannot carry `is_cdc`/`is_msc` fixtures (the config loader rejects it). Details: the `rtt` skill. ## Arguments - **Board:** `-b BOARD_NAME`, repeatable for a subset (`-b a -b b`); omit to run all boards in the config. Give a whole set to ONE run rather than one run per board: it spreads the boards across host controllers in dispatch order, while a second `hil_test.py` running alongside multiplies the load on the same xHCI cards. (A marginal DUT port bouncing under concurrent batteries has killed a uPD720201: fix the port or pull the board.) - **Pass-through:** `-v`, `-r N`, etc. forwarded unchanged. If `local.json` is missing on a dev PC, ask the user to supply one before a `hil_test.py` or `hil_pool_check.py` run. An agent that cannot ask (`hil-operator`) does not run `hil_test.py`: it returns one `ran: false` row per requested board whose `detail` names the missing `test/hil/local.json`. Fall back to `tinyusb.json` only when the request or task scope says so; a manual session locks by board name as Board locks says. ## Local execution Set `CONFIG` from `hostname` first (`test/hil/local.json` on a dev PC, `test/hil/tinyusb.json` on ci, `test/hil/hfp.json` on tusb): ```bash CONFIG=test/hil/local.json # on ci use: CONFIG=test/hil/tinyusb.json # All boards in the config: python3 test/hil/hil_test.py "$CONFIG" # A single board (replace stm32f723disco): python3 test/hil/hil_test.py -b stm32f723disco "$CONFIG" ``` ## Remote execution (dev PC → ci.lan only) `scripts/hil_remote.py` takes `hil_test.py`'s own arguments, minus the config. It stages the harness, the config and the firmware the run will read under `-B` (default `cmake-build`, the `build` skill's `--shared` layout), runs `hil_test.py` on `ci.lan` with `tinyusb.json`, and copies the report pair and `<config>.failed` back to the checkout root: ```bash # All boards built under cmake-build/: python3 .claude/skills/hil/scripts/hil_remote.py # A subset — repeat -b, ONE invocation for the whole set: python3 .claude/skills/hil/scripts/hil_remote.py -b raspberry_pi_pico2 -b stm32f723disco -t host/cdc_msc_hid -r 1 ``` An agent launches it with `--run-id` and waits on it as Timing says. One invocation per board is wrong here, not merely slow: each run `rm -rf`s `REMOTE_DIR` and rewrites the report, so only the last board's rows survive. A second run sharing `REMOTE_DIR` is refused, before the wipe, while the first holds `<REMOTE_DIR>.lock`. The rig needs `flock`; setup fails without it. Before touching the rig it refuses a board not in the config, then applies `--flasher` and `--exclude-flasher` (`no board left after the flasher filter` when none survives), then refuses a requested board the filter kept with none of its `<-B>/cmake-build-<variant>` dirs, naming the dirs it looked for (a variant's build flags are in the config); a board the filter drops needs no build. Without `-b` it refuses with `nothing to test` when no board the filter kept is built, and stages what is built without a word about unbuilt variants. With `-b` it also warns for each variant of a requested board with no build; Prerequisites says what `hil_test.py` then reports for it. `--build` is refused: the rig receives binaries only. A delegated run is bound to its build: the build writes a build receipt of the HEAD it built, and the run passes it, refusing before it touches the rig when HEAD, the roster or a staged file no longer match it, or a tracked file (the harness it stages, the roster) has changed since HEAD (rebuild, which writes a new one). A retry of a subset of those boards reuses it; a local `hil_test.py` run takes none. ```bash python3 .claude/skills/build/scripts/check_build.py --board raspberry_pi_pico2 --board stm32f723disco --shared --variants test/hil/tinyusb.json --receipt .hil-remote/build-1790000000.json python3 .claude/skills/hil/scripts/hil_remote.py --run-id hil-1790000000 --receipt .hil-remote/build-1790000000.json -b raspberry_pi_pico2 -b stm32f723disco ``` The build refuses a receipt for a tree that is not clean before or after it (commit a `hw/bsp/family.json` it rewrote, then build again). `.hil-remote/` is ignored. A CI firmware artifact is the one delegated run without a receipt: download it straight into a fresh `-B` dir under `cmake-build/`, never a symlink to it (the staging rsync would copy the link). Run it from a checkout whose harness and roster match the artifact's head, since those are staged from the checkout. Report the artifact's name, run, head and staged sha256s beside the results. ```bash mkdir -p cmake-build && mkdir cmake-build/ci-36166669952 gh run download 36166669952 -n 'binaries-arm-gcc--b raspberry_pi_pico_w' -D cmake-build/ci-36166669952 python3 .claude/skills/hil/scripts/hil_remote.py --run-id hil-1790000000 -B cmake-build/ci-36166669952 -b raspberry_pi_pico_w ``` Exit 200 means the remote tree stopped being this run's after staging (another run sharing `REMOTE_DIR` replaced it): `hil_test.py` did not run and nothing was copied back, so any local `hil_report` pair or `<config>.failed` is from an earlier run. Re-run; never report from it. Env overrides: `REMOTE`, `REMOTE_DIR`, `CONFIG`, `ROOT_DIR`. ## Timing Runs take 2-5 min per board, but a stuck fleet runs to `HIL_POOL_TIMEOUT` — 60 min unless the env pins it. The run logs its guard in the startup line; never declare a run stuck before THAT value has elapsed. The Bash tool caps a foreground timeout at 10 min, so **run it in the background** -- never a foreground timeout, which would kill the run before its own guard can write a report. NEVER cancel early. A remote run is launched once, in the background, with a new `--run-id` (a used one is refused), then waited on in the foreground; write the id literally in both calls, since a shell variable does not survive from one tool call to the next: ```bash # Bash run_in_background: true; a delegated run adds its --receipt (Remote execution) python3 .claude/skills/hil/scripts/hil_remote.py --run-id hil-1790000000 --receipt .hil-remote/build-1790000000.json -b raspberry_pi_pico2 -b stm32f723disco # foreground, Bash timeout 600000 python3 .claude/skills/hil/scripts/hil_remote.py wait hil-1790000000 ``` `wait` blocks up to 9.5 min and prints one JSON line: - `done` (exit 0): the run's `exit` and the `reports` it copied back. Read only those; an empty list means no report is from this run, whatever the checkout holds. - `running` (exit 3): call `wait` again. - `dead` (exit 4): the run ended without its done record (killed, or its session died); no local report is from it. Never spend a tool call only to check on a run or to pass time: the next call is `wait`. A local `hil_test.py` run has no done record; wait for its completion notification. ## Reporting Two audiences, two shapes. Interactively, the answer to a HIL run IS the tool's summary table: paste the complete per-board table (and footer counts) verbatim — never truncate rows or reduce