usb-kernel-recover

name	usb-kernel-recover
description	Use when a USB device or fixture on a HIL rig's Linux host (ci.lan, hifiphile/tusb, a bench PC) is wedged, not enumerating, or when processes touching USB (testusb, JLinkExe, uhubctl, openocd, libusb tools) hang in D state. Linux-host side only — a bus owned by a TinyUSB host is out of reach.

USB Recovery on the HIL Rig (Linux kernel side)

The rule: a wedged usbfs ioctl holds that device's device_lock (usbdev_do_ioctl takes usb_lock_device, the uninterruptible variant — v6.12.96 devio.c:2609) and the driver under it waits in a plain wait_for_completion() with no timeout (usbtest.c:1404; usb_sg_wait, message.c:765). Nothing that also takes that lock can help. Only two levers don't: failing the URB at the device (rung 1) and the port-side data-line drop (rung 2).

1. Triage: find the holder

ps -eo pid,stat,etimes,wchan:22,args | awk '$2 ~ /D/'
sudo cat /proc/<pid>/stack          # never opens the node, so it cannot block

• S = victim. Lock-taking sysfs reads use usb_lock_device_interruptible (sysfs.c:124-139, 11 sites), so readers are killable and timeout bounds them. Ignore them; they unwind by themselves.
• D = the holder, or a writer that took the uninterruptible path.

Stack shows	Meaning	Go to
usbdev_ioctl + a driver module ([usbtest])	owner, holds the lock	rung 3 — terminal
usbdev_ioctl, no driver frames	owner waiting on a URB	rung 1 (DUT) / rung 2 (probe)
usbdev_open, sysfs reads	victim	ignore
tee .../usbtest/new_id, bind, unbind	victim that SPREADS it	stop issuing them
hub_event in a kworker	teardown stuck behind an owner	rung 3

Driver-bind writes are not passive: __device_driver_lock (drivers/base/dd.c) takes device_lock() uninterruptibly and device_lock(parent), because usb_bus_type sets .need_parent_lock = true (driver.c:2048) — each one holds the HUB's lock, which is how one wedged port takes a whole bus down.

Map the holder to a busport with lock-free attrs only (devnum, idVendor, idProduct are usb_descriptor_attr*, plain sysfs_emit, sysfs.c:688-705):

for d in /sys/bus/usb/devices/<bus>-*/; do
  [ "$(cat $d/devnum)" = "<devnum>" ] && echo "$d $(cat $d/idVendor):$(cat $d/idProduct)"
done
grep -l <SERIAL> /sys/bus/usb/devices/*/serial    # only on a HEALTHY device

2. Shield first (prerequisite for anything using libusb)

A wedged device blocks every enumerator that reads its locking attributes — JLinkExe, uhubctl, openocd's HID fallback. chmod 000 makes the VFS reject the read before ->show() runs, so they skip it and keep enumerating:

for f in bNumInterfaces bmAttributes bMaxPower configuration bConfigurationValue \
         product manufacturer serial avoid_reset_quirk; do
  sudo chmod 000 /sys/bus/usb/devices/<busport>/$f
done

• Shield the leaf, its parent hub, and the root hub (usb<N>) — a stuck uhubctl locks the root hub too.
• Run the recovery tool as NON-root: root has CAP_DAC_OVERRIDE, ignores the 000, and blocks anyway.
• Only those nine. descriptors, busnum, devnum, speed, idVendor, idProduct are lock-free and libusb needs them; a blanket chmod breaks enumeration instead of fixing it.
• chmod never blocks (inode setattr, no show()), so it works on a fully wedged device.
• Not needed for openocd pinned with vid_pid — it matches the cached descriptor and skips a foreign device before libusb_open (cmsis_dap_usb_bulk.c:107, bulk backend; the HID fallback ignores the pin).
• Leaf shields vanish on re-enumeration; the root hub's must be restored: sudo chmod "$(stat -c %a /sys/bus/usb/devices/usb<healthy>/$f)" …/usb<N>/$f

3. The rungs — go straight to the one triage names

Rung 1 — wedged DUT: reset it through its own probe.

printf "r\ng\nq\n" > /tmp/rec.jlink
JLinkExe -device <DEV> -if SWD -speed 4000 -SelectEmuBySN <probe-sn> \
         -autoconnect 1 -nogui 1 -CommandFile /tmp/rec.jlink

Reset before park-flash: non-destructive (the firmware under test survives for autopsy), no flash wear, and no bad park image — a wfe/wfi park has bricked SWD on mimxrt1064_evk and max32666fthr through a power cycle. ResetTarget measures 128-129 ms; cleared 57 → 0, 26 → 0 and 5 → 0 D-state processes, single shot each. Mechanism: chip reset drops the pull-up → usb_hcd_flush_endpoint unlinks the URB -ESHUTDOWN (hcd.c:1783) → the completion fires → the ioctl returns → the lock releases.

Works on i.MX RT (USBCMD.RS = 0 detaches, RT1050 RM Rev 3 p.2453) and on DWC2 — measured 2026-08-16 on stm32f407disco: r; g gave usb 13-2.2: USB disconnect, device number 107, re-enumerating 325 ms later. (A bare halt does not: the core keeps running with the pull-up asserted.)

Park-flash (--recover-board/--recover-fw, what usbtest.py automates) is the fallback where the reset cannot reach the peripheral. Delivery must be convoy-safe: openocd pinned with vid_pid, or esptool (-p <ttyACM>). JLinkExe selects by serial, which needs libusb_open, so it needs the shield.

Rung 2 — wedged PROBE: root-cycle. A probe has no probe to reset it, so the port-side drop is the only lock-free lever left. It commands the ROOT hub and never touches the wedged device's lock.

sudo usb_recover.sh root-cycle <busport> [expected-serial]

Bounces every fixture under that root port (up to 25 here). Renesas ppps disables D+/D− only — VBUS stays up, so it is a forced re-enumeration, not a power cycle. Success is the sysfs inode changing, not uhubctl's exit code.

Rung 3 — terminal case: a driver ioctl that OWNS the lock. No software cure: the task is uninterruptible and SIGKILL is queued, not delivered. Reboot with sysrq, never reboot(2) — a graceful reboot runs device_shutdown(), which takes every device lock and stalls on the wedged one.

echo b | sudo tee /proc/sysrq-trigger     # after: sync; sudo umount -a

Rung 4 — hypervisor. ci.lan only, and never needed in eight recorded wedges: qm stop <vmid> && qm start <vmid> from the PVE host. A VM reboot is not reliable — hubs can latch across the PCIe reset.

3b. If the CONTROLLER is dead, not a device

Signature: xhci-pci-renesas <addr>: Timeout while waiting for setup device command, devices on that controller failing to enumerate, or its buses gone — as opposed to ONE device wedged. The rungs above cannot help; the controller itself needs re-initialising.

sudo usb_recover.sh pci-rebind <pciaddr>     # unbind + bind the whole xHCI
sudo usb_recover.sh pci-bind   <pciaddr>     # only if it ends up driverless

Measured on ci.lan 2026-08-17 02:34:41 after a hub-cycle failed to take: unbind deregistered buses 17 and 18, the re-bind registered new buses 1 and 2 one second later, and every fixture re-enumerated. It renumbers every bus that controller owns, so hold all affected boards' locks first (hil_lock.py hold --all) and re-derive busports afterwards.

Do NOT reach for it while a device-lock convoy is live — see Common mistakes.

4. If nothing is in D state

The device is dead or silent, not wedged. sudo usb_recover.sh authorized <busport> unconfigures and reconfigures it (usb_set_configuration(dev, -1) then re-choose, hub.c) — it fixes stale driver/interface state, does not replug: the usb_device survives, so most probes keep their sysfs node. If that does not take, the device is wedged rather than silent — go to rung 1 or 2. resolve <dev-node> maps /dev/ttyACM3 → busport.

It takes usb_lock_device uninterruptibly (hub.c usb_deauthorize_device), so it is safe only while nothing is in D state.

5. Before declaring the rig healthy

ps -eo stat,args | awk '$1 ~ /D/' | wc -l    # must be 0
timeout 15 lsusb                             # rc 0 and a sane device count
sudo uhubctl -l <bus> -p <port>              # "0000 off" = never came back
sudo uhubctl -l <bus> -p <port> -a on

Observed: 5 boards missing with a completely clean D-state list, because usb17-port2 sat at disable=1.

Common mistakes

• uhubctl -a cycle on a root port without -S. It writes sysfs disable, and disable_store takes usb_lock_device(hdev) uninterruptibly then calls usb_disconnect(child) inside it (port.c) — against a wedged child that blocks while holding the root hub's lock, poisoning the bus. usb_recover.sh passes -S.
• echo 1 > .../remove to make a wedged device "go away": remove_store is the one attribute in sysfs.c taking the uninterruptible usb_lock_device (sysfs.c:765). It joins the convoy instead of clearing it.
• authorized on anything wedged — same uninterruptible lock. Driver unbind/bind (/sys/bus/usb/drivers/usb/{unbind,bind}) does the same unconfigure/reconfigure via usb_generic_driver_disconnect (generic.c) but ALSO takes the parent hub's lock (need_parent_lock), so it is strictly worse; it was removed from usb_recover.sh for that reason.
• pci-rebind for a wedged DEVICE. It is the cure for a dead CONTROLLER (see below), not for a device-lock convoy: with a live D-state URB the re-bind can hang and leave the controller with no driver and every fixture offline (observed once). Recover that with pci-bind <addr>.
• Writing /sys/bus/pci/devices/<addr>/reset — no rig controller has FLR, so it becomes a bus reset behind a live driver: card halted, host power cycle.
• Resetting a victim's board. Two boards were reset innocently before anyone found the holder. Map by devnum, not by which board "should" be running.
• Assuming one controller. Observed: 26 D-state processes across three xHCI controllers, all cleared by one probe reset on one device.

Rig layout (ci.lan, bus numbers renumber every boot)

readlink -f /sys/bus/usb/devices/usb<N> → its PCI address. AMD 0000:02:00.0 has no port-power switching; Renesas 0000:01:00.0 (probe tree) and 0000:03:00.0/0000:05:00.0 (DUT hubs) have real per-port ppps.