infra

name

infra

description

Operate and debug the distributed teleop infrastructure (Mac+Quest operator node, GCP cloud relay, Orin+YAM robot node, and remote GPU model services). Use when: something is down or stale (video, joints, input, policy, tracker); choosing/provisioning hardware for inference; restarting/deploying any tier; remote-managing a GPU or the Orin over ssh; or when a new infra issue gets root-caused — append it to the Issue log here so it is never re-debugged from scratch.

infra — operating & debugging the teleop stack

Before debugging anything: check the small-errors skill — known papercuts with 30-second fixes (dead joystick, doze fallout, stale IPs). This skill is for issues that need actual investigation; confirmed quick fixes graduate to small-errors.

Endpoints, tokens, start commands, and the live architecture diagram live in docs/SESSION-HANDOFF.md (keep that current). This skill is the operating discipline + the issue log.

Operating rules (each one bought with hours)

1. Kill and start in SEPARATE ssh sessions. A compound ssh 'pkill -f X; ... X ...' kills its own remote shell (the cmdline matches the pattern) → exit 255, nothing ran. Also use pgrep -f "name[.]py" so the probe can't match itself.
2. Remote daemons: ssh host 'setsid nohup CMD > /tmp/x.log 2>&1 < /dev/null &' — without setsid + stdin redirect the child dies with the ssh session.
3. Singletons by port-bind. One owner per resource: CAN bus, each /dev/video*, camera_relay :8089, serve :5599. Two camera relays racing over devices garbles capture rates silently (see ISSUE-001). The port bind is the lock — if a start "succeeds" but the port was taken, suspect a zombie.
4. The Quest kills ALL sockets when it sleeps (headset off → video LISTEN + service conn die; on wake nothing auto-resumes except our retry loops). Re-enter IP / re-LISTEN, or disable Auto-Sleep / tape the proximity sensor for work sessions.
5. i2rt swallows SIGINT. To stop a serve remotely: kill -9 it, then run YAM_control/turn_off.py (fresh bus connection) for guaranteed torque-off. turn_off timing out = arm unpowered (fine, report it).
6. Measure freshness, not fps. camera_relay re-sends its latest JPEG at 30 Hz whether or not it's new — "30 fps received" proves nothing about staleness. The relay burns HH:MM:SS.d #frame into every frame; read the clock in the image vs a wall clock. Any staleness question is answered by a screenshot, at any point in the pipeline.
7. Python envs on the Orin: serve/turn_off → ~/i2rt/.venv/bin/python (has dm_env/i2rt, NO cv2); camera/agent → ~/miniforge3/envs/xr/bin/python (has cv2).

GPU placement and launch gate

GPU-required services must never silently fall back to CPU. A running process or a 200 /health response only proves liveness; it does not prove that the service is usable.

Before launching or restarting a model service:

1. Inventory the target GPU with nvidia-smi, including every process and its VRAM.
2. Estimate the new service's observed peak, then retain at least 20% total VRAM or 4 GB, whichever is larger. If it will not fit, place it on a separate suitable GPU. A request to launch a model means to arrange accelerator capacity, not to accept CPU execution. If provisioning cost is not already authorized, state the exact GPU required and ask.
3. In the service's exact environment, require torch.cuda.is_available() == True and run one real model request. Treat device=cpu, cudaErrorUnknown, OOM, or a timed-out real request as DOWN. Stop the bad fallback instead of leaving it advertised as healthy.
4. Verify the complete path through its tunnel and consumer (for example, click/API-equivalent Rerun SAM3), not only the server-local endpoint. Only then call the service up.
5. Record the actual host/GPU, model, device, and real-request result in the handoff.

Current SO101 eval placement baseline (observed peaks, not marketing estimates):

Service	Observed VRAM	Placement
MolmoAct2 policy + action expert	~12.7 GB	Existing eval 4090
SAM3 prompt server	~4.1 GB resident; ~5.6 GB peak reserved	Existing eval 4090
SAM3 video tracker	~4.0 GB process; ~3.5 GB peak reserved	Existing eval 4090

The operator-approved canonical eval topology is all three services on the existing 24 GB 4090; the A100 is training-only. This is a deliberate headroom exception: after a real prompt and live tracker inference on 2026-07-16 the GPU used 20.8/24.6 GB and had 3.27 GB free. Do not start any additional GPU workload on that 4090. If the service OOMs or a CUDA context becomes poisoned while old GPU processes still work, restart the affected instance; accepting a CPU fallback or moving SAM3 to the training A100 is not a recovery. Do not provision another GPU for SAM3 unless the operator explicitly changes this placement decision.

Playbook: "video is stale / black / old"

The pipeline has exactly four buffer points; bisect with the burned-in clock:

camera → camera_relay → cloud relay ──► fleet-UI browser <img>   ← checkpoint A
                              └──► ① cv2 reader (eval grabber) → ② grabber tiles
                                   → encoder → ③ sender→Quest TCP → ④ Quest decode queue

• A: open the fleet UI cameras. Live clock in browser = everything through the cloud is good; the problem is Mac-side or headset-side. Stale in browser = robot-side (check for duplicate camera_relay processes, dead capture).
• ① reader: must be one drain-thread per camera (fixed in ISSUE-001; never read two network streams sequentially in one loop — TCP queues, staleness grows without bound).
• ② tiles: dead streams must show "NO SIGNAL", never a frozen last frame.
• ③ sender: must have a small SO_SNDBUF + drop-on-backpressure (stereo_sender AND camera_relay have it since ISSUE-004; legacy mono path does NOT — sleep builds minutes of backlog).
• Probe trap: a FRESH connection can show 0.1 s freshness while every LONG-LIVED stream is seconds behind (new conns start at the relay's current frame; old conns carry their queue). Test sustained: hold one connection ≥15 s, drain continuously, clock the LAST frame.
• ④ headset: fresh LISTEN/panel-open = fresh queue. Clock in headset vs wall clock is the final end-to-end measurement.

Issue log (append; newest on top)

ISSUE-017 · 2026-07-18 · Terminal compact export was skipped when full checkpoints were disabled

Symptom: the crop-first-0.2s continuation repeatedly trained from compact step 1750 through optimizer step 2000, completed all terminal validation, and exited with code zero, but live_exports/latest.json stayed at step 1750. The sidecar retried the same 250 steps three times, then correctly tripped .sidecar_failed_no_progress; the outer ablation queue stopped before No-base and 50%-base. Separately, the No-base short recipe filled disk while writing a 25 GB step500-tmp even though its recipe claimed no full optimizer checkpoint. Root cause: the compact exporter only ran before fetching the next batch. Pinned MolmoAct2 breaks its loop immediately at stop_at, so no fetch occurs after the terminal optimizer step. Earlier recovery logic could extract from a final DCP checkpoint, but the crop continuation intentionally set full-checkpoint retention to zero. The seven short recipe YAMLs also contradicted their documented storage policy by setting full-checkpoint retention to one, forcing an unnecessary terminal DCP save. Fix: wrap Trainer.fit() as a second, required terminal export seam while the trained FSDP action expert and trainer state are still live. Keep ordinary pre-batch exports nonfatal, but propagate terminal export failures. Set all seven 500-step comparison recipes to retain zero full checkpoints; their compact step-250/500 snapshots remain SHA-verified and HF-archived. Regression tests simulate the exact no-next-batch terminal boundary and verify terminal failures are required while intermediate failures remain nonfatal. Reference: docs/refs/molmoact2/live-action-expert-export.md and memory/2026-07-18-molmoact2-terminal-export-recovery.md.

ISSUE-016 · 2026-07-18 · UI reload stranded an active eval series behind the session timer

Symptom: the checkpoint cards and durable episode counts survived a UI reload, but Resume Eval was disabled or returned HTTP 500. The controller reported an active, incomplete series while the process-local eval state either had no previous config or had resume_remaining_s: 0.0 after the one-hour session window expired. Root cause: checkpoint-series progress was durable, but eval configuration and monotonic timer state lived only in the UI process. resume_eval() required both the old in-memory config and positive time remaining, even when the durable series was still active. Fix: an active incomplete series is now a restart-safe resume authority. After a UI reload, Resume reconstructs the canonical task/config, restores aggregate success/failure counts from the series, keeps the exact series/checkpoint identity, and starts a fresh one-hour execution window when the prior window expired. It does not clear or create a new checkpoint series. Regression coverage exercises recovery from an empty process-local eval state. Verified live by resuming molmoact2-crop02s-current-vs-original-2k at 185/1000 on the current step-1500 checkpoint.

ISSUE-015 · 2026-07-18 · Laptop restart removed eval tunnels; sandboxed recovery could not exec SSH

Symptom: the Mac camera/UI and all three local eval ports were down after a laptop restart even though the 4090 policy, SAM3 prompt, and SAM3 tracker remained healthy under Supervisor. An attempted recovery appeared to create the three detached screen sessions, but they exited immediately and local ports :8202/:8213/:8214 stayed closed; the screen logs included Cannot exec 'ssh': Operation not permitted. Root cause: the laptop-local camera relay, UI, W&B tracker, and SSH tunnels intentionally live in screen, so they do not survive a laptop restart. The first recovery was launched inside the managed filesystem/network sandbox; screen itself could start, but its detached child was not permitted to exec the persistent SSH tunnel process. Fix: run scripts/start_so101_eval_stack.sh outside the managed sandbox. The canonical launcher recreates three isolated SSH tunnels, starts exactly one camera relay and UI, rejects built-in Mac cameras, and gates success on camera/policy/prompt/tracker/UI health plus a real SAM3 detect request. Verified the UI, follower/leader connections, semantic front/side/wrist frames, crop-0.2s step-500 policy identity/SHA on CUDA graphs, accepted ball prompt, and a seeded CUDA video tracker. Do not diagnose remote GPU services as dead until checking Supervisor directly; in this failure they had survived for hours.

ISSUE-014 · 2026-07-18 · Missing USB camera silently shifted SO101 AVFoundation indices

Symptom: the eval UI's wrist card either failed or showed the wrong view after a laptop restart, even though indices 0, 1, and 2 could each open briefly. Root cause: the launcher treated AVFoundation indices as stable identities. Only two of the three external EMEET cameras were enumerated; macOS filled index 0 with the built-in FaceTime camera and re-enumerated the remaining external views. Fresh frame comparison against a labeled July 4 episode identified current index 1 as side and current index 2 as wrist; the external front camera was absent from both SPCameraDataType and the USB registry. After reconnecting it, all three EMEET C950s reported the same model and the same SN0001 serial, and their indices changed again between one-shot open/close probes. Numeric indices are therefore process-lifetime routes, not persistent camera identities on this rig. Fix / runbook: stop every relay/UI camera owner, enumerate devices with AVFoundation, grab one frame from each external camera, and compare against labeled front, side, and wrist episode frames before launch. Since isolated probes can themselves reorder identical devices, the authoritative mapping comes from the singleton relay's stable endpoints after it opens all three together; restart only the UI when correcting semantic routes. The launcher now enumerates configured AVFoundation indices and fails closed if any resolves to FaceTime, Built-in, or a Continuity Camera. Do not persist a new mapping while an expected external camera is missing. A camera that opens once must also pass a sustained freshness test.

ISSUE-013 · 2026-07-17 · Repeat checkpoint activation followed the existing live symlink

Symptom: staging a new single-checkpoint eval downloaded and verified the requested weights, then failed during activation with IsADirectoryError; the atomic handoff tried to replace the old checkpoint directory rather than live/current. Root cause: the checkpoint preparer's CLI called Path.resolve() on --current-link. On the first launch that path does not exist, but on every subsequent launch it is a symlink, so resolving it converts the swap destination into the currently served checkpoint directory. Fix: preserve the absolute link pathname without resolving symlinks before the atomic replacement. A regression test starts with live/current pointing at an existing checkpoint and verifies that the CLI passes the link itself to the preparer. The weights remain SHA-checked before activation, and the currently running policy is left untouched if activation fails.

ISSUE-012 · 2026-07-17 · Validation-only MolmoAct2 repo was registered under its episode selector

Symptom: after three comparisons had completed, the No-live recipe failed before step 1 with Missing repo-to-tag mapping for LeRobot repo 'local/...live...'. Its three-strike sidecar correctly wrote .sidecar_failed_no_progress and stopped, but the outer queue's Supervisor policy repeatedly restarted the deterministic failure until the queue became FATAL. Root cause: our validation injector wrote LEROBOT_REPO_TO_TAG using the full repo@episodes spec. Pinned MolmoAct2 commit c2282820f9b188b60e66ea1636b3efd81c45cbb4 parses away the episode selector before looking up that map. Earlier recipes hid the bug because training had already registered the same physical repo; No-live was the first recipe where the live repo existed only in validation. Fix: validation now always registers the physical repo key, preserves a canonical training mapping when present, and reuses the first mapping across validation-only slices. Regressions cover both validation-only live (No-live) and validation-only base (No-base). The queue Supervisor treats exit 1 as a durable deterministic stop, while retaining restart behavior for genuinely unexpected exit codes. Local verification passed all 426 project tests. After deployment and removal of only the diagnosed failure marker, No-live crossed the former crash point and reached real optimizer step 70/500 with the A100 active. Reference: docs/refs/molmoact2/dataset-tag-constraint.md.

ISSUE-011 · 2026-07-17 · Fresh eval UI never created the SAM3 video tracker's first seed

Symptom: after restarting the local eval UI, the tracker frame counter advanced and the cup prompt succeeded, but the ball stayed pending indefinitely unless the operator clicked Rerun SAM3. Root cause: the external sam3_video branch intentionally bypasses synchronous image seeding. Its empty-mask path reused the periodic-grounding gate, while the canonical launcher sets SO101_SUCCESS_BALL_SAM3_EVERY_N_FRAMES=0 to disable periodic re-grounding. Zero therefore disabled both periodic correction and the mandatory boot seed. Even an async prompt result was not registered as a seed that the video tracker could consume. Fix: missing-mask grounding now has an independent bounded 15-frame retry cadence. The first accepted prompt mask is stored in process-local automatic slot 4, preserving manual slots 1-3, and is handed to the video tracker as a reset seed. Once tracking exists, periodic grounding remains disabled at cadence zero. A regression test boots the full async seed → video-tracker path with periodic grounding off.

ISSUE-010 · 2026-07-17 · Mask card froze while SAM3 tracking remained healthy

Symptom: the UI's MASKS card intermittently held one old frame even though SAM3 video tracking continued and successes were still detected. Root cause: the card used a long-lived /api/success.mjpg <img>. When that multipart browser connection stalled or ended after its initial 200 response, the browser retained the last decoded frame. Unlike the three camera cards, the mask card had no per-frame freshness watchdog or reconnect path. Live diagnosis showed the tracker frame counter advancing from 7 to 141, sam3:video masks arriving, and about 0.39 s capture-to-browser latency while the display appeared frozen; a direct four-second stream probe delivered 57 frames, isolating the problem to recovery from intermittent browser-stream stalls rather than SAM3 inference. Fix: serve the existing cached /api/success.jpg overlay through the same 100 ms snapshot poller and 1.5 s retry watchdog as the camera cards. This removes the permanent-stale failure mode and avoids one continuous overlay-render loop per browser connection. Keep the MJPEG endpoint for compatibility, but the eval UI no longer depends on it.

ISSUE-009 · 2026-07-17 · MolmoAct2 eval launched on the eager inference path

Symptom: live policy requests took about 1.25 s median server-side and about 1.4 s end to end, making each non-realtime action chunk visibly pause. Root cause: the 4090 Supervisor command explicitly passed --no-enable-cuda-graph; the local environment and generic launcher also defaulted MOLMOACT2_ENABLE_CUDA_GRAPH=0. SAM3 prompt/tracker contention added variance but did not explain the missing fast path. The pinned upstream SO101 rollout recipe enables inference CUDA graphs and documents an approximately 2x action-expert speedup at additional VRAM cost. Fix: the canonical Supervisor config, local environment, example, and both launcher paths now enable the graph. Stop eval before service restart, reload the durable series' exact step/SHA, perform a non-motion graph-capture request, then resume. Policy inference and hot reload remain serialized by model_lock. The first capture took 2.743 s; six steady requests measured 0.579 s median and 0.600 s max, versus about 1.25 s before. GPU memory remained safe at about 21.2/24.6 GB with all three Supervisor services healthy. policy_error attempts are now invalid infrastructure events and cannot penalize a checkpoint. Reference: docs/refs/molmoact2/inference-cuda-graph.md.

ISSUE-008 · 2026-07-17 · Fresh SAM3 prompt host restart-looped on an empty scratch directory

Symptom: a newly provisioned 4090 had the gated SAM3 cache and CUDA ready, but sam3_eval_4090_prompt immediately entered Supervisor BACKOFF with no frames found in /workspace/sam3-frames. Root cause: sam3_prompt_ui.py required a seed JPEG during process construction even though live eval exclusively uses /api/detect_image and supplies the image in each request. The requirement was accidental state inherited from its offline frame-browser mode. Fix: create the scratch directory if absent and allow an empty frame list/zero initial dimensions; /api/detect still rejects an invalid frame index while /api/detect_image works normally. The Supervisor command now also uses explicit --device cuda --require-cuda, so a fresh host exits instead of advertising a CPU fallback. Regression tests cover both empty-directory boot and CUDA-required failure. Verified after a supervised restart with CUDA health and a real tunneled /api/detect_image request.

ISSUE-007 · 2026-07-17 · MolmoAct2 source ablation crashed because one repo used two tags

Symptom: the 50%-CC comparison exited before step 1 with ValueError: Repo ... appears under multiple tags, then its sidecar repeatedly relaunched it. Root cause: clean and CC are weighted episode selectors from the same physical LeRobot repo, but the generated specs labeled them so101_blueball_val3_clean and so101_blueball_val3_cc. Pinned MolmoAct2 commit c2282820f9b188b60e66ea1636b3efd81c45cbb4 intentionally requires one repo→tag mapping; episode selectors are removed before that check. A tag is a schema/normalization identity, not an experiment-group label. Fix: both base-repo training selectors now use so101_blueball_val3_base; selectors, weights, and clean/CC group identity remain distinct in the resolved recipe/snapshot. Live data keeps so101_blueball_live18. Added a three-strike no-progress circuit breaker with a durable marker so deterministic startup errors become visible failures instead of infinite GPU-burning retries. Verified the corrected 50%-CC run reached real optimizer steps with the A100 at 100% utilization. Reference: docs/refs/molmoact2/dataset-tag-constraint.md.

ISSUE-006 · 2026-07-17 · Final live export missed, causing an already-complete trainer loop

Symptom: No-CC reached and saved step 500, but live_exports/latest.json remained at step 250. For more than four hours the sidecar restored step 500, the trainer exited, and the sidecar relaunched it. The experiment queue correctly refused to advance without the final HF archive receipt. Root cause: the in-process live exporter runs before fetching the next train batch. At the exact max_duration boundary there is no next batch fetch, so the final 250-step hook can be missed even though the normal DCP checkpoint is complete. The sidecar used only the compact export step as its completion signal and had no DCP→compact recovery path. Fix: a normal checkpoint at or beyond MAX_DURATION is now authoritative completion. The sidecar extracts and hashes the action expert from that checkpoint, atomically republishes latest.json, waits for a verified private-HF receipt, then removes optimizer state only after manifest/receipt/local-weight step and SHA identities match. It refuses to launch a trainer whose resumable step already meets the target. No-CC step 500 was recovered without retraining and archived at HF commit d0eeb99d0bc86e6baaf5a6af792391b90db7f6cf (weights SHA prefix 2b33363de901).

ISSUE-005 · 2026-07-16 · SAM3 looked healthy while prompt inference had fallen back to CPU

Symptom: /health on :8213 returned ok: true, but Rerun SAM3 produced no cup or ball masks. The UI reported two 15 s request timeouts and the video tracker reported no_sam3_seed. Immediate accelerator failure: the co-located 4090 SAM3 prompt process began returning cudaErrorUnknown; after restart, new Torch processes on that instance could not initialize CUDA even though the already-running MolmoAct2 and SAM3-video processes retained their old contexts. The provider/driver-level trigger was not exposed inside the container. Why the stack failed open: sam3_prompt_ui.py automatically chose CPU whenever torch.cuda.is_available() was false; /health always returned ok: true; the launch path accepted HTTP liveness and a slow real request instead of enforcing device=cuda; and the current 4090 SAM services were unmanaged nohup processes. Policy + prompt + tracker also consumed about 22.5/24.6 GB, leaving unsafe headroom. Recovery: stop the broken 4090 prompt/tracker, run both SAM3 services under Supervisor on the already-running idle A100, keep MolmoAct2 step 2,000 on the 4090, point isolated local tunnels :8213/:8214 to the A100, and require a real /api/detect_image plus the UI's Rerun SAM3. Verified accepted cup and ball masks and seed_ready. Permanent rule: apply the GPU placement and launch gate above. The canonical launcher must keep policy/prompt/tracker on the eval 4090, keep the A100 training-only, fail closed on CUDA readiness, use Supervisor ownership, and verify tunnel ownership plus a real prompt.

2026-07-16 placement follow-up: Per operator decision, SAM3 prompt and tracker belong on the existing eval 4090, not the A100 and not a newly provisioned GPU. Installed dedicated Supervisor programs sam3_eval_4090_prompt and sam3_eval_4090_tracker, pointed them at the 4090's verified offline gated-model cache, and moved only local tunnels :8213/:8214 to that host. A UI-equivalent Rerun SAM3 accepted the cup (score 0.83, area 5281) and ball (score 0.94, area 884), and live SAM3 video tracking completed frames on CUDA. MolmoAct2 step 1,000 remained healthy on :8202. The A100 SAM3 programs were verified stopped. The remaining structural work is to make the canonical launcher install/start these Supervisor programs and fail closed on a real prompt request automatically.

ISSUE-004 · 2026-06-11 · Cameras seconds behind on EVERY long-lived stream (headset + viewer)

Symptom: camera video "incredibly behind" (multi-second, drifting) in both the headset and the fleet-UI browser — yet a fresh probe connection measured 0.1 s freshness. That contradiction IS the diagnosis: new connections start at the current frame; long-lived ones carry their backlog (see "Probe trap" in the playbook above). Root cause (two, both in camera_relay.py's per-client send loop):

1. Blocking wfile.write with a default-size TCP send buffer — any throughput dip (Wi-Fi hiccup, cloud-relay congestion) queued frames that were all still delivered in order; once behind, a stream stayed behind forever. Same disease as ISSUE-001 cause 2, one hop earlier in the pipeline.
2. The loop re-sent the LATEST jpeg at 30 Hz even when no new frame existed — duplicate frames doubled bandwidth on the constrained path, making the queueing more likely (and it's the known received-fps ≠ freshness trap from ISSUE-001). Fix (deployed to the Orin): only-new-frame sends (identity check on the jpeg buffer) + SO_SNDBUF 128 KB + select() writability check that DROPS the frame when the client stalls. Slow consumers now get FEWER frames, never OLDER frames (choppy beats laggy for teleop). Verified: last frame of a 15 s sustained stream was 0.1 s old. Also: when operator and robot share a LAN (home), point the eval DIRECTLY at the Orin (--cameras http://192.168.0.185:8089/{0,2} --serve-host 192.168.0.185 --serve-port 5599) — routing through GCP from ten feet away adds internet RTT and an unneeded choke point. Relay endpoints (127.0.0.1:18089/:15599) are for remote operation.

ISSUE-003 · 2026-06-11 · GREEN screen in stereo camera view after changing Wi-Fi

Symptom: at a cafe (network hop home→Starbucks→venue), the headset's ZEDMINI camera view showed a solid GREEN screen on Listen, even with the correct new IP typed. Green = the app's video texture allocated but its decoder never received a single frame. Root cause: when a device changes networks, its established TCP sessions die WITHOUT a FIN — the peer keeps them ESTABLISHED forever. StereoVisionServer._serve handled one control client at a time with a blocking read, so it sat blocked on the morning's half-dead session; the Quest's NEW connection completed its handshake in the kernel backlog (netstat: ESTABLISHED with the 191-byte OPEN_CAMERA sitting in Recv-Q, never read) and was never accepted. Each further Listen press queued another zombie. Diagnostic that cracked it: netstat -an | grep 13579 showing TWO established sessions — the old network's pair still present + a new pair with bytes stuck in Recv-Q. If Recv-Q is nonzero on a port we serve, the app behind it is not reading — find what it's blocked on. Fix: newest-connection-wins preemption in stereo_sender.py — every accept closes the previous control conn (unblocking its reader thread) and takes over; only the CURRENT session may stop the video stream. Regression test: fake_quest_stereo.py run with a parked zombie connection on :13579 (see scripts) — must still PASS. Lesson: any of-ours TCP server that serves "the one operator" must treat a new connection as the operator moving networks, not as an intruder: preempt, never queue.

ISSUE-002 · 2026-06-11 · Typing IPs into the headset every session

Symptom: every headset launch needed the Mac IP pecked into the Network panel on the in-VR keyboard (and once into the camera panel) — slow, error-prone, every session. Root cause: the Quest app ALREADY auto-discovers the PC service — it listens on UDP :29888 and pops a one-click IP-select dialog on a valid announce. Our PC service runs inside Docker on macOS, so its announce (a) broadcasts onto the container's bridge subnet, never the LAN, and (b) would advertise the container IP anyway. Doubly broken → silent fallback to manual typing. Compounding: the Network-panel IP field is never persisted by design (discovery is the intended UX); the camera-panel IP IS persisted (PlayerPrefs) after the first entry + clean app exit. Fix: scripts/xrtk_announce.py — replays the exact announce packet natively on the Mac every 5 s (subnet broadcast + --unicast <quest-ip> belt-and-braces, since Android may filter subnet broadcasts and the client binds 0.0.0.0). Start it with the Mac stack (runbook in SESSION-HANDOFF.md). Wire format + client parse rules: docs/refs/xrobotoolkit/discovery-announce.md. Lesson: when an upstream tool demands tedious manual config, read its source for the automation it already has — ours was broken by our own containerization, not missing.

ISSUE-001 · 2026-06-11 · Headset showed old/black video while browser was live

Symptom: headset video "not streaming", later recognized as old frames; fleet-UI browser cameras perfectly live; sender socket connected and streaming the whole time. Three stacked causes (each alone sufficient, which is why every single fix "didn't work"):

1. CameraGrabber read two network MJPEG streams sequentially in one thread → consumed slower than produced → TCP backpressure queued frames server-side → composite drifted 32 s stale after 25 s of runtime, unbounded. Fix: one drain-to-latest thread per camera + paced compositor.
2. Mac→Quest sender had unbounded TCP buffering; headset sleep froze the app but not its TCP stack → minutes of video queued → decoder replayed the backlog in order on wake ("black", then old frames). Fix: SO_SNDBUF ~128 KB + drop frames on backpressure (built into stereo_sender.py; mono path remains legacy-broken).
3. Two camera_relay processes briefly raced over /dev/video* (manual restart vs the fleet agent's arm_on auto-start) → erratic capture rates/counters. Fix: let the port bind enforce the singleton; check pgrep -af camera_relay before manual starts. Instrument that cracked it: burning capture wall-time + frame counter into every frame at the source — staleness became readable in any view, including by the operator in the headset. Keep it. Measurement trap found: received-fps ≠ freshness (relay re-sends latest at 30 Hz).