anthias-hardware

# Anthias board / display / codec knowledge Durable engineering findings from field debugging across the board fleet. Each bullet carries the mechanism plus the fixing PR / CalVer release where known; OPEN items are called out. Boards map to a rendering stack by userspace arch, so read the display-stack section first. ## Display stack fundamentals (QPA: linuxfb / eglfs / wayland-cage) - **Board → QPA plugin mapping.** Pi 1/2/3-32 (armhf/Qt5) run `QT_QPA_PLATFORM=linuxfb` (needs `/dev/fb0`); Pi 3-64/Pi 4 (arm64/Qt6) run `eglfs` (DRM/GBM, holds DRM master, no fb0); Pi 5/x86/Rock Pi 4 (arm64/Qt6) run cage/wlroots wayland. Pi 4 switched linuxfb→eglfs in #2904 for HW video. Stale `start_viewer.sh` comments claiming "Pi 4 runs linuxfb" were corrected in #2962. - **DRM master ownership determines what can blank/rotate/recover.** On eglfs/linuxfb the Qt app (or drm_fb_helper) owns DRM master; on wayland the cage compositor owns the connector. This single fact drives blanking, hotplug recovery, and rotation mechanism per stack. - **Board selection is by userspace arch (`dpkg --print-architecture`), NOT `uname -m` or model string.** 32-bit Raspberry Pi OS ships a 64-bit kernel by default on Pi 3 (`arm_64bit=1`), so `uname -m`=`aarch64` while docker/apt are armhf. Gating on kernel arch mispicked `pi3-64` and pulled `linux/arm/v8` (no matching manifest). Fixed in PR #3076 (2026-06-16) to gate on userspace arch. `arm64` catch-all branch still uses `uname -m` (no armhf image for generic SBCs). No in-place arch switch — only reflash changes streams. ## Display blanking (per-platform power-off) - **Wayland boards can true-power-off; eglfs/linuxfb cannot.** `wlr-randr --output <NAME> --off` DPMS-powers-off the monitor on wayland (run as `viewer` user with `XDG_RUNTIME_DIR=/run/user/1000` + `WAYLAND_DISPLAY=wayland-0`). On eglfs/linuxfb the Qt app holds DRM master so external blank is rejected (`echo 3 > /sys/class/drm/cardN-HDMI-A-1/dpms` → EBUSY under atomic KMS; `vcgencmd display_power 0` no-op under vc4-KMS) — the only blank without stopping the viewer is to paint the screen black. PR #3065 added first-class `blank`/`unblank` viewer commands (Redis `anthias.viewer` channel) that do wlr-randr off on wayland, paint `BLACK_SCREEN` (`img/black.png`) on eglfs/linuxfb. ## Headless-boot / HDMI-hotplug recovery (differs sharply by stack) - **linuxfb (32-bit Pi 1/2/3): headless boot = no `/dev/fb0` = crash-loop.** Full KMS (`dtoverlay=vc4-kms-v3d`) only creates `/dev/fb0` when a display is connected at boot; headless → connector `disconnected` → no fb0 → linuxfb hard-requires it → `Unable to figure out framebuffer device` → `no screens available` → Qt heap-corrupts on the no-screen path (`malloc(): unaligned tcache`). Firmware `hdmi_force_hotplug`/`hdmi_*` knobs are a DEAD END (full KMS ignores them); only kernel cmdline `video=<connector>:...` forces a fb, unsafe for a fleet. Fix = graceful wait-and-retry in viewer (#2959 fb0-wait), self-heals when display hotplugs. `/dev/dri/card0`+renderD128 exist even headless; only fbdev emulation is absent. - **eglfs (Pi 3-64/Pi 4): does NOT need the linuxfb stale-`/dev` exit-1 guard.** Verified by clean power-cycle: `/dev/dri/cardN` is created at vc4 driver bind (kernel boot), present headless; `open()` routes by (major,minor)=226,1 to the live device even with a stale-timestamp node; connector status is read from sysfs (truthful). eglfs opens container card1 and modesets HDMI-A-1 after headless-boot+connect. But eglfs boards **refuse to start the viewer while headless** (`wait_for_eglfs_display` in start_viewer.sh from #2962 polls `/sys/class/drm/card*-*/status` for non-`disconnected`) — correct signage behavior but means a headless testbed renders nothing. - **eglfs headless gap was also a crash-loop before #2962.** Pi 4 eglfs headless → `no screens available` → binary exits before D-Bus handshake → container crash-loop. #2962 (`wait_for_eglfs_display`) mirrors the linuxfb fb0 wait. Separately #2969 added capped-backoff retry of the AnthiasViewer spawn (`BROWSER_SPAWN_MAX_ATTEMPTS=30`, `BROWSER_SPAWN_BACKOFF_CAP_SECONDS=15`). - **wayland-cage (Pi 5/x86): headless boot binds ZERO wl_outputs and never recovers on hotplug** (#3155, refactor #3161, shipped 2026-07-08). cage enumerates DRM connectors at startup; if none connected+enabled it binds 0 wl_outputs and runs forever (no exit → no restart → permanent black). It can't recover on hotplug because the viewer container has **no udevd / no `/run/udev`**, so wlroots' libudev hotplug monitor gets no uevent. Contrast: eglfs Qt exits non-zero on no-fb and `restart: always` self-heals. Symptom: viewer log spams `wlr-randr could not apply rotation N`; `grim` → `no wl_output`; scheduler runs normally. Fix = `_wayland_output_watchdog()` in `__init__.py` per asset_loop tick: `sys.exit(1)` if no bindable output past `WAYLAND_OUTPUT_GRACE_S=60` → container restart → cage re-enumerates. - **Gate on BINDABLE, not just `connected`.** A marginally-seated HDMI asserts HPD (`status`=`connected`) but EDID never read → `/sys/class/drm/<c>/modes` EMPTY → cage has no mode to bind → restart-loops forever. `_kernel_has_bindable_display()` requires `connected` AND non-empty `modes`. - **Second bug: restart recovery is racy.** The restart-policy relaunch is nearly instantaneous; if the old cage's DRM-master release + vc4 connector reset haven't drained, the fresh cage races a half-reset connector → EDID not re-read → empty modes → headless again (~50% wedge). `docker restart` recovers reliably (its stop→start gap lets vc4 reset). Fix in `start_viewer.sh` cage branch: on a `connected` connector, `sleep 4` then force `echo detect > status` + bounded-wait for `modes` to repopulate before launching cage. - **HDMI hotplug resolution drop to 1024×768 (issue #3052, fixed PR #3075).** `1024×768` is the kernel `drm_fb_helper` hard-coded fallback mode, triggered when a `vc4-kms-v3d` re-probe races the sink's EDID return (slow TVs). **linuxfb boards vulnerable** (no process holds DRM master; Qt reads `/dev/fb0` once, can't follow a later mode change). **eglfs boards immune** (Qt holds DRM master, keeps its modeset across hotplug). Recover in-container without fbset: `echo "U:1920x1080p-0" > /sys/class/graphics/fb0/mode` (the `-0` refresh field is CORRECT — vc4-kms-v3d fbdev emulation registers modes as `U:WxHp-0`). True active geometry = `fbset -i`; `virtual_size` is the static allocated buffer, `/modes` head is a sticky unreliable list. - **Container `/dev` is a boot-time snapshot.** If the privileged viewer container started headless and a display is plugged in later, the host gets `/dev/fb0` but the container does NOT (only `/sys/class/graphics/fb0` shows through) → `wait_for_framebuffer` parks forever. `docker restart` re-snapshots `/dev`. Only bites hotplug-after-boot. The celery container can NEVER do display recovery (not privileged, no `/dev/fb0` or `/dev/dri`, `/sys` read-only) — only the privileged viewer container. ## Video decode (HW vs SW) by chip - **Pi 4 (bcm2835, stateful v4l2m2m): ≤1080p H.264 HW-decodes; >1080p falls to SW.** `h264_v4l2m2m` (the `+rpt1` ffmpeg build) IS engaged by QtMultimedia for ≤1080p H.264 — `/dev/video10` fd held open whole clip, ~120-140% CPU, ~0 drops. 4K30 / High10 / 4:2:2 / level>4.x → not opened, cores pegged, ~36% drops. `QT_FFMPEG_DECODING_HW_DEVICE_TYPES` only accepts AVHWDeviceType names (cuda/drm/vaapi), NOT v4l2m2m (a standalone AVCodec) — Qt reaches it empirically anyway on `+rpt1`. Probe = `/dev/video10` open + CPU-delta. - **Pi 5 (Hantro G2, v4l2-request): HEVC HW-decodes through QtMultimedia libavcodec.** Verified via `/proc/<pid>/fd`: `/dev/video19` (rpi-hevc-dec), `/dev/dma_heap/linux,cma`, `/dev/media2`, `anon_inode:request`, `/dmabuf:` frames; 4K30/4K60 HEVC both play at 1.0×. The chain QtMultimedia→libavcodec61→v4l2-request→`/dev/video19` is load-bearing; don't strip device passthrough or the `+rpt1` pin. Pi 5 has **no H.264 HW decode block** (upload gate rejects H.264, supported set = `{hevc}`). - **Pi 3-64 (VideoCore IV, `+rpt1` gate): was SW-only until #3110 added `pi3-64` to the board tuple.** `docker/_rpt1-ffmpeg-pin.j2` gated the RPi `+rpt1` ffmpeg on `board in ('pi4-64','pi5','arm64')` — pi3-64 was MISSING, so it shipped stock Debian ffmpeg and SW-decoded (~0.7× realtime). PR #3110 (shipped, HW-validated) → `/dev/video10` held, 30fps realtime, `dropped=0`. H.264-only HW decode on VC IV (`_HW_DECODE_VIDEO_CODECS['pi3-64'] = {'h264'}`). - **Rock Pi 4 (RK3399, stateless v4l2_request): both H.264 and HEVC SW-decode in practice (OPEN).** No-hwaccel picks native SW — what QtMultimedia's libavcodec does. `-hwaccel drm` engages rkvdec for HEVC but produces `Decode fail` at 0.77× (broken on Armbian 6.18 kernel); H.264 has NO v4l2_request hwaccel binding in the `+rpt1` 7.1.3 build. `h264_v4l2m2m`/`hevc_v4l2m2m` = "Could not find a valid device" (stateful M2M wrappers don't bind the stateless RK3399 nodes). Real arm64 HW decode needs QtMultimedia to request `-hwaccel drm` against v4l2_request, and even then only HEVC works (with errors) on this kernel. - **x86 (Intel UHD 610, iHD/VAAPI): HEVC HW decode fails inside cage's GL context (OPEN, worked around).** `vainfo` lists HEVC VLD, container ffmpeg + an offscreen AnthiasViewer both VAAPI-decode HEVC cleanly, but the real display-up viewer emits `FormatError code=2` / 0 frames on HEVC RTSP (H.264 in the same context is fine). VAAPI-HW × GL-context × HEVC interaction; `QT_DISABLE_HW_TEXTURES_CONVERSION=1` did NOT fix it. Workaround: `QT_FFMPEG_DECODING_HW_DEVICE_TYPES=,` (empty → disable HW decode) → SW decode plays. Tradeoff: disables HW decode for ALL codecs on x86; a board-conditional list is better than blanket-off. ## Video presentation bottleneck (Qt6 boards — decode ≠ what reaches screen) - **Qt6 boards present video at ~8-12fps despite perfect HW decode** (issue #2967). Root cause: `QGraphicsVideoItem::paint → QVideoFrame::toImage → qImageFromVideoFrame` — Qt 6.8 converts YUV→RGB on the GPU then reads it back to a QImage, CPU-blits into raster backing store, re-uploads to GPU. Two GPU/CPU crossings per frame. Measured: Pi 4 1080p30 H.264 8.3fps presented; Pi 5 11.7fps. **PR #2975 fix** = QML `VideoOutput` in a QQuickWidget (`player->setVideoSink(item.videoSink)`) → Pi 4 30.0fps/66% CPU. A QOpenGLWidget viewport is NOT sufficient (toImage stays); QVideoWidget = second native window, fails the eglfs single-window constraint. - **#2975 fix does NOT hold at 1080p (OPEN, resolution-scaling).** RC QA (forced 1080p60 headless): 720p H.264 ~28fps, but **1080p H.264 ~5.8fps** (~40% pacer-dropped). Not a regression (Qt6 video byte-identical across releases). #3006 added a pacer that DROPS frames to match scene-render capacity. CONFOUND: measured on a no-EDID headless display; confirm on a real Pi 4 + monitor. Tracked in issue #2987. - **Measure PRESENTED fps via the kernel DRM vblank tracepoint, never `playback-stats.log`.** Count `drm_vblank_event_delivered` events over 10s (`/sys/kernel/debug/tracing`, `count/10 = presented fps`, `crtc=N` disambiguates outputs) — zero observer effect. `playback-stats.log` counts sink deliveries (full decode rate) and is blind to paints coalescing. `frames-rendered` (=`QQuickWindow::afterRendering`) is scene-render into an offscreen FBO, NOT scanout. - **Architecture note (misnomer):** Qt6 boards (pi4/pi5/x86/rockpi4) use `MPVMediaPlayer` — a misnomer, it drives IN-PROCESS QtMultimedia in the C++ webview over D-Bus, NO mpv binary anywhere post-#2904. pi1/2/3-32 use `GstFbdevMediaPlayer` (GStreamer HW → fbdevsink). pi3-64/pi4-64 routed via `force_mpv` env override in `media_player.py`. ## issue #2987 video regressions (2026.6.1, four distinct defects) - **1. pi3-32 portrait stretch** — #2972's `_video_sink()` forced fb-sized caps; fix = CAPS-event pad probe → aspect-fit caps with `pixel-aspect-ratio=1/1` (PR #3004, shipped 2026.06.3). - **2. pi3-32 freeze/cut clips** — bash `while true; gst-launch` loop rebuilt the pipeline per EOS. Fix: playbin `about-to-finish` gapless loop in `gst_fbdev_player.py` (PR #3004). Helper MUST be spawned by path, not `-m` (package `__init__` imports Django). - **3. Pi 4 (Qt6) 1080p60 collapse** — 22.6fps presented, clips killed by slot duration. Fix (PR #3006): intermediate QVideoSink gating on `afterRendering` + single-slot latest-frame mailbox. CEILING ~23fps for 60fps sources; full 60fps needs the bigger single-window rework. `dropped=` counter reads 0 in fast-forward mode — don't use it to detect this. - **4. Backup timeout** — sync tar gzip-9 = 98s for 355MB on Pi 4, browsers abort byte-less requests ~5min. Fix: StreamingHttpResponse over a pipe + compresslevel 1 (PR #3005). Pre-existing. ## Pi 3-64 video BLACK on VideoCore IV (issue #3084 / forum 6720/6730) — RESOLVED via DRM overlay plane - **Root cause: vc4 (VideoCore IV) GLES2 can composite RGB but CANNOT render QtMultimedia's `VideoOutput` scene-graph video node — for HW OR SW frames.** Video plays fine on Pi 4 (V3D 4.2) and Pi 5 (V3D 7.x) but is pure black on Pi 3-64, while still images + webview composite fine. Decode is healthy. Pinned by a red-backdrop diagnostic (`Rectangle{color:RED; VideoOutput{}}` → JUST RED). The #2967 substrate swap (`toImage` CPU-raster → `VideoOutput` RHI-texture) traded slow-but-visible for fast-but-black on VC IV. - **CPU-raster fallback works (unblacks) but caps ~1.5fps** — `QVideoFrame::toImage()` on a 1080p HW SAND/DMABUF frame = ~600-800ms/frame on VC IV. SW decode makes toImage cheap but SW-decoding 1080p30 REBOOTS the 1GB Pi 3. - **RESOLVED PR #3164 (merged 2026-07-09): in-process GStreamer HW decode → kmssink on a vc4 DRM OVERLAY plane, HW-composited alongside eglfs's primary plane, 30fps zero ongoing drops.** Qt eglfs exposes its DRM master fd (`libQt6EglFsKmsSupport.so` native-resource keys `dri_fd`, `dri_crtcid`, `dri_connectorid`, `dri_atomic_request`). Winning pipeline: `filesrc ! qtdemux ! h264parse ! v4l2h264dec capture-io-mode=mmap ! queue max-size-buffers=4 ! kmssink fd=<dri_fd> connector-id=<> plane-id=<overlay> force-modesetting=false can-scale=true skip-vsync=true qos=true`. - **Two keys:** (1) **decoder-direct, NO v4l2convert** — the bcm2835 ISP I420→NV12 convert caps ~15-18fps at 1080p; decoder alone does 30fps+. (2) **`capture-io-mode=mmap`** — default (exported dmabuf) lets kmssink zero-copy and PIN the decoder DPB buffers → pool starves → DEADLOCK (~frame 15). mmap forces kmssink to COPY each frame (~3MB, cheap) and release the decoder buffer immediately. - **Audio runs a fully independent GStreamer pipeline** → `alsasink device=sysdefault:CARD=vc4hdmi` DIRECT to HDMI (not pulsesink; pulse defaults to 3.5mm). Every shared-pipeline attempt froze video at frame 1. Added `gstreamer1.0-libav` for AC3/AAC/MP3. - In-process kmssink on eglfs's SHARED master fd is fragile: a crashed/SIGKILLed kmssink leaves the overlay plane bound → cascading DRM wedge until reboot — a supervised DRM-lease child would be more robust. - **OPEN follow-up: presented fps << decode fps under memory pressure.** On the ~787MB Pi 3-64 the merged build decodes 30fps but kmssink QoS-drops ~11fps as "late" → screen presents ~19-21fps under swap thrash. NOT a code regression. Next lever untried: `kmssink qos=false`. ## Rotation (`screen_rotation` convention + per-stack mechanism) - **`screen_rotation` is CLOCKWISE uniformly on every stack as of 2026.07.3 (PR #3202, 2026-07-16).** Before 2026.07.3 linuxfb turned CW but eglfs/wayland turned CCW for the same setting. Canonical = clockwise (matches linuxfb paint path + GStreamer videoflip). - **linuxfb (Pi 1/2/3-32):** QPA ignores `rotation=N`, so content is turned by hand — images/GIFs in `View::paintEvent` (`rotation.cpp` `linuxfbRotationOverride`), web pages via injected CSS (`webpageRotationScript`), video via `videoflip` (90→CW). - **eglfs (Pi 3-64/Pi 4, Rock Pi 4):** `QT_QPA_EGLFS_ROTATION` NEGATED — `{90:-90, 180:180, 270:90}`. A literal `270` rotates content but skips Qt's width/height swap (viewport stays 1920×1080 → stretched); emit `-90` instead (PR #2973, #2970). - **wayland (Pi 5/x86):** `_wlr_transform_value` — `{90:'270', 180:'180', 270:'90'}` (wlr transforms turn CCW). - **pi3-64 video:** vc4 overlay plane rotate-180 for 0/180 (HW, #3189); 90/270 drop the overlay → eglfs raster fallback inherits the compositor turn.