How to validate Anthias changes on the physical hardware testbeds — the /tmp/used-by lockfile protocol, deploying images (plain docker-compose, not balena), the pure-Python overlay method, force-display on headless eglfs, running a pre-release burn-in/soak, and per-board memory/hardware quirks. Use when testing a viewer/server/webview change on real Pi / x86 / Rock Pi 4 hardware, and whenever a release burn-in or soak is requested (pair it with cut-release).
How to validate Anthias changes on the physical hardware testbeds — the /tmp/used-by lockfile protocol, deploying images (plain docker-compose, not balena), the pure-Python overlay method, force-display on headless eglfs, running a pre-release burn-in/soak, and per-board memory/hardware quirks. Use when testing a viewer/server/webview change on real Pi / x86 / Rock Pi 4 hardware, and whenever a release burn-in or soak is requested (pair it with cut-release).
Testbed QA runbook
A reusable procedure for validating Anthias changes on the physical testbeds.
Fleet coverage is one board per architecture: armhf (Pi 2, Pi 3 32-bit),
arm64 (Pi 3 64-bit, Pi 4, Pi 5), x86, and a Rock Pi 4 (generic
arm64). All testbeds are headless. SSH is ssh <USER>@<BOARD_IP> throughout;
substitute the target board's address (kept out of the repo — it is environment
state, not code).
1. Claim the board — the /tmp/used-by lockfile protocol
Testbeds are shared across concurrent agent sessions. Coordinate via a lockfile
at /tmp/used-by on the device (it holds the current holder's agent id).
Two sessions on one viewer collide — the classic symptom is the container being
restarted out from under you, wiping /dev/shm scratch and contaminating
captures.
IRON RULE: never write /tmp/used-by before reading it. Every claim is
read-check-then-write, including re-acquiring after your own release. A blind
echo "myid" > /tmp/used-by is BANNED.
Read first.cat /tmp/used-by. If it holds another agent's id, BACK
OFF — do not touch the box, do not overwrite the lock. Poll read-only until it
clears, then claim.
Claim only when free/yours. Guard the write, e.g.:
L=$(cat /tmp/used-by 2>/dev/null); [ -z "$L" ] && echo "$MYID ..." > /tmp/used-by || echo "HELD: $L — backing off"
The re-acquire is NOT exempt. Between your release and your re-acquire
(e.g. while a long build ran) another agent may have taken it. Re-read before
re-writing; assume they grabbed it until the read proves otherwise.
Release when done. Clear or rm /tmp/used-by so others can take it.
Restore what you changed before releasing: swapped AnthiasViewer binary,
injected assets, config edits. Keep a pristine copy at AnthiasViewer.orig; a
binary hot-swap hits "Text file busy" while the process runs and needs a
container restart to take effect.
Recovery if you clobbered someone: stop immediately. If the lock still
shows your (wrongly-written) id, rm it to clear the field. Never touch it if
it now shows theirs.
2. Deploy a version (plain docker-compose, NOT balena)
Testbeds run published GHCR images via docker compose at ~/anthias
(project anthias), pinned by a <short-hash>-<board> tag in the device's
generated docker-compose.yml. Their git checkouts may sit on WIP branches with
uncommitted edits, but the running containers are official GHCR builds,
decoupled from that source. Board→tag suffix examples: Pi 4 = pi4-64,
Pi 5 = pi5, Rock Pi 4 = arm64, x86 = x86. "Latest" = master HEAD short
hash (git rev-parse --short=7); CI publishes <hash>-<board> for
server/viewer/redis on merge to master.
Deploy (non-destructive, preserves the board's WIP git state):
cp docker-compose.yml docker-compose.yml.bak-<CURRENT-tag> — name the backup
after the tag it contains, i.e. the one you will restore to, never after
the tag you are about to deploy. See the warning below.
sed -i -E 's#(ghcr\.io/screenly/anthias-(server|viewer|redis):)[0-9a-zA-Z]+-#\1<hash>-#g' docker-compose.yml
(the [0-9a-zA-Z]+- eats only the hash prefix, leaving a hyphenated board
suffix like pi4-64 intact)
sudo docker compose pull
sudo docker compose up -d --remove-orphans
sudo docker image prune -af (running images are kept)
Verify: curl -sw '%{http_code}' localhost/api/v2/assets → 200; then read the
viewer logs.
bak-predeploy-<hash> is ambiguous and has already caused near-misses — always
grep a backup before restoring it. The old wording named the backup after the
hash being deployed, so docker-compose.yml.bak-predeploy-fbe83e9 holds
whatever was pinned before fbe83e9 — frequently latest-<board>. But agents
following the same wording have also read it the other way and produced files
named for their contents, so both conventions now exist side by side on the
fleet with opposite meanings. Live examples: on the Pi 4 and Rock Pi 4,
bak-predeploy-fbe83e9 pins latest-<board>, while bak-predeploy-fbe83e9-qa2
on the same box really does pin fbe83e9. Restoring by filename alone silently
moves a board onto a floating latest tag — which then quietly invalidates the
next "baseline" measurement taken on it.
So: name new backups after their contents (step 1), and before any restore run
grep -oE 'anthias-server:[0-9a-zA-Z]+-[a-z0-9-]+' <backup> and confirm it is the
tag you actually want. Never trust the name.
Streaming a locally-built image (skip GHCR): docker save <img> | ssh <USER>@<BOARD_IP> 'sudo docker load' — no gzip. On a fast LAN the wire is
faster than the CPU can (de)compress, so a gzip pipe makes the transfer slower.
Streaming also avoids an intermediate multi-GB file on tight device storage.
Reserve scp for small files (compose templates).
Small-SD disk-fill gotcha: SD/eMMC cards are small and often 80–90% full. A
fresh viewer+server pull adds several GB of layers BEFORE the old images are
freed, briefly hitting 100% (which produces dbus No space left on device
errors). On tight boards, prune unused images first to free room, then pull,
then prune again. Rule of thumb: remove the old image before loading the new one.
Watcher pitfall: a pgrep -f "docker compose pull" polling loop matches its
own ssh command line and reports the job "running" forever. Confirm completion
by side effect (images present, disk stable), not by pgrep.
3. Validate a pure-Python fix without an image rebuild (overlay method)
For server/viewer/common Python changes on a full-stack testbed:
Build a combined tree, tar czf only the changed src/ files
(git diff --name-only origin/master..HEAD -- src/).
scp the tarball over, then per container:
docker cp fixes.tgz <c>:/tmp/ && docker exec <c> tar xzf /tmp/fixes.tgz -C /usr/src/app
(code lives at /usr/src/app/src, editable-installed, so a docker restart
picks it up).
Validate: docker exec <server> python <script> (set
DJANGO_SETTINGS_MODULE=...settings for logic checks); curl http://localhost/api/... for HTTP. To exercise the viewer's mtime-gated
playlist reload, inject rows via Asset.objects.create then
touch ~/.anthias/anthias.db.
Restore to pristine:cd ~/anthias && docker compose up -d --force-recreate <services> drops the overlay writable layer back to the
pinned image. Delete any injected assets first.
Never build the overlay from the shared working tree — extract from the commit
object. The dev-host checkout is shared across concurrent sessions and will
be switched to another branch mid-run (observed twice in one validation, and the
branch under test also gained new commits while it was being tested). Use
git archive <sha> -- src/ | tar x -C <staging> (or git show <sha>:<path>) and
then assert the in-container blob md5 matches that commit, so the report names a
commit you can actually stand behind. Never git checkout on the shared tree to
set up your own run.
Beware: the pinned image can be older than your branch base. Overlaying
<branch>'s files onto an image built from an older commit also carries every
unrelated change to those files between the two. Diff
<image-commit>..<branch-base> for the files you overlay and say in the report
what else came along, or you are attributing a mixed result to one commit.
4. Force-display on headless eglfs boards
The eglfs boards (Pi 3 64-bit, Pi 4) refuse to start the viewer when headless —
wait_for_eglfs_display blocks until a DRM connector reads connected. Correct
signage behavior, but it means a headless testbed renders nothing and you cannot
verify the eglfs pipeline. The debugfs force attribute alone does NOT flip
status on vc4-kms. The reliable way is a kernel cmdline modifier + reboot:
# /boot/firmware/cmdline.txt — append on the single line (space-separated):
video=HDMI-A-1:1920x1080@60e # trailing 'e' forces the connector enabled/connected
After reboot the connector reads connected, the viewer starts, and video
HW-decodes (/dev/video10, frames in playback-stats). Revert by restoring the
.bak-qa copy and rebooting. Wayland boards (Pi 5, x86, Rock Pi 4) do NOT need
this — cage tolerates a missing output. Since all testbeds are headless, you are
always verifying the software pipeline (decode→sink→fb/compositor), not
physical HDMI pixels.
Don't decide by the board's nominal Qt platform — check the display driver.
"linuxfb board" is not the same as "has /dev/fb0". The Pi 3 A+ (32-bit)
runs QT_QPA_PLATFORM=linuxfb but boots dtoverlay=vc4-kms-v3d, so headless it
has no/dev/fb0 at all (DRM finds no CRTC → no fbdev emulation) and the
viewer sits on no framebuffer (/dev/fb0) yet — waiting. It needs the same
force-display treatment as the eglfs boards. The Pi 2 is the genuine legacy-fbdev
case and does have /dev/fb0 regardless. Check ls /dev/fb* and
grep dtoverlay /boot/firmware/config.txt rather than assuming from the Qt
platform name.
Also note the forced mode is a request, not a guarantee: on both the Pi 3 A+
and the Pi 3 64-bit the fbdev/scanout surface came up 1024x768 despite
video=HDMI-A-1:1920x1080@60e on the cmdline and resolution = 1920x1080 in
anthias.conf. Read the actual surface geometry (fb0 byte size, or the DRM mode)
before computing expected pixel geometry for a capture comparison.
Check for a pre-existing modifier before you add one, and leave a .bak-qa.
The Pi 3 64-bit carried a force-display cmdline from an earlier session for
weeks with no backup — which both means the board was silently not
representative of a headless device, and left later sessions with nothing safe to
restore. A cmdline.txt.orig is not a substitute: on that board .orig is
the imaging-time file (console=tty1, ds=nocloud, no cgroup-memory flags), so
"restoring" it would silently change the console and drop the cgroup flags the
stack needs.
5. Standing rules
Never heavy-compute on an SBC. No ffmpeg transcode, image build, or big
compression on a testbed — it is glacial, starves the viewer under test, and
skews the measurement. Produce test media/artifacts on the dev host and copy
them over.
Never broad-pkill on the dev host. The dev host runs many concurrent
Docker services across worktrees; loose patterns (pkill -f buildx) cascade
into unrelated containers. Target by exact PID (kill $(pgrep -fa "<unique substring>") after confirming) or cancel a harness-spawned background job by
its task id. List before killing.
pkill -f / pgrep -f match their own command line. The self-match trap
isn't only a pgrep watcher problem: a pkill -f "<pattern>" sent over SSH
matches the pattern inside its own ssh command line and kills the session
running it (observed mid-run; the board was unaffected but the agent lost its
shell). Use an exact PID, or a bracketed pattern ([m]y-sampler) that cannot
match the literal string in the invoking command.
No persistent logs in /tmp, and no large fixtures either./tmp is a
~948 MB tmpfs: it is wiped on reboot — exactly the event a hotplug/power logger
is trying to capture — and it is mounted usrquota with the quota shared
across every concurrent session. Filling it does more than fail a write: once
the quota is hit, the harness's own Bash output capture breaks and even echo
fails. Multi-megapixel fixtures blow the budget fast (a 24 MP BMP is 72 MB, a
24 MP TIFF 95 MB), so build media and write logs under the user's home
(/home/<user>/<task>-<board>/) or /var/log. Check with df -h /tmp; note
hidden dirs (.venv-style) don't show up in du -sh <dir>/*/.
Disable idle testbeds' assets. After a run, PATCH every asset
{"is_enabled": false} via the v2 API so the viewer idles to black.
Re-enable only the board currently under measurement. Idle rotation wears the
SD card, keeps the decoder/GPU warm (noise), and visually conflates "testing
now" with "yesterday's run".
Validate arch-specific fixes on REAL target hardware. An armhf-image-on-a-
Pi-4-host proxy has a different RAM ceiling, CPU, thermal and timing profile —
it undersells (or misses) the bug. Use the proxy only when no real unit is
reachable, and say so explicitly in the PR.
A single-process / offscreen / linuxfb container is NOT a valid repro for
Cortex-A7 (armhf) crashes. Such a harness takes a different code path and
failed to reproduce a field-known SIGBUS even on the A7 where the crash
definitely exists, so it is worthless as a negative control anywhere. The field
crash lives in the FULL multi-process WebEngine init on a real display.
E2E means the full integrated stack on a real board. Do not tick a PR's
end-to-end box after a component-level harness run. The integrated run (real
service spawning the real asset-rotation → player path) is where a whole class
of bugs — arg plumbing, service wiring, audio defaults, rotation — surfaces
that a standalone module never shows.
Never tick a PR's device-testing box from intent. It goes back only with
measured numbers attached. Writing the checklist before the run is how an
unverified claim ships.
Measure viewer memory as an A/B inside ONE viewer lifetime. Settled-idle
AnthiasViewer RSS varies enormously between restarts on the same board with
the same image and an empty playlist (86.9 / 199 / 209 MB observed on the Pi 2,
depending on whether splash-page memory was still resident). A
restart-to-restart before/after comparison is therefore meaningless. Drive both
cases through one running viewer and sample the phase label alongside memory so
each number is attributable.
memory.peak is fd-local, and that silently corrupts results. Linux's
resettable cgroup peak resets per open file descriptor, so a cat issued after
a tee-based reset returns the since-boot peak, not the peak since your
reset. A naive reset-then-read reports garbage that looks plausible. Either hold
one fd across reset and read, or sample memory.current on an interval and take
the maximum yourself.
6. Burn-in / pre-release soak
A burn-in is not "leave it running and see if it dies". Every rule below exists
because a soak looked clean while measuring nothing.
Verify the playlist is really ROTATING, not just really enabled. The
existing §7 rule says confirm assets are disabled before quoting an idle
number; the converse bites just as hard. A 2026.08.1 burn-in ran a Pi 3 A+ with
a single enabled asset whose duration was 300s (every other board: 20s),
left over from an earlier session parking the board. One static PNG for five
minutes exercises no asset-loop transition, no decode/teardown, no memory churn
— it would have soaked for hours, reported clean, and proved nothing. Audit
enabled count AND dwell time per board and state the resulting full-cycle
duration (healthy fleet values were 40-90s).
Soak every supported asset type on every board — image, video, webpage,
streaming. Quietly reducing the set per board (e.g. "skip webpages on the small
ones") is a coverage loss disguised as caution. If a board genuinely cannot run
a type, that is a FINDING to report, not a config to silently adopt.
Never deviate from the shipping configuration without flagging it loudly.
Stopping celery to keep a 1 GB board off swap-zero makes the board survive and
makes the result meaningless — and it buries the actual finding, which is that
the shipped stack does not fit. It also silently skips whatever runs in celery
(e.g. the CEC display-power beat). If you must deviate to keep a board alive,
the deviation goes in the report as a headline, not a footnote, and the
un-deviated failure gets measured first.
The /tmp/used-by lock is advisory and does NOT hold. During one release
burn-in, two assets were enabled on a board by something other than the lock
holder while the lockfile still read the holder's id, untouched. All API writes
arrive from the docker bridge gateway (172.18.0.1), so the server access log
cannot attribute them to a session. Consequence: a finding was published and had
to be withdrawn. Capture the full playlist state at the START and END of the
window and diff it, and treat docker logs -t on anthias-server (grep
PATCH /api/v2/assets) as the audit trail when a result looks wrong.
Verify core dumps by FILE TYPE, never by filename.core_pattern is core
on every board (dumps land in the crashing process's CWD, i.e. /usr/src/app;
there is no systemd-coredump, so an empty coredumpctl proves nothing). Two
traps:
Every file named core on the fleet is a jsonschema vocabulary JSON
inside site-packages. A find -name 'core*' sweep reports ~100 phantom
"dumps" per board, and a -delete on that pattern would strip package files
out of the image layers. Check ELF magic 7f454c46ande_type == 4
(ET_CORE) before believing or deleting anything.
Rock Pi 4 sets core_uses_pid=0, so its dumps are a bare core, not
core.<pid>. A sweep matching only core\.[0-9]+ silently skips that board.
Background on why dumps appear here at all: see the CEC/libcec teardown bullet
in anthias-release → Ops hygiene.
ssh inside a while read loop eats the loop's stdin and the loop runs
ONCE. A fleet-wide loop reading a board list silently processed only the first
board and reported success — the remaining six were never touched. Use ssh -n
(or < /dev/null) for every ssh inside a loop; conversely you must NOT pass
-n when you are piping a script in via ssh 'bash -s', because -n redirects
stdin from /dev/null and the remote shell receives an empty script (it exits 0
with no output, which looks like a silent pass). Note mapfile does not exist
in zsh, which is the dev host's shell.
A guarded destructive test beats a wedge. When soaking a board you expect to
OOM, poll on a short interval and back the load off automatically at a
threshold (e.g. disable the heavy asset when avail<40 MB or swap_free<15 MB).
That converts "board wedged, needs a physical power cycle nobody can do
remotely" into a recorded measurement, and it is the only safe way to test the
≤1 GB boards under a full playlist.
7. Per-board memory / hardware quirks
On the 1 GB Rock Pi 4 it is celery, not the viewer, that exhausts swap.
This bullet previously claimed the board "could not settle the latest viewer
even with an empty playlist" and blamed the Chromium/QWebEngine baseline. That
was measured on a board whose playlist was not actually empty — four assets
were still enabled, including a webpage and a streaming asset, left over from an
earlier run (see the "disable idle testbeds' assets" rule in §5, which is
exactly the trap). Re-measured with the playlist genuinely disabled:
viewer alone (201 MB RSS + 188 MB QtWebEngine, under its ~773 MB cgroup cap)
sits comfortably; the board holds at ~173 MB available.
bringing celery up on top drives swap to zero on both the release build
and the previous baseline — i.e. it is the steady state of this board, not a
regression in any given build.
with celery stopped (this section's mitigation — but see §6: if you stop
celery during a soak, that deviation from the shipping config is a headline
finding, not a quiet workaround) it is stable at 390-405 MB
available and 368-396 MB swap free.
So: read a wedge here as total memory pressure and suspect celery + leftover
enabled assets first. Verify the playlist is really disabled before quoting
any idle memory number — an "idle" measurement on a board that is quietly
rendering a webpage is worthless. A genuine wedge (SSH banner-exchange timeout
while ICMP and TCP/22 stay up) still needs a physical power cycle.
Read any "wedge / drop / SSH unresponsive" on a ≤1 GB board as memory
pressure first, not CPU. To exercise an arm64 code path without the RAM
ceiling, override DEVICE_TYPE=arm64 on a stable board (Pi 4 / Pi 5) instead.
A self-sustaining wedge won't survive a power cycle either. If the wedge is
caused by an enabled asset (a 1080p video on a 512 MB board), rebooting just
replays it: the board boots, the viewer loads the same asset, and it wedges
again. Break the loop instead — race a docker stop anthias-anthias-viewer-1
against the boot (poll SSH in a tight loop and fire the moment it answers),
then disable the asset before restarting the viewer.
512 MB is too little for Anthias. On a 512 MB Pi 3 A+ the official
installer dogfoods cleanly, but the running stack OOM-wedges: idle already
leans on swap, and the moment the viewer loads a webpage the QtWebEngine
(Chromium) spike exhausts RAM+swap until sshd and even docker stop
responding. Mitigation for testing: docker stop celery and enable only a
light image asset. (Image and video rotation still validate here; webpage
paths do not.)
x86 disk EIO can wedge sshd. A failing block device can throw EIO while
Docker removes an old container's filesystem; the root fs then goes read-only,
forked login shells die, and sshd resets every session at key exchange while
ping/TCP-22 stay up. Recovery needs a physical power cycle (no remote power on
testbeds). Treat repeat EIO as suspect storage hardware, not a script fault.
x86 headless capture needs WLR_BACKENDS=headless. With no monitor, every
DRM connector reads disconnected, cage has no wl_output, and grim fails
with no wl_output. Add - WLR_BACKENDS=headless to the viewer service
environment: and up -d it — cage then creates a virtual HEADLESS-1
output you can screenshot. grim is not in the viewer image (apt-get install -y grim in the container); any up -d that recreates the viewer wipes it.
Remove the env line on restore. When grabbing video frames, sample several
spaced frames and check mean luma before concluding "black = broken" (pre-roll
can be black).
SSH is pubkey-only to a non-root user. Log in as <USER>@<BOARD_IP> with
the installed SSH public key. The Pi boards set PasswordAuthentication no;
probing with -o PubkeyAuthentication=no, -o PreferredAuthentications=password, or sshpass returns "Permission denied
(publickey)" and looks locked out. It is not — drop the password flags and
let pubkey auth run.
Fresh Debian x86 with a root password set ships no sudo / no sudo-group / no
curl (this is a website-doc gap, not an installer bug). Recovery as root:
su - → apt install -y sudo → usermod -aG sudo <USER> → re-login. Debian
only auto-installs sudo and adds the first user to the sudo group when the root
password is left blank.
