Testbed QA

Testbed QA runbook

A reusable procedure for validating Anthias changes on the physical testbeds. Fleet coverage is one board per architecture: armhf (Pi 2, Pi 3 32-bit), arm64 (Pi 3 64-bit, Pi 4, Pi 5), x86, and a Rock Pi 4 (generic arm64). All testbeds are headless. SSH is ssh <USER>@<BOARD_IP> throughout; substitute the target board's address (kept out of the repo — it is environment state, not code).

1. Claim the board — the /tmp/used-by lockfile protocol

Testbeds are shared across concurrent agent sessions. Coordinate via a lockfile at /tmp/used-by on the device (it holds the current holder's agent id). Two sessions on one viewer collide — the classic symptom is the container being restarted out from under you, wiping /dev/shm scratch and contaminating captures.

IRON RULE: never write /tmp/used-by before reading it. Every claim is read-check-then-write, including re-acquiring after your own release. A blind echo "myid" > /tmp/used-by is BANNED.

• Read first. cat /tmp/used-by. If it holds another agent's id, BACK OFF — do not touch the box, do not overwrite the lock. Poll read-only until it clears, then claim.
• Claim only when free/yours. Guard the write, e.g.: L=$(cat /tmp/used-by 2>/dev/null); [ -z "$L" ] && echo "$MYID ..." > /tmp/used-by || echo "HELD: $L — backing off"
• The re-acquire is NOT exempt. Between your release and your re-acquire (e.g. while a long build ran) another agent may have taken it. Re-read before re-writing; assume they grabbed it until the read proves otherwise.
• Release when done. Clear or rm /tmp/used-by so others can take it.
• Restore what you changed before releasing: swapped AnthiasViewer binary, injected assets, config edits. Keep a pristine copy at AnthiasViewer.orig; a binary hot-swap hits "Text file busy" while the process runs and needs a container restart to take effect.
• Recovery if you clobbered someone: stop immediately. If the lock still shows your (wrongly-written) id, rm it to clear the field. Never touch it if it now shows theirs.

2. Deploy a version (plain docker-compose, NOT balena)

Testbeds run published GHCR images via docker compose at ~/anthias (project anthias), pinned by a <short-hash>-<board> tag in the device's generated docker-compose.yml. Their git checkouts may sit on WIP branches with uncommitted edits, but the running containers are official GHCR builds, decoupled from that source. Board→tag suffix examples: Pi 4 = pi4-64, Pi 5 = pi5, Rock Pi 4 = arm64, x86 = x86. "Latest" = master HEAD short hash (git rev-parse --short=7); CI publishes <hash>-<board> for server/viewer/redis on merge to master.

Deploy (non-destructive, preserves the board's WIP git state):

1. cp docker-compose.yml docker-compose.yml.bak-<CURRENT-tag> — name the backup after the tag it contains, i.e. the one you will restore to, never after the tag you are about to deploy. See the warning below.
2. sed -i -E 's#(ghcr\.io/screenly/anthias-(server|viewer|redis):)[0-9a-zA-Z]+-#\1<hash>-#g' docker-compose.yml (the [0-9a-zA-Z]+- eats only the hash prefix, leaving a hyphenated board suffix like pi4-64 intact)
3. sudo docker compose pull
4. sudo docker compose up -d --remove-orphans
5. sudo docker image prune -af (running images are kept)

Verify: curl -sw '%{http_code}' localhost/api/v2/assets → 200; then read the viewer logs.

bak-predeploy-<hash> is ambiguous and has already caused near-misses — always grep a backup before restoring it. The old wording named the backup after the hash being deployed, so docker-compose.yml.bak-predeploy-fbe83e9 holds whatever was pinned before fbe83e9 — frequently latest-<board>. But agents following the same wording have also read it the other way and produced files named for their contents, so both conventions now exist side by side on the fleet with opposite meanings. Live examples: on the Pi 4 and Rock Pi 4, bak-predeploy-fbe83e9 pins latest-<board>, while bak-predeploy-fbe83e9-qa2 on the same box really does pin fbe83e9. Restoring by filename alone silently moves a board onto a floating latest tag — which then quietly invalidates the next "baseline" measurement taken on it.

So: name new backups after their contents (step 1), and before any restore run grep -oE 'anthias-server:[0-9a-zA-Z]+-[a-z0-9-]+' <backup> and confirm it is the tag you actually want. Never trust the name.

Streaming a locally-built image (skip GHCR): docker save <img> | ssh <USER>@<BOARD_IP> 'sudo docker load' — no gzip. On a fast LAN the wire is faster than the CPU can (de)compress, so a gzip pipe makes the transfer slower. Streaming also avoids an intermediate multi-GB file on tight device storage. Reserve scp for small files (compose templates).

Small-SD disk-fill gotcha: SD/eMMC cards are small and often 80–90% full. A fresh viewer+server pull adds several GB of layers BEFORE the old images are freed, briefly hitting 100% (which produces dbus No space left on device errors). On tight boards, prune unused images first to free room, then pull, then prune again. Rule of thumb: remove the old image before loading the new one.

Watcher pitfall: a pgrep -f "docker compose pull" polling loop matches its own ssh command line and reports the job "running" forever. Confirm completion by side effect (images present, disk stable), not by pgrep.

3. Validate a pure-Python fix without an image rebuild (overlay method)

For server/viewer/common Python changes on a full-stack testbed:

1. Build a combined tree, tar czf only the changed src/ files (git diff --name-only origin/master..HEAD -- src/).
2. scp the tarball over, then per container: docker cp fixes.tgz <c>:/tmp/ && docker exec <c> tar xzf /tmp/fixes.tgz -C /usr/src/app (code lives at /usr/src/app/src, editable-installed, so a docker restart picks it up).
3. Validate: docker exec <server> python <script> (set DJANGO_SETTINGS_MODULE=...settings for logic checks); curl http://localhost/api/... for HTTP. To exercise the viewer's mtime-gated playlist reload, inject rows via Asset.objects.create then touch ~/.anthias/anthias.db.
4. Restore to pristine: cd ~/anthias && docker compose up -d --force-recreate <services> drops the overlay writable layer back to the pinned image. Delete any injected assets first.

Never build the overlay from the shared working tree — extract from the commit object. The dev-host checkout is shared across concurrent sessions and will be switched to another branch mid-run (observed twice in one validation, and the branch under test also gained new commits while it was being tested). Use git archive <sha> -- src/ | tar x -C <staging> (or git show <sha>:<path>) and then assert the in-container blob md5 matches that commit, so the report names a commit you can actually stand behind. Never git checkout on the shared tree to set up your own run.

Beware: the pinned image can be older than your branch base. Overlaying <branch>'s files onto an image built from an older commit also carries every unrelated change to those files between the two. Diff <image-commit>..<branch-base> for the files you overlay and say in the report what else came along, or you are attributing a mixed result to one commit.

4. Force-display on headless eglfs boards

The eglfs boards (Pi 3 64-bit, Pi 4) refuse to start the viewer when headless — wait_for_eglfs_display blocks until a DRM connector reads connected. Correct signage behavior, but it means a headless testbed renders nothing and you cannot verify the eglfs pipeline. The debugfs force attribute alone does NOT flip status on vc4-kms. The reliable way is a kernel cmdline modifier + reboot:

# /boot/firmware/cmdline.txt — append on the single line (space-separated):
video=HDMI-A-1:1920x1080@60e     # trailing 'e' forces the connector enabled/connected

After reboot the connector reads connected, the viewer starts, and video HW-decodes (/dev/video10, frames in playback-stats). Revert by restoring the .bak-qa copy and rebooting. Wayland boards (Pi 5, x86, Rock Pi 4) do NOT need this — cage tolerates a missing output. Since all testbeds are headless, you are always verifying the software pipeline (decode→sink→fb/compositor), not physical HDMI pixels.

Don't decide by the board's nominal Qt platform — check the display driver. "linuxfb board" is not the same as "has /dev/fb0". The Pi 3 A+ (32-bit) runs QT_QPA_PLATFORM=linuxfb but boots dtoverlay=vc4-kms-v3d, so headless it has no /dev/fb0 at all (DRM finds no CRTC → no fbdev emulation) and the viewer sits on no framebuffer (/dev/fb0) yet — waiting. It needs the same force-display treatment as the eglfs boards. The Pi 2 is the genuine legacy-fbdev case and does have /dev/fb0 regardless. Check ls /dev/fb* and grep dtoverlay /boot/firmware/config.txt rather than assuming from the Qt platform name.

Also note the forced mode is a request, not a guarantee: on both the Pi 3 A+ and the Pi 3 64-bit the fbdev/scanout surface came up 1024x768 despite video=HDMI-A-1:1920x1080@60e on the cmdline and resolution = 1920x1080 in anthias.conf. Read the actual surface geometry (fb0 byte size, or the DRM mode) before computing expected pixel geometry for a capture comparison.

Check for a pre-existing modifier before you add one, and leave a .bak-qa. The Pi 3 64-bit carried a force-display cmdline from an earlier session for weeks with no backup — which both means the board was silently not representative of a headless device, and left later sessions with nothing safe to restore. A cmdline.txt.orig is not a substitute: on that board .orig is the imaging-time file (console=tty1, ds=nocloud, no cgroup-memory flags), so "restoring" it would silently change the console and drop the cgroup flags the stack needs.

5. Standing rules

• Never heavy-compute on an SBC. No ffmpeg transcode, image build, or big compression on a testbed — it is glacial, starves the viewer under test, and skews the measurement. Produce test media/artifacts on the dev host and copy them over.
• Never broad-pkill on the dev host. The dev host runs many concurrent Docker services across worktrees; loose patterns (pkill -f buildx) cascade into unrelated containers. Target by exact PID (kill $(pgrep -fa "<unique substring>") after confirming) or cancel a harness-spawned background job by its task id. List before killing.
• pkill -f / pgrep -f match their own command line. The self-match trap isn't only a pgrep watcher problem: a pkill -f "<pattern>" sent over SSH matches the pattern inside its own ssh command line and kills the session running it (observed mid-run; the board was unaffected but the agent lost its shell). Use an exact PID, or a bracketed pattern ([m]y-sampler) that cannot match the literal string in the invoking command.
• No persistent logs in /tmp, and no large fixtures either. /tmp is a ~948 MB tmpfs: it is wiped on reboot — exactly the event a hotplug/power logger is trying to capture — and it is mounted usrquota with the quota shared across every concurrent session. Filling it does more than fail a write: once the quota is hit, the harness's own Bash output capture breaks and even echo fails. Multi-megapixel fixtures blow the budget fast (a 24 MP BMP is 72 MB, a 24 MP TIFF 95 MB), so build media and write logs under the user's home (/home/<user>/<task>-<board>/) or /var/log. Check with df -h /tmp; note hidden dirs (.venv-style) don't show up in du -sh <dir>/*/.
• Disable idle testbeds' assets. After a run, PATCH every asset {"is_enabled": false} via the v2 API so the viewer idles to black. Re-enable only the board currently under measurement. Idle rotation wears the SD card, keeps the decoder/GPU warm (noise), and visually conflates "testing now" with "yesterday's run".
• Validate arch-specific fixes on REAL target hardware. An armhf-image-on-a- Pi-4-host proxy has a different RAM ceiling, CPU, thermal and timing profile — it undersells (or misses) the bug. Use the proxy only when no real unit is reachable, and say so explicitly in the PR.
• A single-process / offscreen / linuxfb container is NOT a valid repro for Cortex-A7 (armhf) crashes. Such a harness takes a different code path and failed to reproduce a field-known SIGBUS even on the A7 where the crash definitely exists, so it is worthless as a negative control anywhere. The field crash lives in the FULL multi-process WebEngine init on a real display.
• E2E means the full integrated stack on a real board. Do not tick a PR's end-to-end box after a component-level harness run. The integrated run (real service spawning the real asset-rotation → player path) is where a whole class of bugs — arg plumbing, service wiring, audio defaults, rotation — surfaces that a standalone module never shows.
• Never tick a PR's device-testing box from intent. It goes back only with measured numbers attached. Writing the checklist before the run is how an unverified claim ships.
• Measure viewer memory as an A/B inside ONE viewer lifetime. Settled-idle AnthiasViewer RSS varies enormously between restarts on the same board with the same image and an empty playlist (86.9 / 199 / 209 MB observed on the Pi 2, depending on whether splash-page memory was still resident). A restart-to-restart before/after comparison is therefore meaningless. Drive both cases through one running viewer and sample the phase label alongside memory so each number is attributable.
• memory.peak is fd-local, and that silently corrupts results. Linux's resettable cgroup peak resets per open file descriptor, so a cat issued after a tee-based reset returns the since-boot peak, not the peak since your reset. A naive reset-then-read reports garbage that looks plausible. Either hold one fd across reset and read, or sample memory.current on an interval and take the maximum yourself.

6. Burn-in / pre-release soak

A burn-in is not "leave it running and see if it dies". Every rule below exists because a soak looked clean while measuring nothing.

• Verify the playlist is really ROTATING, not just really enabled. The existing §7 rule says confirm assets are disabled before quoting an idle number; the converse bites just as hard. A 2026.08.1 burn-in ran a Pi 3 A+ with a single enabled asset whose duration was 300s (every other board: 20s), left over from an earlier session parking the board. One static PNG for five minutes exercises no asset-loop transition, no decode/teardown, no memory churn — it would have soaked for hours, reported clean, and proved nothing. Audit enabled count AND dwell time per board and state the resulting full-cycle duration (healthy fleet values were 40-90s).
• Soak every supported asset type on every board — image, video, webpage, streaming. Quietly reducing the set per board (e.g. "skip webpages on the small ones") is a coverage loss disguised as caution. If a board genuinely cannot run a type, that is a FINDING to report, not a config to silently adopt.
• Never deviate from the shipping configuration without flagging it loudly. Stopping celery to keep a 1 GB board off swap-zero makes the board survive and makes the result meaningless — and it buries the actual finding, which is that the shipped stack does not fit. It also silently skips whatever runs in celery (e.g. the CEC display-power beat). If you must deviate to keep a board alive, the deviation goes in the report as a headline, not a footnote, and the un-deviated failure gets measured first.
• The /tmp/used-by lock is advisory and does NOT hold. During one release burn-in, two assets were enabled on a board by something other than the lock holder while the lockfile still read the holder's id, untouched. All API writes arrive from the docker bridge gateway (172.18.0.1), so the server access log cannot attribute them to a session. Consequence: a finding was published and had to be withdrawn. Capture the full playlist state at the START and END of the window and diff it, and treat docker logs -t on anthias-server (grep PATCH /api/v2/assets) as the audit trail when a result looks wrong.
• Verify core dumps by FILE TYPE, never by filename. core_pattern is core on every board (dumps land in the crashing process's CWD, i.e. /usr/src/app; there is no systemd-coredump, so an empty coredumpctl proves nothing). Two traps:
  • Every file named core on the fleet is a jsonschema vocabulary JSON inside site-packages. A find -name 'core*' sweep reports ~100 phantom "dumps" per board, and a -delete on that pattern would strip package files out of the image layers. Check ELF magic 7f454c46 and e_type == 4 (ET_CORE) before believing or deleting anything.
  • Rock Pi 4 sets core_uses_pid=0, so its dumps are a bare core, not core.<pid>. A sweep matching only core\.[0-9]+ silently skips that board. Background on why dumps appear here at all: see the CEC/libcec teardown bullet in anthias-release → Ops hygiene.
• ssh inside a while read loop eats the loop's stdin and the loop runs ONCE. A fleet-wide loop reading a board list silently processed only the first board and reported success — the remaining six were never touched. Use ssh -n (or < /dev/null) for every ssh inside a loop; conversely you must NOT pass -n when you are piping a script in via ssh 'bash -s', because -n redirects stdin from /dev/null and the remote shell receives an empty script (it exits 0 with no output, which looks like a silent pass). Note mapfile does not exist in zsh, which is the dev host's shell.
• A guarded destructive test beats a wedge. When soaking a board you expect to OOM, poll on a short interval and back the load off automatically at a threshold (e.g. disable the heavy asset when avail<40 MB or swap_free<15 MB). That converts "board wedged, needs a physical power cycle nobody can do remotely" into a recorded measurement, and it is the only safe way to test the ≤1 GB boards under a full playlist.

7. Per-board memory / hardware quirks

• On the 1 GB Rock Pi 4 it is celery, not the viewer, that exhausts swap. This bullet previously claimed the board "could not settle the latest viewer even with an empty playlist" and blamed the Chromium/QWebEngine baseline. That was measured on a board whose playlist was not actually empty — four assets were still enabled, including a webpage and a streaming asset, left over from an earlier run (see the "disable idle testbeds' assets" rule in §5, which is exactly the trap). Re-measured with the playlist genuinely disabled:
  • viewer alone (201 MB RSS + 188 MB QtWebEngine, under its ~773 MB cgroup cap) sits comfortably; the board holds at ~173 MB available.
  • bringing celery up on top drives swap to zero on both the release build and the previous baseline — i.e. it is the steady state of this board, not a regression in any given build.
  • with celery stopped (this section's mitigation — but see §6: if you stop celery during a soak, that deviation from the shipping config is a headline finding, not a quiet workaround) it is stable at 390-405 MB available and 368-396 MB swap free. So: read a wedge here as total memory pressure and suspect celery + leftover enabled assets first. Verify the playlist is really disabled before quoting any idle memory number — an "idle" measurement on a board that is quietly rendering a webpage is worthless. A genuine wedge (SSH banner-exchange timeout while ICMP and TCP/22 stay up) still needs a physical power cycle.
• Read any "wedge / drop / SSH unresponsive" on a ≤1 GB board as memory pressure first, not CPU. To exercise an arm64 code path without the RAM ceiling, override DEVICE_TYPE=arm64 on a stable board (Pi 4 / Pi 5) instead.
• A self-sustaining wedge won't survive a power cycle either. If the wedge is caused by an enabled asset (a 1080p video on a 512 MB board), rebooting just replays it: the board boots, the viewer loads the same asset, and it wedges again. Break the loop instead — race a docker stop anthias-anthias-viewer-1 against the boot (poll SSH in a tight loop and fire the moment it answers), then disable the asset before restarting the viewer.
• 512 MB is too little for Anthias. On a 512 MB Pi 3 A+ the official installer dogfoods cleanly, but the running stack OOM-wedges: idle already leans on swap, and the moment the viewer loads a webpage the QtWebEngine (Chromium) spike exhausts RAM+swap until sshd and even docker stop responding. Mitigation for testing: docker stop celery and enable only a light image asset. (Image and video rotation still validate here; webpage paths do not.)
• x86 disk EIO can wedge sshd. A failing block device can throw EIO while Docker removes an old container's filesystem; the root fs then goes read-only, forked login shells die, and sshd resets every session at key exchange while ping/TCP-22 stay up. Recovery needs a physical power cycle (no remote power on testbeds). Treat repeat EIO as suspect storage hardware, not a script fault.
• x86 headless capture needs WLR_BACKENDS=headless. With no monitor, every DRM connector reads disconnected, cage has no wl_output, and grim fails with no wl_output. Add - WLR_BACKENDS=headless to the viewer service environment: and up -d it — cage then creates a virtual HEADLESS-1 output you can screenshot. grim is not in the viewer image (apt-get install -y grim in the container); any up -d that recreates the viewer wipes it. Remove the env line on restore. When grabbing video frames, sample several spaced frames and check mean luma before concluding "black = broken" (pre-roll can be black).
• SSH is pubkey-only to a non-root user. Log in as <USER>@<BOARD_IP> with the installed SSH public key. The Pi boards set PasswordAuthentication no; probing with -o PubkeyAuthentication=no, -o PreferredAuthentications=password, or sshpass returns "Permission denied (publickey)" and looks locked out. It is not — drop the password flags and let pubkey auth run.
• Fresh Debian x86 with a root password set ships no sudo / no sudo-group / no curl (this is a website-doc gap, not an installer bug). Recovery as root: su - → apt install -y sudo → usermod -aG sudo <USER> → re-login. Debian only auto-installs sudo and adds the first user to the sudo group when the root password is left blank.