Anthias Release

Anthias release, fleet, CI & observability knowledge

Durable operational knowledge for shipping Anthias to the balena fleets and the apt/ansible installs. Each bullet carries the mechanism plus the fixing PR/CalVer where known; OPEN items are flagged. <org>/<fleet-slug> are the public screenly / screenly_ose/anthias-* identifiers.

Balena cloud API (device mutation, pinning, OTA trigger)

• Filtered bulk PATCH mutates the whole matching set in one call. PATCH /v7/device?$filter=<odata> with a JSON body applies that body to every matching device in one request (the whole matching set in seconds). Use for any fleet-wide cloud-side mutation (unpin, tags, notes); put exclusions inside the $filter. Validated clearing a large pin backlog (PR #2980).
• Pin field trap: a device's pin is is_pinned_on__release. should_be_running__release is computed (pin if set, else the fleet's tracked release) — filtering it ne null matches the whole fleet, and the API rejects it as a PATCH body property. Supervisor pin/target is should_be_managed_by__release.
• OData encoding: PineJS rejects percent-encoded OData syntax. Keep $ = ( ) , : / ' literal; encode only spaces / value contents. curl --data-urlencode mangles the filter — space-encode (%20) manually. GET /v7/device/$count?$filter=... returns {"d": N} — cheap post-mutation verification.
• resinhup OS-HUP triggers need a descriptive User-Agent. The balena actions host that serves the raw cloud-API OS-update trigger sits behind a WAF that rejects the default Python-urllib/x.y User-Agent; set any descriptive User-Agent and the request goes through. api.balena-cloud.com itself does NOT need this. A trigger can still return a transient HTTP 500 {"error":"Device is busy: configuring"} — retry.
• Log API caps at 1000 lines. GET /device/v2/<uuid>/logs?count=N returns at most 1000 lines regardless of N. On a device in a tight loop the whole window can be server/celery/redis chatter with ZERO viewer lines — classify such a device ANOMALY (logs saturated), not STILL-BROKEN. A standby device emits ~2 lines/5s so upgrade markers scroll out within minutes — use device DB state as ground truth, not just logs.
• Mass remediation: deploy a self-healing release, do NOT script per-device SSH. balena's SSH gateway throttles hard under concurrency (~6 concurrent → minutes each; ~12 → all time out). A self-healing OTA reaches every device (including offline ones on reconnect) with no SSH. Reserve SSH for one-off diagnosis. CLI 25.x is balena device ssh <uuid> [service]; force -t, sleep before commands, pipe through sed 's/\r//g' (never redirect stdout to a file — it suppresses output).
• Never print device identifiers from public-repo CI. Anthias is public → Actions logs are world-readable. balena uuids + online status + hourly timestamps are correlatable customer data. Any script a public workflow runs must default to aggregate-only output; per-device detail goes behind an explicit --verbose flag documented as local-only.

balenaOS track topology & fleet roller

• Two tracks, mutually unreachable at their tops (verified 2026-06-03): Regular — latest 7.2.0 (pi1/3/4/5), 7.0.5 (x86); pi2 (raspberrypi2, ARMv7) is arch-capped at 5.1.20 with no 6.x/7.x and no ESR track. ESR (CalVer) — pi3/4/5 newest 2026.1.0 (next/pre-GA), 2025.7.0 = current/GA; x86 ESR current 2026.4.0. 7.2.0 and 2026.1.0 are each track-tops with no HUP target and cannot cross to the other track. Net: regular-7.2.0 and ESR-2026.1.0 pools can only be merged by reflash, not HUP.
• Direction: standardize on ESR, GA/current (not next). ESR has long support windows suited to signage. Targets: pi1/3/4/5 → 2025.7.0; x86 → 2026.4.0; pi2 special-cased on regular 5.1.20 (no ESR). Devices already on 2026.1.0 (next) are ahead of GA and can't be pulled back (no downgrade HUP).
• Disk image build (build-balena-disk-image.yaml) currently does balena os download --version latest = regular track; the ESR decision requires the per-device (current) ESR (balena os versions <type> --esr, pick the (current) line; menu-esr is interactive-only).
• Fleet roller = bin/balena_fleet_maintenance.py — cautious staged HUP + release-unpin roller: dry-run by default, online-only, --percent tranches, resumable via ./.balena-maint-state.tsv. Add a same-track pre-filter before widening past the canary. Only online devices (a minority of the registered fleet) are HUP-able.
• Fleet config.txt is IAC, not the dashboard. config.txt knobs live in balena-host-config.json at repo root, applied by the Apply fleet host configuration step (upsert each BALENA_HOST_CONFIG_<key> via balena env set, then prune vars not in the file). Audit doc: docs/balena-fleet-host-config.md. Malformed HOST_CONFIG values → supervisor reconcile/reboot loop (Reboot has been scheduled ... {} every ~0.5s, never converging). Issue #2947: fleet stored dtoverlay="vc4-kms-v3d" with literal quotes → firmware never loaded the overlay → eglfs_kms couldn't start.

Balena release cutting, version stamping & OTA deploy

• CalVer YYYY.0M.MICRO — always check the current month. A new month resets MICRO to 0 (after 2026.05.2 the next is 2026.06.0, not 2026.05.3). Bump three files: package.json + pyproject.toml zero-padded (2026.06.0); uv.lock's name = "anthias" entry uses strict semver 2026.6.0 (month loses its leading zero). Git tag = v2026.06.0.
• Balena renders CalVer as 0.0.0 unless normalized. balena reads version from balena.yml and requires strict 3-segment semver with no leading zeros. Fixed PR #2952: bin/render_balena_yml.sh <dest> <raw-version> normalizes (2026.05.1→2026.5.1) and stamps an unquoted version: line. Re-deploying the same version is intended (balena auto-appends +rev1, +rev2…).
• Release-cut sequencing (don't skip): publishing GitHub release v<ver> triggers build-balena-disk-image.yaml (release: published). Its preflight verifies every ghcr.io/<org>/anthias-{server,viewer,redis}:<short-hash>-<board> image already exists in GHCR; those are built by docker-build.yaml on the master push. So: merge → wait for the master Docker Image Build to finish (all boards; pi2/pi3 QEMU jobs finish last) → THEN publish the release. Cutting early fails preflight (guard catches it, no harm). Re-trigger with gh workflow run build-balena-disk-image.yaml -f tag=v<ver>.
• balena-cloud-deploy jobs are the fleet OTA; balena-build-images builds bootable disk images. OTA can be green while the disk-image build fails — independent. deploy-balena-manual.yaml is the standalone "OTA-deploy an existing tag" hook.
• Deploying master to fleets (no release): run bin/balena_ota_deploy.sh <board> <version> <short-hash> locally per board (pi2, pi3, pi3-64, pi4-64, pi5, x86, rockpi4). Sequence: version-bump PR → merge → wait for the bump commit's master Docker build (all boards) → preflight docker buildx imagetools inspect on all <short-hash>-<board> GHCR refs → run the script per board sequentially (~7–20 min each). Run from a checkout of the exact release commit. After an OTA there is still NO git tag / GH release / disk image — cutting the GH release later is a separate explicit step.
• Transient disk-image failures → rerun, don't debug. balena os download --version latest intermittently dies with ERR_STREAM_PREMATURE_CLOSE (a CDN blip). gh run rerun <run-id> --failed fixes it.
• Notes/publish flow: gh release create v<ver> --draft --generate-notes --title <ver> --target <FULL-sha> (short hash is rejected as target_commitish; draft avoids triggering deploy while editing), then gh release edit --notes-file, then gh release edit v<ver> --draft=false --latest.
• P0: pre-rebrand balena upgrade wiped playlists (data unreferenced, not lost). Pre-rebrand devices store data at /data/.screenly/screenly.db + /data/screenly_assets/; the post-rebrand container reads /data/.anthias/anthias.db + /data/anthias_assets/. bin/migrate_legacy_paths.sh only ran on the apt/ansible host install, never on balena → start_server.sh found no anthias.db, ran a fresh migrate, and the viewer came up Playlist is empty. Detection gotcha: Playlist is empty / standby.png looks "healthy" to a naive crash-scan — a device that played before and is empty after IS the regression. Fixed 2026.06.0 (PR #2964): base-path-parameterized migration (adopt legacy db, symlink assets); idempotent, healed the affected devices.
• pi4 config-reconcile reboot-loop (2026.05.2 incident, RESOLVED #2961). pi4 was the only board with a dtparam (i2c_arm=on,spi=on,audio=on) — balena's config.txt backend never converges on the comma-joined dtparam, so every pi4 supervisor thrashed into a reboot loop (the entire pi4 fleet; unpinned pi4 devices went offline). Updating the supervisor does NOT fix it. Fix: pi4 dtparam → single-value audio=on; i2c/spi are unused by a signage viewer.

Docker image builder & toolchain

• Validate Dockerfile/image changes with the real artifact, not throwaway containers. For any docker/Dockerfile.*.j2 or tools/image_builder/ change, run uv run python -m tools.image_builder --build-target <board> --service <svc> --disable-cache-mounts, then docker save | ssh device 'docker load' and run the actual container. Re-running apt install in a hand-stitched debian:trixie proves nothing (misses cache-mount semantics, multi-stage copies, board-specific Jinja2 conditionals).
• No parallel image builds. Run tools.image_builder invocations sequentially — concurrent arm64 buildx-under-QEMU jobs thrash BuildKit and crash mid-build. Build → save+upload → deploy → validate → next board.
• Userspace-only — no custom kernel/distro/Yocto. Anthias ships a userspace docker stack on top of whatever Debian/Armbian/Pi OS the operator installed. Kernel-side fixes not in upstream channels are out of scope — file upstream or degrade gracefully (reject at upload, drop a feature under memory pressure, log a breadcrumb). The +rpt1 ffmpeg repo (archive.raspberrypi.com) is acceptable (Raspberry Pi maintains it as a first-party userspace addition).
• Linaro EOL froze all latest-<board> tags (RESOLVED #3070). build_qt5.sh fetched the Linaro gcc-7.4.1-2019.02 armhf cross-compiler from releases.linaro.org, which was retired → pi2/pi3 armhf jobs fail. docker-build.yaml's publish-latest mirrors onto the floating latest-<board> tag ONLY after the entire matrix is green → two dead 32-bit jobs froze all latest-*, stranding merged fixes for anyone tracking latest. Per-board pinned <hash>-<board> images still push on a red run. Fix #3070: install Debian crossbuild-essential-armhf (gcc 14, same prefix), symlink under the legacy /src/gcc-linaro-.../bin/ path the frozen Qt5 qmake.conf bakes into CROSS_COMPILE.
• Diagnosing "latest still broken": read the code inside the image, don't compare digests (a fresh latest build ≠ the <hash> build byte-for-byte even from identical source). Use docker create + docker cp <path> + grep.
• docker-build.yaml runs on push to master ONLY, not on PRs — a build-system fix can't be validated by the PR's own CI; validate locally (branch name must have NO slash — it becomes a Docker tag).
• bun blocks postinstall scripts by default. bun install (incl. -g) blocks npm lifecycle scripts — logs "Blocked N postinstalls" but does NOT fail. Packages shipping native binaries via postinstall break silently. Bit Anthias on bun install -g balena-cli@25.1.3 (@balena/compose-parser fetches a Go binary → balena deploy failed with Unexpected end of JSON input). Fix: targeted bun pm trust <pkg> (in $HOME/.bun/install/global); prefer targeted over --trust (all).

CI behavior & test infrastructure

• PR test CI runs on the ephemeral merge SHA, not the branch head. pull_request workflows (Run Unit Tests / docker-test, mypy, Python Linter, OpenAPI Schema, CodeQL) run against refs/pull/<N>/merge. gh api .../commits/<HEAD>/check-runs shows only head-SHA checks (SonarCloud + copilot-pull-request-reviewer) and looks like the suite "didn't run". Find the real runs via gh run list --json name,headSha,event,conclusion,status and filter by name. Rapid pushes each spawn a fresh merge SHA + run set (no concurrency: cancel block); the newest merge SHA is the one that matters.
• Host test isolation lives in the ROOT conftest.py (as of #2215): fake redis, gi/pydbus stubs, ENVIRONMENT=test. It used to live only in tests/conftest.py, which pytest does NOT load for src/anthias_server/api/tests/ — so running the API suite alone had no redis mock and every viewer-notifying view 500'd with ConnectionError to redis:6379. Two process-wide singletons in settings.py (ViewerPublisher, ReplyCollector) also leaked a real client across tests — the root conftest's _mock_redis autouse fixture seeds both with the per-test fake and clears them after. When a host-only 500 doesn't reproduce in CI, suspect these singletons or a connect_to_redis bypass; verify against the full CI-parity run (pytest -m "not integration").
• E2E means the integrated stack, not a component harness. Don't tick a PR's "end-to-end test" box on a standalone module run. On PR #3004 a pi3 video helper passed standalone but the first true integrated run immediately exposed a real defect (a failing audio branch killed video wholesale). If a surface genuinely can't be covered, say so explicitly in the PR body instead of ticking the box.

Sentry conventions & diagnostics

• Project: org/project slug anthias/anthias; issues named ANTHIAS-<n>. Error tracking shipped in 2026.6.2 (PR #3007) — "first seen" dates before that release are meaningless. Sentry release string = <CalVer>+<GIT_SHORT_HASH> (get_sentry_release); the fleet runs many releases at once.
• API quirks: GET /api/0/organizations/ may 403 while /api/0/projects/ works (scope-dependent). Issue list uses ?query=is:unresolved; statsPeriod only accepts '', 24h, 14d. Mutations need an issue-write-scoped token. Resolve with {"status":"resolvedInNextRelease"}; rate limit 15 req/s (add sleep 1 in loops). The Sentry plugin's MCP server is not always connected in the dev-host session — fall back to the REST API.
• resolvedInNextRelease ignores build metadata. Sentry picks the issue's highest-seen release as the reopen baseline and ignores semver +hash build metadata — an issue baselined at 2026.7.0 while the fleet is on 2026.7.1 can transiently REOPEN from un-upgraded devices until the fix ships.
• environment tag vs baggage mismatch = test/CI stack leaking into prod Sentry, not a device failure. An event tagged environment=production whose Celery task headers carry baggage: sentry-environment=test means the producer ran in a test stack but the worker fell back to production defaults. Root cause (event in normalize_video_asset, "no such table: assets"): anthias-celery in docker-compose.test.yml lacked ENVIRONMENT=test/ANTHIAS_TEST_DB_PATH. settings.py defaults BOTH the DSN and the DB path off ENVIRONMENT, so any compose service missing the var silently becomes "production". Fixed #3013 by sharing the env block via a YAML anchor. When triaging, compare the event's environment tag against sentry-environment in the task baggage and the hostnames.
• Device context tags (PR #3021 / #3025). Events are sent from inside containers, so Sentry's stock OS context never sees the host. #3021 tags device_type (build-time ENV), kernel_release/kernel_machine (containers share the host kernel), board_model (/proc/device-tree/model, absent on x86). #3025 adds balena: true|false. Keep settings.py import-light — importing anthias_common.utils there crashes django-stubs' mypy plugin in CI's slim env, so is_balena_deploy() is inlined. Segment webview-crash cohorts by kernel_machine (aarch64 → pi3-64 stream; armv7l → Qt5.15 WebEngine build).
• Triage rule: resolve every unresolved issue not seen on the current release (Sentry auto-reopens as regression if it recurs). Real code defects fixed in the 2026-07 sweep: PR #3180 (reconcile_stuck_processing bare time_limit=300 → SIGKILL trio), PR #3181 (unguarded await self.send() in asset_update consumer, disconnect-vs-broadcast race → narrow except RuntimeError), PR #3182 (before_send drop for client-disconnect Response content shorter than Content-Length static noise).

Telemetry

• GA4 device double-counting (root cause FIXED 2026-06-03). Telemetry (src/anthias_server/lib/telemetry.py, hourly beat) attributes each device by client_id = a random 15-char device_id stored ONLY in Redis; GA counts a new device whenever the id changes. Dockerfile.redis.j2 launched redis-server with NO config file, so dir defaulted to CWD (/) and RDB/AOF landed on the ephemeral writable layer instead of the mounted redis-data:/var/lib/redis volume → every container recreation wiped device_id. Fix = explicit --dir /var/lib/redis --appendonly yes --save "3600 1 300 100 60 10000".
• OPEN telemetry findings (not fixed): device_type is always unknown (celery's environment: never exports DEVICE_TYPE); hardware_model is unknown on x86 (parse_cpu_info only matches exact-case Model, x86 uses lowercase model); identity is random, not anchored to stable hardware, so any Redis flush mints a new device; send_telemetry only catches RequestException but GA /mp/collect returns 2xx even for bad payloads. Stale docstring: the actual event name is device_active, not version.

Ops hygiene (testbeds)

• When a testbed's df -h / is ~100%, check for core-dump accumulation first. A Pi 5 filled its 15 GB SD with ~1283 celery core dumps (core.<pid>, ~38 MB each, every ~5 min), crash-looping celery with No space left on device. With docker's containerd snapshotter the dumps were NOT under /var/lib/docker and NOT visible via docker exec — they lived at /var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/<N>/fs/usr/src/app/core.*. Root cause fixed 2026-05-31: the get_display_power beat task ran _CEC_QUERY_SCRIPT as a python -c subprocess; on a Pi 5 with no usable CEC adapter, libcec's adapter thread aborts at interpreter exit (FATAL: exception not rethrown, SIGABRT — a C++/pthread teardown abort the try/except can't catch) and dumps a core. Fix: write the answer, flush, then os._exit(0) to skip Python/libcec teardown.

Website / marketing

• website/assets/images/screenshots/*.png is gitignored — captures come from the marketing-screenshots.yaml workflow artifact, downloaded at build time (never committed, to avoid merge noise and stale snapshots). CI: deploy-website.yaml runs gh run download before hugo build. Local: bun run screenshots:fetch. Anything consuming the captures must assume they may be absent and provide a graceful fallback; never git add a .png under screenshots/.