# Anthias Release > Release, fleet, CI and observability operations for Anthias — balena cloud API + balenaOS track topology, CalVer version stamping and the release-cut/OTA sequencing, the Docker image builder + toolchain, CI-runs-on-merge-SHA behavior, host test isolation, Sentry conventions/diagnostics, and telemetry. Read before cutting a release, deploying to fleets, debugging CI, or triaging Sentry. - Skill: `screenly/anthias-release` (Agent Skill) - Install (CLI): `npx skillmds@latest add screenly/anthias-release` - Raw SKILL.md: https://api.skillmd.com/api/skills/screenly/anthias-release/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: DevOps & Infra - Author: screenly (https://skillmd.com/u/screenly) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/screenly/anthias-release --- # Anthias release, fleet, CI & observability knowledge Durable operational knowledge for shipping Anthias to the balena fleets and the apt/ansible installs. Each bullet carries the mechanism plus the fixing PR/CalVer where known; OPEN items are flagged. ``/`` are the public `screenly` / `screenly_ose/anthias-*` identifiers. ## Balena cloud API (device mutation, pinning, OTA trigger) - **Filtered bulk PATCH mutates the whole matching set in one call.** `PATCH /v7/device?$filter=` with a JSON body applies that body to *every* matching device in one request (the whole matching set in seconds). Use for any fleet-wide cloud-side mutation (unpin, tags, notes); put exclusions inside the `$filter`. Validated clearing a large pin backlog (PR #2980). - **Pin field trap:** a device's pin is `is_pinned_on__release`. `should_be_running__release` is *computed* (pin if set, else the fleet's tracked release) — filtering it `ne null` matches the whole fleet, and the API rejects it as a PATCH body property. Supervisor pin/target is `should_be_managed_by__release`. - **OData encoding:** PineJS rejects percent-encoded OData syntax. Keep `$ = ( ) , : / '` literal; encode only spaces / value contents. `curl --data-urlencode` mangles the filter — space-encode (`%20`) manually. `GET /v7/device/$count?$filter=...` returns `{"d": N}` — cheap post-mutation verification. - **resinhup OS-HUP triggers need a descriptive User-Agent.** The balena actions host that serves the raw cloud-API OS-update trigger sits behind a WAF that rejects the default `Python-urllib/x.y` User-Agent; set any descriptive `User-Agent` and the request goes through. `api.balena-cloud.com` itself does NOT need this. A trigger can still return a transient `HTTP 500 {"error":"Device is busy: configuring"}` — retry. - **Log API caps at 1000 lines.** `GET /device/v2//logs?count=N` returns at most 1000 lines regardless of N. On a device in a tight loop the whole window can be server/celery/redis chatter with ZERO viewer lines — classify such a device ANOMALY (logs saturated), not STILL-BROKEN. A standby device emits ~2 lines/5s so upgrade markers scroll out within minutes — use device DB state as ground truth, not just logs. - **Mass remediation: deploy a self-healing release, do NOT script per-device SSH.** balena's SSH gateway throttles hard under concurrency (~6 concurrent → minutes each; ~12 → all time out). A self-healing OTA reaches every device (including offline ones on reconnect) with no SSH. Reserve SSH for one-off diagnosis. CLI 25.x is `balena device ssh [service]`; force `-t`, sleep before commands, pipe through `sed 's/\r//g'` (never redirect stdout to a file — it suppresses output). - **Never print device identifiers from public-repo CI.** Anthias is public → Actions logs are world-readable. balena uuids + online status + hourly timestamps are correlatable customer data. Any script a public workflow runs must default to aggregate-only output; per-device detail goes behind an explicit `--verbose` flag documented as local-only. ## balenaOS track topology & fleet roller - **Two tracks, mutually unreachable at their tops** (verified 2026-06-03): **Regular** — latest `7.2.0` (pi1/3/4/5), `7.0.5` (x86); pi2 (`raspberrypi2`, ARMv7) is arch-capped at `5.1.20` with **no 6.x/7.x and no ESR track**. **ESR** (CalVer) — pi3/4/5 newest `2026.1.0` (`next`/pre-GA), `2025.7.0` = `current`/GA; x86 ESR current `2026.4.0`. `7.2.0` and `2026.1.0` are each track-tops with no HUP target and cannot cross to the other track. Net: regular-7.2.0 and ESR-2026.1.0 pools can only be merged by reflash, not HUP. - **Direction: standardize on ESR, GA/`current` (not `next`).** ESR has long support windows suited to signage. Targets: pi1/3/4/5 → `2025.7.0`; x86 → `2026.4.0`; pi2 special-cased on regular `5.1.20` (no ESR). Devices already on `2026.1.0` (next) are ahead of GA and can't be pulled back (no downgrade HUP). - **Disk image build** (`build-balena-disk-image.yaml`) currently does `balena os download --version latest` = **regular** track; the ESR decision requires the per-device `(current)` ESR (`balena os versions --esr`, pick the `(current)` line; `menu-esr` is interactive-only). - **Fleet roller = `bin/balena_fleet_maintenance.py`** — cautious staged HUP + release-unpin roller: dry-run by default, online-only, `--percent` tranches, resumable via `./.balena-maint-state.tsv`. Add a same-track pre-filter before widening past the canary. Only online devices (a minority of the registered fleet) are HUP-able. - **Fleet config.txt is IAC, not the dashboard.** config.txt knobs live in `balena-host-config.json` at repo root, applied by the `Apply fleet host configuration` step (upsert each `BALENA_HOST_CONFIG_` via `balena env set`, then prune vars not in the file). Audit doc: `docs/balena-fleet-host-config.md`. **Malformed HOST_CONFIG values → supervisor reconcile/reboot loop** (`Reboot has been scheduled ... {}` every ~0.5s, never converging). Issue #2947: fleet stored `dtoverlay="vc4-kms-v3d"` with literal quotes → firmware never loaded the overlay → eglfs_kms couldn't start. ## Balena release cutting, version stamping & OTA deploy - **CalVer `YYYY.0M.MICRO` — always check the current month.** A new month resets MICRO to 0 (after `2026.05.2` the next is `2026.06.0`, not `2026.05.3`). Bump three files: `package.json` + `pyproject.toml` zero-padded (`2026.06.0`); `uv.lock`'s `name = "anthias"` entry uses **strict semver `2026.6.0`** (month loses its leading zero). Git tag = `v2026.06.0`. - **Balena renders CalVer as `0.0.0` unless normalized.** balena reads `version` from `balena.yml` and requires strict 3-segment semver with **no leading zeros**. Fixed PR #2952: `bin/render_balena_yml.sh ` normalizes (`2026.05.1`→`2026.5.1`) and stamps an **unquoted** `version:` line. Re-deploying the same version is intended (balena auto-appends `+rev1`, `+rev2`…). - **Release-cut sequencing (don't skip):** publishing GitHub release `v` triggers `build-balena-disk-image.yaml` (`release: published`). Its `preflight` verifies every `ghcr.io//anthias-{server,viewer,redis}:-` image already exists in GHCR; those are built by `docker-build.yaml` on the **master push**. So: **merge → wait for the master Docker Image Build to finish (all boards; pi2/pi3 QEMU jobs finish last) → THEN publish the release.** Cutting early fails preflight (guard catches it, no harm). Re-trigger with `gh workflow run build-balena-disk-image.yaml -f tag=v`. - **`balena-cloud-deploy` jobs are the fleet OTA; `balena-build-images` builds bootable disk images.** OTA can be green while the disk-image build fails — independent. `deploy-balena-manual.yaml` is the standalone "OTA-deploy an existing tag" hook. - **Deploying master to fleets (no release):** run `bin/balena_ota_deploy.sh ` locally per board (pi2, pi3, pi3-64, pi4-64, pi5, x86, rockpi4). Sequence: version-bump PR → merge → wait for the bump commit's master Docker build (all boards) → preflight `docker buildx imagetools inspect` on all `-` GHCR refs → run the script per board sequentially (~7–20 min each). Run from a checkout of the exact release commit. After an OTA there is still NO git tag / GH release / disk image — cutting the GH release later is a separate explicit step. - **Transient disk-image failures → rerun, don't debug.** `balena os download --version latest` intermittently dies with `ERR_STREAM_PREMATURE_CLOSE` (a CDN blip). `gh run rerun --failed` fixes it. - **Notes/publish flow:** `gh release create v --draft --generate-notes --title --target ` (short hash is rejected as `target_commitish`; draft avoids triggering deploy while editing), then `gh release edit --notes-file`, then `gh release edit v --draft=false --latest`. - **P0: pre-rebrand balena upgrade wiped playlists (data unreferenced, not lost).** Pre-rebrand devices store data at `/data/.screenly/screenly.db` + `/data/screenly_assets/`; the post-rebrand container reads `/data/.anthias/anthias.db` + `/data/anthias_assets/`. `bin/migrate_legacy_paths.sh` only ran on the apt/ansible host install, never on balena → `start_server.sh` found no `anthias.db`, ran a fresh `migrate`, and the viewer came up `Playlist is empty`. **Detection gotcha:** `Playlist is empty` / `standby.png` looks "healthy" to a naive crash-scan — a device that played before and is empty after IS the regression. Fixed 2026.06.0 (PR #2964): base-path-parameterized migration (adopt legacy db, symlink assets); idempotent, healed the affected devices. - **pi4 config-reconcile reboot-loop (2026.05.2 incident, RESOLVED #2961).** pi4 was the only board with a `dtparam` (`i2c_arm=on,spi=on,audio=on`) — balena's config.txt backend never converges on the comma-joined dtparam, so every pi4 supervisor thrashed into a reboot loop (the entire pi4 fleet; unpinned pi4 devices went offline). Updating the supervisor does NOT fix it. Fix: pi4 dtparam → single-value `audio=on`; i2c/spi are unused by a signage viewer. ## Docker image builder & toolchain - **Validate Dockerfile/image changes with the real artifact, not throwaway containers.** For any `docker/Dockerfile.*.j2` or `tools/image_builder/` change, run `uv run python -m tools.image_builder --build-target --service --disable-cache-mounts`, then `docker save | ssh device 'docker load'` and run the actual container. Re-running `apt install` in a hand-stitched `debian:trixie` proves nothing (misses cache-mount semantics, multi-stage copies, board-specific Jinja2 conditionals). - **No parallel image builds.** Run `tools.image_builder` invocations sequentially — concurrent arm64 buildx-under-QEMU jobs thrash BuildKit and crash mid-build. Build → save+upload → deploy → validate → next board. - **Userspace-only — no custom kernel/distro/Yocto.** Anthias ships a userspace docker stack on top of whatever Debian/Armbian/Pi OS the operator installed. Kernel-side fixes not in upstream channels are out of scope — file upstream or degrade gracefully (reject at upload, drop a feature under memory pressure, log a breadcrumb). The `+rpt1` ffmpeg repo (`archive.raspberrypi.com`) is acceptable (Raspberry Pi maintains it as a first-party userspace addition). - **Linaro EOL froze all `latest-` tags (RESOLVED #3070).** `build_qt5.sh` fetched the Linaro `gcc-7.4.1-2019.02` armhf cross-compiler from `releases.linaro.org`, which was retired → pi2/pi3 armhf jobs fail. `docker-build.yaml`'s `publish-latest` mirrors onto the floating `latest-` tag ONLY after the *entire* matrix is green → two dead 32-bit jobs froze all `latest-*`, stranding merged fixes for anyone tracking `latest`. Per-board pinned `-` images still push on a red run. Fix #3070: install Debian `crossbuild-essential-armhf` (gcc 14, same prefix), symlink under the legacy `/src/gcc-linaro-.../bin/` path the frozen Qt5 `qmake.conf` bakes into `CROSS_COMPILE`. - **Diagnosing "latest still broken": read the code inside the image, don't compare digests** (a fresh `latest` build ≠ the `` build byte-for-byte even from identical source). Use `docker create` + `docker cp ` + grep. - **`docker-build.yaml` runs on push to master ONLY, not on PRs** — a build-system fix can't be validated by the PR's own CI; validate locally (branch name must have NO slash — it becomes a Docker tag). - **bun blocks postinstall scripts by default.** `bun install` (incl. `-g`) blocks npm lifecycle scripts — logs "Blocked N postinstalls" but does NOT fail. Packages shipping native binaries via postinstall break silently. Bit Anthias on `bun install -g balena-cli@25.1.3` (`@balena/compose-parser` fetches a Go binary → `balena deploy` failed with `Unexpected end of JSON input`). Fix: targeted `bun pm trust ` (in `$HOME/.bun/install/global`); prefer targeted over `--trust` (all). ## CI behavior & test infrastructure - **PR test CI runs on the ephemeral merge SHA, not the branch head.** `pull_request` workflows (Run Unit Tests / docker-test, mypy, Python Linter, OpenAPI Schema, CodeQL) run against `refs/pull//merge`. `gh api .../commits//check-runs` shows only head-SHA checks (SonarCloud + `copilot-pull-request-reviewer`) and looks like the suite "didn't run". Find the real runs via `gh run list --json name,headSha,event,conclusion,status` and filter by name. Rapid pushes each spawn a fresh merge SHA + run set (no `concurrency:` cancel block); the newest merge SHA is the one that matters. - **Host test isolation lives in the ROOT `conftest.py`** (as of #2215): fake redis, gi/pydbus stubs, `ENVIRONMENT=test`. It used to live only in `tests/conftest.py`, which pytest does NOT load for `src/anthias_server/api/tests/` — so running the API suite alone had no redis mock and every viewer-notifying view 500'd with `ConnectionError` to `redis:6379`. Two process-wide singletons in settings.py (`ViewerPublisher`, `ReplyCollector`) also leaked a real client across tests — the root conftest's `_mock_redis` autouse fixture seeds both with the per-test fake and clears them after. When a host-only 500 doesn't reproduce in CI, suspect these singletons or a `connect_to_redis` bypass; verify against the full CI-parity run (`pytest -m "not integration"`). - **E2E means the integrated stack, not a component harness.** Don't tick a PR's "end-to-end test" box on a standalone module run. On PR #3004 a pi3 video helper passed standalone but the first true integrated run immediately exposed a real defect (a failing audio branch killed video wholesale). If a surface genuinely can't be covered, say so explicitly in the PR body instead of ticking the box. ## Sentry conventions & diagnostics - **Project:** org/project slug `anthias/anthias`; issues named `ANTHIAS-`. Error tracking shipped in 2026.6.2 (PR #3007) — "first seen" dates before that release are meaningless. Sentry release string = `+` (`get_sentry_release`); the fleet runs many releases at once. - **API quirks:** `GET /api/0/organizations/` may 403 while `/api/0/projects/` works (scope-dependent). Issue list uses `?query=is:unresolved`; `statsPeriod` only accepts `''`, `24h`, `14d`. Mutations need an issue-write-scoped token. Resolve with `{"status":"resolvedInNextRelease"}`; rate limit 15 req/s (add `sleep 1` in loops). The Sentry plugin's MCP server is not always connected in the dev-host session — fall back to the REST API. - **`resolvedInNextRelease` ignores build metadata.** Sentry picks the issue's highest-seen release as the reopen baseline and **ignores semver `+hash` build metadata** — an issue baselined at `2026.7.0` while the fleet is on `2026.7.1` can transiently REOPEN from un-upgraded devices until the fix ships. - **environment tag vs baggage mismatch = test/CI stack leaking into prod Sentry, not a device failure.** An event tagged `environment=production` whose Celery task headers carry `baggage: sentry-environment=test` means the *producer* ran in a test stack but the *worker* fell back to production defaults. Root cause (event in normalize_video_asset, "no such table: assets"): `anthias-celery` in docker-compose.test.yml lacked `ENVIRONMENT=test`/`ANTHIAS_TEST_DB_PATH`. settings.py defaults BOTH the DSN and the DB path off `ENVIRONMENT`, so any compose service missing the var silently becomes "production". Fixed #3013 by sharing the env block via a YAML anchor. When triaging, compare the event's `environment` tag against `sentry-environment` in the task baggage and the hostnames. - **Device context tags (PR #3021 / #3025).** Events are sent from inside containers, so Sentry's stock OS context never sees the host. #3021 tags `device_type` (build-time ENV), `kernel_release`/`kernel_machine` (containers share the host kernel), `board_model` (`/proc/device-tree/model`, absent on x86). #3025 adds `balena: true|false`. Keep `settings.py` import-light — importing anthias_common.utils there crashes django-stubs' mypy plugin in CI's slim env, so `is_balena_deploy()` is inlined. Segment webview-crash cohorts by `kernel_machine` (aarch64 → pi3-64 stream; armv7l → Qt5.15 WebEngine build). - **Triage rule:** resolve every unresolved issue not seen on the current release (Sentry auto-reopens as regression if it recurs). Real code defects fixed in the 2026-07 sweep: PR #3180 (`reconcile_stuck_processing` bare `time_limit=300` → SIGKILL trio), PR #3181 (unguarded `await self.send()` in `asset_update` consumer, disconnect-vs-broadcast race → narrow `except RuntimeError`), PR #3182 (before_send drop for client-disconnect `Response content shorter than Content-Length` static noise). ## Telemetry - **GA4 device double-counting (root cause FIXED 2026-06-03).** Telemetry (`src/anthias_server/lib/telemetry.py`, hourly beat) attributes each device by `client_id` = a random 15-char `device_id` stored ONLY in Redis; GA counts a new device whenever the id changes. `Dockerfile.redis.j2` launched `redis-server` with NO config file, so `dir` defaulted to CWD (`/`) and RDB/AOF landed on the ephemeral writable layer instead of the mounted `redis-data:/var/lib/redis` volume → every container recreation wiped `device_id`. Fix = explicit `--dir /var/lib/redis --appendonly yes --save "3600 1 300 100 60 10000"`. - **OPEN telemetry findings (not fixed):** `device_type` is always `unknown` (celery's `environment:` never exports `DEVICE_TYPE`); `hardware_model` is `unknown` on x86 (`parse_cpu_info` only matches exact-case `Model`, x86 uses lowercase `model`); identity is random, not anchored to stable hardware, so any Redis flush mints a new device; `send_telemetry` only catches `RequestException` but GA `/mp/collect` returns 2xx even for bad payloads. Stale docstring: the actual event name is `device_active`, not `version`. ## Ops hygiene (testbeds) - **When a testbed's `df -h /` is ~100%, check for core-dump accumulation first.** A Pi 5 filled its 15 GB SD with ~1283 celery core dumps (`core.`, ~38 MB each, every ~5 min), crash-looping celery with `No space left on device`. With docker's containerd snapshotter the dumps were NOT under `/var/lib/docker` and NOT visible via `docker exec` — they lived at `/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots//fs/usr/src/app/core.*`. Root cause fixed 2026-05-31: the `get_display_power` beat task ran `_CEC_QUERY_SCRIPT` as a `python -c` subprocess; on a Pi 5 with no usable CEC adapter, libcec's adapter thread aborts at interpreter exit (`FATAL: exception not rethrown`, SIGABRT — a C++/pthread teardown abort the try/except can't catch) and dumps a core. Fix: write the answer, flush, then `os._exit(0)` to skip Python/libcec teardown. ## Website / marketing - **`website/assets/images/screenshots/*.png` is gitignored** — captures come from the `marketing-screenshots.yaml` workflow artifact, downloaded at build time (never committed, to avoid merge noise and stale snapshots). CI: `deploy-website.yaml` runs `gh run download` before `hugo build`. Local: `bun run screenshots:fetch`. Anything consuming the captures must assume they may be absent and provide a graceful fallback; never `git add` a `.png` under `screenshots/`.