# Lxc Container Patterns > LXC container provisioning and configuration patterns. Use when creating LXC services, managing container networking, or handling LXC template operations. - Skill: `krmorehead/lxc-container-patterns` (Agent Skill) - Install (CLI): `npx skillmds@latest add krmorehead/lxc-container-patterns` - Raw SKILL.md: https://api.skillmd.com/api/skills/krmorehead/lxc-container-patterns/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Coding & Dev Tools - Author: krmorehead (https://skillmd.com/u/krmorehead) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/krmorehead/lxc-container-patterns --- # LXC Container Patterns ## When to Test (Proactive Testing Triggers) **Test IMMEDIATELY when:** - Creating new LXC container services - Adding Docker-in-LXC functionality - Modifying container networking or provisioning - Using `pct exec` commands in configuration - Encountering variable scoping issues with containers - Any container-related Ansible code changes **Previous lesson**: We should have tested after creating the `homeassistant_configure` role instead of debugging blind. Always test after container provisioning code changes. **Environment validation for LXC work:** ```bash set -a && source test.env && set +a ssh -o StrictHostKeyChecking=no root@$PRIMARY_HOST "pct list" ansible home -m ping ``` ## Load Relevant Skills Proactively When working with LXC containers, load these skills IMMEDIATELY: - `lxc-container-patterns` (this skill) - `molecule-testing` for test patterns - `ansible-conventions` for coding standards - `proxmox-safety-rules` for Proxmox operations ## LXC Provisioning Pattern 1. LXC containers use the shared `proxmox_lxc` role via `include_role`. Each service's `_lxc` role is a thin wrapper: ```yaml # roles/pihole_lxc/tasks/main.yml --- - name: Provision Pi-hole container ansible.builtin.include_role: name: proxmox_lxc vars: lxc_ct_id: "{{ pihole_ct_id }}" lxc_ct_hostname: pihole lxc_ct_dynamic_group: pihole lxc_ct_memory: 256 lxc_ct_cores: 1 lxc_ct_disk: "4" lxc_ct_onboot: true lxc_ct_startup_order: 3 ``` 2. The `proxmox_lxc` role handles: template upload, `pct create`, networking, device bind mounts, auto-start, container start, readiness wait, and `add_host` registration. 3. The "Start container" task MUST start containers that exist but are stopped, not only newly created containers. Use `'stopped' in (_lxc_status.stdout | default(''))` as an additional condition alongside `not lxc_exists`. Previous bug: `molecule converge` (idempotent re-run) found kiosk containers in "stopped" state on ai/mesh2. The start task was gated only on `not lxc_exists`, so it skipped the start. "Wait for container init" then failed with "container '401' not running!". ## LXC Readiness and Connection 3. For readiness: use `ls /` not `hostname` (BusyBox containers may lack it). For OpenWrt LXC: use `--ostype unmanaged`. 4. Configure plays use `ansible.builtin.uri` to push config via the NodeManager API (`/api/container/{id}/exec`, `/api/config/push`) over VPN. This is the standard path for 9/10 configure roles. NEVER use `pct exec` or `pct_remote` in configure roles when an API endpoint exists. The only exceptions are `kiosk_configure` (bootstrap — deploys the NM itself) and `openwrt_configure` (no HTTP API on OpenWrt VM). Legacy `pct_remote` references remain in provisioning roles for hypervisor-side operations (`pct config`, `pct status`). 5. For OpenWrt containers (no Python), use `ansible.builtin.raw` with commands wrapped in `/bin/sh -c '...'`. ## Container Networking Requirements 6. LXC container networking MUST match the host's actual topology. Hosts behind OpenWrt (`router_nodes`, `lan_hosts`) use the OpenWrt LAN subnet. Hosts directly on WAN use `ansible_default_ipv4.gateway/prefix`. 7. NEVER hardcode all containers to the OpenWrt LAN subnet. ## Bridge Allocation Patterns 8. Different VM/container types consume bridges differently: - **Router VMs** (OpenWrt): ALL bridges — WAN on `net0`, remaining as LAN ports - **Service VMs**: typically ONE LAN bridge — `proxmox_all_bridges[1]` (first non-WAN) - **LXC containers on LAN hosts**: `proxmox_all_bridges[1]` (LAN bridge) - **LXC containers on WAN hosts**: `proxmox_wan_bridge` — NEVER use `proxmox_all_bridges[1]` ## Template Management 9. Templates are stored locally in `images/` and uploaded to the Proxmox host during provisioning. NEVER use `pveam download` — the host may not have internet access. 10. Previous bug: `pveam download` failed with template name mismatch. Switching to local hosting eliminated the dependency. ## Upload Pattern for LXC Templates 11. ```yaml - name: Upload LXC template to Proxmox ansible.builtin.copy: src: "{{ role_path }}/../../{{ lxc_ct_template_path }}" dest: "/var/lib/vz/template/cache/{{ lxc_ct_template }}" mode: "0644" ``` 12. ALWAYS use `role_path` for the source path. Relative paths like `../../images/...` break when Molecule runs from non-default scenarios. ## LXC Package Management 13. ALWAYS use `install_recommends: false` when installing packages in LXC containers. Many packages Recommend kernel-related metapackages that pull in 70+ MB kernel images. 14. Previous bug: `wireguard-tools` Recommends `wireguard` metapackage which depends on `linux-image-rt-amd64`. Without `install_recommends: false`, apt pulled in a 70MB kernel image that filled the 1GB container disk. ## LXC Disk Sizing Requirements 15. LXC templates compress well (~5:1 for Debian). ALWAYS verify that the rootfs disk is large enough for the EXTRACTED template, not just the compressed size. 16. Rule of thumb: Set disk to at least 2× the compressed template size, minimum 2 GB for any Debian-based container with monitoring or database services. 17. Previous bug: Netdata template was 314MB compressed but expanded to ~1013MB. 1GB rootfs caused `pct create` to fail mid-extraction. ## Per-Host IP Indexing 18. When a service deploys to multiple LAN hosts, each container needs a unique IP on the shared OpenWrt LAN. Use the host's index in its flavor group: `offset + groups[''].index(inventory_hostname)`. 19. WAN hosts use private NAT subnets (10.99.{container_subnet_id}.0/24) via `vmbr_ct` bridge. Container IPs use just the service offset (no group indexing). Collisions are impossible — each host has its own /24. 20. Previous bug: WAN containers shared the household subnet (192.168.86.x). The offset+base+index formula produced IP collisions with host IPs and household devices. Replaced with per-host NAT bridges. ## DNAT for Inbound Connections on NAT Containers 22a. WAN host containers live on private NAT subnets (10.99.x.x) that are unreachable from the LAN or other hosts. When a container needs to accept inbound connections from outside the host, add DNAT rules in the provisioning role (`_lxc/tasks/main.yml`): ```yaml - name: Add DNAT rule for service port (WAN hosts) ansible.builtin.shell: cmd: | set -o pipefail WAN_IF="{{ proxmox_wan_bridge }}" CT_IP="{{ _lxc_net_ip }}" if ! iptables -t nat -C PREROUTING -i "$WAN_IF" -p tcp --dport 9001 \ -j DNAT --to-destination "$CT_IP:9001" 2>/dev/null; then iptables -t nat -A PREROUTING -i "$WAN_IF" -p tcp --dport 9001 \ -j DNAT --to-destination "$CT_IP:9001" fi executable: /bin/bash when: not (_lxc_net_on_lan | bool) changed_when: true - name: Add FORWARD rule for DNAT traffic (WAN hosts) ansible.builtin.shell: cmd: | set -o pipefail CT_IP="{{ _lxc_net_ip }}" if ! iptables -C FORWARD -d "$CT_IP" -p tcp --dport 9001 \ -j ACCEPT 2>/dev/null; then iptables -A FORWARD -d "$CT_IP" -p tcp --dport 9001 -j ACCEPT fi executable: /bin/bash when: not (_lxc_net_on_lan | bool) changed_when: true ``` Key points: - Gate on `not (_lxc_net_on_lan | bool)` — LAN containers are directly reachable and don't need DNAT. - Use `-C` (check) before `-A` (append) for idempotency. - BOTH `PREROUTING DNAT` and `FORWARD ACCEPT` rules are needed. - Services currently using DNAT: Manager API (port 9001, kiosk_lxc), WireGuard (UDP 51820, wireguard_lxc). Previous bug (2026-04-09): Cluster Manager on home (LAN) couldn't reach child Managers on WAN hosts (ai, mesh2, bridge-1, bridge-2). Their kiosk containers were on 10.99.x.19 — unreachable from the LAN. DNAT rules forwarding port 9001 from the host IP to the container IP fixed it. ## Dynamic Inventory and Failed Hosts 21. ALWAYS use `ansible_play_hosts` (not `ansible_play_hosts_all`) in `add_host` loops that register containers. `ansible_play_hosts_all` includes hosts that failed in earlier plays, leading to phantom container registrations for containers that were never created. 22. Previous bug: `ai` failed in infrastructure (clock skew). The `add_host` loop used `ansible_play_hosts_all`, registering `wireguard-ai` even though no container existed. The configure play then failed with UNREACHABLE. ## Container PID Retrieval 23. NEVER use `pct pid` — it does not exist in Proxmox VE. Use `lxc-info -n ` and parse the PID line with `awk '/^PID:/{print $2}'`. 24. Previous bug: `openwrt_mesh_lxc` used `pct pid 103` to get the container PID for WiFi PHY namespace move. It failed with "ERROR: unknown command 'pct pid'". ## pct_remote Connection Requirements 25. The `community.proxmox.proxmox_pct_remote` connection plugin requires the `paramiko` Python package on the controller. ALWAYS add `paramiko` to `requirements.txt` when using `pct_remote`. 26. Previous bug: `ModuleNotFoundError: No module named 'paramiko'` at runtime because the dependency wasn't in `requirements.txt`. ## pct_remote gather_facts Buffer Overflow 38. NEVER use `gather_facts: true` on plays targeting `pct_remote` dynamic groups unless the configure role actually references `ansible_*` facts. The `pct_remote` connection plugin has a limited JSON output buffer. Hosts with many block devices (LVM thin pools, loop devices, device-mapper entries) produce `ansible_devices` output that exceeds the buffer, causing "Module result deserialization failed: No end of json char found". 39. ALWAYS set `gather_facts: false` for `pct_remote` configure plays. None of the current configure roles (wireguard, pihole, rsyslog, netdata) use gathered facts — they only use role variables and facts from `set_fact`. 40. If a future role needs specific facts, use `gather_subset: ['!hardware']` to skip the device-heavy hardware facts instead of gathering all facts. 41. Previous bug: `wireguard-ai` failed during fact gathering via `pct_remote` because the `ai` host (AMD Ryzen, 128GB SSD with LVM thin pool) had 15+ block devices. The full setup module JSON exceeded the `pct_remote` buffer. Other hosts with fewer devices succeeded. Fix: set `gather_facts: false` for all pct_remote configure plays. ## SSH Connection Exhaustion with Parallel pct_remote 42. When multiple `pct_remote` containers run in parallel, each task opens a NEW SSH connection to the Proxmox host. With 4+ containers × N tasks, this causes rapid SSH churn that overwhelms the SSH server, resulting in "Connection reset by peer", "Broken pipe", and "SSH session not active" errors. 43. ALWAYS add `serial: 2` to configure plays that target 4+ `pct_remote` containers. This processes containers in batches of 2, reducing simultaneous SSH connections. 44. Previous bug: WireGuard configure play ran 4 containers in parallel (wireguard-home, wireguard-mesh1, wireguard-ai, wireguard-mesh2). Tasks like `wg genkey | wg pubkey` opened 4+ SSH connections through `home`'s SSH server simultaneously. Two connections dropped with "Connection reset by peer (104)" and "Broken pipe". Fix: added `serial: 2` to the WireGuard configure play. ## pct_remote Intermittent Hangs 55. The `community.proxmox.proxmox_pct_remote` connection plugin uses paramiko for SSH, which can hang indefinitely during connection setup — even with `serial: 1`. Adding `timeout = 60` to `ansible.cfg` and `host_key_auto_add = True` under `[paramiko_connection]` does NOT prevent the hang. 56. NEVER use `pct exec ... systemctl is-active` to check service health. The heartbeat system (4-tier callhome chain) monitors all services continuously. Use the fleet readiness API (`/api/fleet/ready?services=netdata`, `/api/container/netdata/ready`) for health verification. If the fleet API reports a service as not ready, the service IS not ready. If the fleet API is unreachable, the heartbeat watchdog kills the Ansible run within 5 seconds. 57. Previous bug: `Configure Netdata` play targeted `netdata` dynamic group with `pct_remote`. Hung intermittently on `Detect netdata config directory` task (0% CPU, blocked on paramiko SSH handshake). The fleet API would have returned the health status in <1 second without any SSH overhead. ## Device Passthrough in LXC Containers 45. For device passthrough (`/dev/dri`, `/dev/snd`, `/dev/input`) to LXC containers, use `lxc.mount.entry` directives directly in the container config file (`/etc/pve/lxc/.conf`), NOT Proxmox's `-mp` mount option. The `-mp` pre-start hook rejects device directories in unprivileged containers (exit code 32). 46. Use privileged containers (`unprivileged: false`) with `nesting=1` for reliable device access. Add both `lxc.mount.entry` bind mounts AND `lxc.cgroup2.devices.allow` rules. 47. Apply device config atomically: stop container → append config → start container. Gate on whether the entries already exist to make tasks idempotent. 48. NEVER use `echo "..." >> /etc/pve/lxc/.conf` to append to LXC config files. pmxcfs (Proxmox cluster filesystem) rejects in-place append with "File exists" error. ALWAYS use read-modify-write: `cp` to `/tmp`, modify there, `cp` back. 49. Previous bug: `lxc_device_passthrough.yml` used `echo >> $lxc_conf` to append mount entries. This worked on some hosts but failed on ai and mesh2 with "File exists" because pmxcfs doesn't support POSIX append. Fix: copy to /tmp, append there, copy back. 50. Previous bug: Jellyfin and Kodi containers failed with `lxc.hook.pre-start` exit code 32 when using Proxmox `-mp` mounts for `/dev/dri`. Fix: switched to privileged containers with `lxc.mount.entry` directives appended directly to the LXC config file. ## VA-API in Headless LXC Containers 52. ALWAYS use `vainfo --display drm` when running vainfo inside headless LXC containers. Plain `vainfo` tries to connect to an X server (DISPLAY env var), which doesn't exist in headless containers. The `--display drm` flag uses the DRM render node directly. 53. Previous bug: `jellyfin_configure` ran `pct exec {{ ct_id }} -- vainfo` without `--display drm`. Failed with "error: can't connect to X server!" on every run. Fix: added `--display drm` and `failed_when: false`. ## File Deployment via copy + pct push 49. NEVER use heredocs inside `pct exec -- bash -c 'cat > file << EOF ... EOF'` in Ansible `command` or `shell` modules. YAML string folding destroys heredoc structure, causing bash syntax errors ("unexpected token `<'"). 50. ALWAYS use the `copy` + `pct push` pattern for deploying config files into containers: ```yaml - name: Write config to host temp ansible.builtin.copy: content: | {{ variable }} dest: /tmp/service_config.xml mode: "0644" - name: Push config into container ansible.builtin.command: cmd: pct push {{ ct_id }} /tmp/service_config.xml /etc/service/config.xml changed_when: true - name: Clean up host temp ansible.builtin.file: path: /tmp/service_config.xml state: absent ``` 51. Previous bug: `pct exec {{ ct_id }} -- bash -c 'cat > /etc/jellyfin/network.xml << "NETWORK_EOF" ...'` via `ansible.builtin.command` with `>-` folded scalar collapsed newlines. Bash received `cat > file << "EOF" content EOF` on a single line and failed with `syntax error near unexpected token '<'`. Fix: use `ansible.builtin.copy` to write to host temp, then `pct push` into container. ## Configure Play Target Rules 54. **Non-Docker LXC configure plays** MUST target the dynamic group (e.g., `netdata`, `pihole`, `rsyslog`) with `pct_remote` connection. The `pct_remote` plugin handles all commands inside the container transparently. 55. Previous bug: `netdata_configure` targeted `monitoring_nodes` (Proxmox hosts) instead of the `netdata` dynamic group. All commands (`command`, `template`, `systemd`) ran on the HOST, not inside the container. The role checked `/etc/netdata` on the Proxmox host (doesn't exist) and failed. Fix: changed `site.yml` to `hosts: netdata`. ## Docker-in-LXC Configuration Patterns 27. **Docker-in-LXC configure play target**: Configure plays for Docker-in-LXC services MUST target the Proxmox HOST group (e.g., `service_nodes`), NOT the container dynamic group (e.g., `homeassistant`). The role uses `pct exec` commands which only exist on the Proxmox host. 28. **Container ID handling**: Use `homeassistant_ct_id` from `group_vars/all.yml`. NEVER reference undefined `proxmox_vmid` variable in configure roles that run on the host. 29. **File deployment in containers**: Use `pct exec -- bash -c 'cat > file << "EOF" ... EOF'` (heredoc) instead of chained `echo` commands. This preserves YAML structure and avoids quoting nightmares: ```yaml - name: Create config file in container ansible.builtin.shell: cmd: | set -o pipefail pct exec {{ ct_id }} -- bash -c 'cat > /path/config.yml << "CONFIG_EOF" key: value nested: item: data CONFIG_EOF chmod 0644 /path/config.yml' executable: /bin/bash changed_when: true ``` 30. **LXC nesting requirements**: Docker-in-LXC requires `nesting=1` feature. Pass via `lxc_ct_features: ["nesting=1"]` to `proxmox_lxc`. Configure Docker daemon with `cgroupfs` driver for cgroup v2 compatibility (bake into image). 31. **Jinja2 conflicts with Docker templates**: Docker Go template syntax (`{{.Repository}}`) uses the same delimiters as Jinja2. In Ansible tasks, escape with `{{ "{{" }}` and `{{ "}}" }}`, or avoid Docker `--format` flags entirely (use `docker image inspect` instead). 32. Previous bug: Configure play targeted `homeassistant` group (pct_remote connection), but the role used `pct exec` commands. `pct` only exists on the Proxmox host, not inside the container. Fix: change site.yml to target `service_nodes` (host). 33. Previous bug: Docker `--format "{{.Repository}}:{{.Tag}}"` in verify.yml was interpreted by Ansible's Jinja2 engine as undefined variables. Fix: use `docker image inspect` instead of `--format`. ## Dynamic Host Fact Detection in Per-Feature Scenarios 52. Configure roles that need host-level facts (e.g., render device GID for iGPU) MUST detect them dynamically from the host instead of relying on cached facts from infrastructure roles. Per-feature molecule scenarios don't run infrastructure plays, so facts like `igpu_render_gid` are undefined. 53. Pattern: use `stat -c '%g' /dev/dri/renderD*` to detect the render GID at runtime: ```yaml - name: Detect render device GID from host ansible.builtin.shell: cmd: | set -o pipefail stat -c '%g' /dev/dri/renderD* 2>/dev/null | head -1 executable: /bin/bash register: _render_gid changed_when: false failed_when: _render_gid.stdout | trim | length == 0 ``` 54. Previous bug: `moonlight_configure` referenced `igpu_render_gid` (exported by `proxmox_igpu`). The per-feature scenario didn't run infrastructure plays, so the variable was undefined. Fix: detect GID dynamically with `stat`. ## Configure Role Connection Decision 34. When a configure role needs facts from the Proxmox host (e.g., `igpu_render_gid`, `igpu_render_device` from `proxmox_igpu`), it MUST target the Proxmox host group (`media_nodes`, `service_nodes`) and use `pct exec` commands. The `pct_remote` connection cannot access host-side facts because the dynamic group hosts are separate inventory entries. 35. When a configure role only needs container-internal state (e.g., pihole, netdata, rsyslog), it should target the dynamic group with `pct_remote` and use direct commands without `pct exec` wrapping. 36. Previous bug: Jellyfin configure play targeted `hosts: jellyfin` (dynamic group with `pct_remote`) but the role used `pct exec {{ jellyfin_ct_id }} --` wrapping on every command. The `pct_remote` plugin already wraps commands in `pct exec`, causing double wrapping: `pct exec 300 -- pct exec 300 -- systemctl ...`. Fix: change site.yml to target `media_nodes` (Proxmox hosts) since the role needs host-side igpu facts. ## Pi-hole v6 Configuration 33. Pi-hole v6 uses `/etc/pihole/pihole.toml` (NOT `setupVars.conf` or `pihole-FTL.conf`). Use `/usr/bin/pihole-FTL --config ` for programmatic configuration. 34. Pi-hole v6 uses FTL's embedded web server (NOT lighttpd). Do not manage `lighttpd.service`. 35. ALWAYS run `pihole -g` (gravity update) BEFORE switching the container's `resolv.conf` to `127.0.0.1`. Gravity downloads blocklists from the internet, which needs working DNS. 36. For unattended install, pre-seed `/etc/pihole/pihole.toml` with at least `dns.upstreams` before running the installer. Use `PIHOLE_SKIP_OS_CHECK=true` in LXC containers. 37. Previous bug: Configure role set `resolv.conf` to `127.0.0.1`, then ran `pihole -g`. Gravity hung because FTL's DNS wasn't fully initialized. ## Display-Exclusive Hookscript and DRI-Sharing Containers 38. Desktop LXC shares the iGPU DRI render node via bind mount (not exclusive PCI passthrough). The display-exclusive hookscript manages conflicts — when Desktop LXC starts, it stops other DRI-sharing containers (Kodi, Kiosk, Moonlight) and vice versa. 39. Configure roles for DRI-sharing containers MUST include a defensive "ensure container is running" guard: check `pct status`, start if stopped, wait for readiness via `pct exec -- ls /`. This handles re-runs where hookscripts have already stopped the container. 40. Verify assertions for DRI-sharing services MUST skip `systemctl is-active` checks when a competing container (Desktop LXC) has the DRI render node exclusively. The service legitimately can't start without GPU access. 41. When detecting render group GID from `/dev/dri/renderD*`, use `failed_when: false` and fall back to a cached `igpu_render_gid` fact from `proxmox_igpu`. The DRI device may be temporarily unavailable when a competing container has exclusive use. 42. Previous bug: `kiosk_configure` ran `stat -c '%g' /dev/dri/renderD*` but a competing container held exclusive DRI access. No DRI devices existed on the host. Fix: made detection non-fatal with fallback to cached fact. ## Callhome Config Rewrite Pattern (CRITICAL) 47. When rewriting `/etc/default/callhome` on sibling containers (to point them at the local NodeManager after kiosk deployment), NEVER overwrite the entire file. Use targeted `sed -i` to update ONLY `CALLHOME_SERVER` and `CALLHOME_PUBLIC_KEY`. This preserves baked-in variables like `CALLHOME_HOSTNAME` which determine fleet identity. 48. The `kiosk_configure` role's "Rewrite callhome config on sibling containers" task iterates all running containers on the host and updates their callhome config. Two paths: - Debian containers (have `/opt/callhome/`): `bash -c "sed -i ..."` + restart `callhome.service` - OpenWrt containers (have `/usr/sbin/callhome.sh`, no `/opt/callhome`): `sh -c "sed -i ..."` + restart cron 49. OpenWrt containers use BusyBox `sed`, which supports `-i` without a backup extension. ALWAYS use `sh -c` (not `bash -c`) for OpenWrt containers. 50. Previous bug (2026-04-17): `kiosk_configure` used `printf` to overwrite the entire `/etc/default/callhome` on OpenWrt mesh/bridge containers. This destroyed `CALLHOME_HOSTNAME=openwrt-mesh` and `CALLHOME_HOSTNAME=openwrt-bridge`. The containers then heartbeated with their CT hostname instead of the expected fleet service name. The fleet readiness API couldn't find them, causing verify failures. Fix: `sed -i` to update only the changed values. ## Host-Level Socat Proxy Port Conflicts (CRITICAL) 43. When an LXC provisioning role deploys a host-level socat proxy (e.g., `kiosk-display-proxy` on port 6080), a PREVIOUSLY DEPLOYED service with a different name on the SAME port will prevent the new service from starting. The new service crash-loops with `Result=exit-code` and `NRestarts=` while appearing as "activating" to `systemctl is-active`. 44. ALWAYS stop and remove the legacy service BEFORE deploying the replacement in the provisioning role. Include `pkill -f 'socat.*TCP-LISTEN:'` as a belt-and-suspenders cleanup. 45. `cleanup_lan_host.yml` MUST clean up ALL host-level systemd units deployed by provisioning roles. LAN hosts are excluded from the main cleanup play (`proxmox:!lan_hosts`), so without explicit LAN cleanup, stale units survive indefinitely. 46. Previous catastrophe (2026-04-14): KasmVNC migration renamed `kiosk-vnc-proxy` → `kiosk-display-proxy`. mesh1 (LAN host) was excluded from main cleanup. Old `kiosk-vnc-proxy` held port 6080. New service crash-looped 957 times. Masked by adding 100s of verify retries instead of diagnosing the port conflict. The root cause took multiple test cycles to surface because the retries delayed the failure without ever succeeding.