Vm Lifecycle Architecture

VM Lifecycle Architecture

Context

This project manages multiple VM and LXC container types on Proxmox. Each service follows a two-role pattern. Shared infrastructure runs once per host. This skill covers the GENERAL patterns that apply to ALL service types. Service-specific patterns (OpenWrt networking, Home Assistant setup, etc.) belong in their own skills.

Rules

1. Every service type gets exactly two roles: <type>_vm or <type>_lxc (provision) and <type>_configure (setup). NEVER mix provisioning and configuration in one role.
2. LXC provision roles (<type>_lxc) MUST use include_role: proxmox_lxc with service-specific vars. NEVER duplicate container creation logic.
3. Shared roles (proxmox_bridges, proxmox_backup, proxmox_pci_passthrough, proxmox_igpu) run ONCE per host. They export facts consumed by all service roles.
4. Service-specific variables live in role defaults/main.yml. Shared variables live in group_vars/all.yml. NEVER put service-specific defaults in group_vars.
5. Each provision role MUST check for existing VM/container before creating. Guard ALL creation tasks with when: not vm_exists | bool (VMs) or when: not lxc_exists | bool (containers via proxmox_lxc).
6. VMIDs: 100s network, 200s services, 300s media, 400s desktop, 500s observability, 600s gaming. All defined in group_vars/all.yml.
7. The <type>_vm role adds the VM to dynamic inventory via add_host. The <type>_configure role runs in a separate play targeting that group.
8. NEVER reference another role's defaults. Use set_fact with cacheable: true or add_host variables to pass data.
9. Every provision play targeting Proxmox hosts MUST include deploy_stamp as its last role.
10. Every new role MUST have meta/main.yml with author, license: proprietary, role_name, description, min_ansible_version, and platforms.
11. Provision plays target flavor groups (e.g., router_nodes, service_nodes), NOT proxmox directly. Shared infra targets proxmox.
12. Every VM MUST configure --onboot 1 --startup order=N via qm set. This task runs unconditionally to self-heal. Define <type>_vm_startup_order in role defaults.
13. When a role deploys files to the host, ALWAYS add them to playbooks/cleanup.yml only (unified cleanup; molecule/default/cleanup.yml imports it — one list to maintain).
14. Optional env variables go in role defaults/main.yml via lookup('env', ...) | default('', true). NEVER add optional vars to REQUIRED_ENV in build.py.
15. Every configured feature MUST have a corresponding assertion in verify.yml. "VM is running" is NOT sufficient — verify services, network topology, auto-start, and state files.
16. Feature plays that target dynamic groups MUST be paired with a deploy_stamp play targeting the flavor group (Proxmox host). The VM doesn't store vm_builds.fact — the host does.
17. Post-baseline features are implemented as separate task files in the configure role, NOT as separate roles. This avoids cross-role variable dependencies and keeps the configure role as the single owner of VM configuration.
18. Every entry point that targets a dynamic group as a separate ansible-playbook invocation (per-feature converge, verify, cleanup/rollback) MUST reconstruct the group first. add_host state is ephemeral.
19. LXC container networking MUST match the host's actual topology. Hosts behind OpenWrt (router_nodes, lan_hosts) use the OpenWrt LAN subnet. Hosts directly on WAN use ansible_default_ipv4.gateway/prefix. NEVER hardcode all containers to the OpenWrt LAN subnet.
20. add_host loops in proxmox_lxc MUST use ansible_play_hosts (not ansible_play_hosts_all). The latter includes hosts that failed in earlier plays, creating phantom container registrations.
21. Bake, don't configure at runtime (see project-structure.mdc Design principles). Custom images are REQUIRED. Provision roles verify the image exists and hard-fail if missing. Configure roles NEVER install packages — all packages are baked into the image. To add a package, update the image build script and rebuild. Three documented exceptions exist (see below).
22. One path, no fallbacks (see project-structure.mdc Design principles). NEVER add stock/generic image fallback logic. One tested code path per feature. Missing prerequisites fail with an actionable error message.
23. Follow community standards (see project-structure.mdc Design principles). Check upstream tooling before writing custom workarounds.
24. Documented exceptions to bake principle: three cases where runtime installation is acceptable. Each MUST be explicitly documented in the project plan with rationale:
    • Docker pull of pinned image tag: deterministic and versioned. Used for Docker-in-LXC services (e.g., Home Assistant pre-pulls homeassistant/home-assistant:stable).
    • Desktop LXC via build-images.sh: the desktop rootfs tarball is built with all packages (KDE, GNOME, KasmVNC) baked in. GPU drivers are selected at build time based on the target host's igpu_vendor.
    • Windows VMs via ISO + autounattend.xml: install-from-ISO IS the bake approach for Windows. The ISO + autounattend.xml produce a deterministic, unattended install with virtio drivers pre-injected. This is NOT an exception — it is how Windows images are built. Any OTHER runtime package installation (apt install, opkg install, pip install during converge) is rejected. If you need a new package, add it to the image build script.
25. Service migration: stop legacy before deploying replacement. When renaming a systemd service (e.g., kiosk-vnc-proxy → kiosk-display-proxy), the provisioning role MUST stop and remove the old service BEFORE deploying the new one. Add a pre-deployment task that systemctl stop/disable + rm -f + pkill the legacy service. Without this, the old service holds the port and the new service crash-loops. Applies to host-level socat proxies AND container-internal services.
26. LAN host cleanup parity. The main cleanup play targets proxmox:!lan_hosts. LAN hosts ONLY get cleanup via tasks/cleanup_lan_host.yml. When adding host-level systemd units in a provisioning role, ALWAYS add the unit name to BOTH playbooks/cleanup.yml (primary hosts) AND tasks/cleanup_lan_host.yml (LAN hosts).

Playbook execution order (site.yml)

Play 0: proxmox_backup             (targets: proxmox, tag: backup)
         deploy_stamp: backup
Play 1: proxmox_bridges            (targets: proxmox — shared infra)
         proxmox_pci_passthrough
         proxmox_igpu
         deploy_stamp: infrastructure
Play 2: <type>_vm                  (targets: <flavor_group> — VM provision)
         deploy_stamp: <type>_vm
Play 3: <type>_configure           (targets: <type> — dynamic group)
Play 4+: Feature plays             (targets: <type> — dynamic group, per-feature tags)
         deploy_stamp on <flavor_group> after each feature play
Play N: Bootstrap cleanup          (targets: proxmox — remove temp networking)

Feature plays (post-baseline)

When a configure role grows beyond the initial baseline, new features are added as separate task files in the configure role (e.g., roles/<type>_configure/tasks/security.yml). Each feature gets a PAIR of plays in site.yml:

- name: Apply <feature> to <type>
  hosts: <dynamic_group>
  tags: [<type>-<feature>]
  gather_facts: false
  tasks:
    - name: Include <feature> tasks
      ansible.builtin.include_role:
        name: <type>_configure
        tasks_from: <feature>.yml

- name: Record <feature> deployment
  hosts: <flavor_group>
  tags: [<type>-<feature>]
  gather_facts: true
  roles:
    - role: deploy_stamp
      vars:
        deploy_play: <type>_<feature>

Feature plays run in the default (no-tags) E2E flow AND can be invoked individually via --tags. The paired deploy_stamp play runs on the Proxmox host (flavor group), not the VM, because vm_builds.fact lives on the host. NEVER use Ansible's never special tag — it silently skips plays in the default pipeline, creating untested dead code.

This pattern enables:

• Per-feature molecule scenarios that converge only the relevant task file
• Version-aware convergence via deploy_stamp
• Independent feature rollback via cleanup.yml tags

Device flavors (inventory groups)

Hosts belong to child groups under proxmox that control which VMs they receive:

proxmox:
  children:
    router_nodes:     # hosts that get OpenWrt
      hosts:
        home: {}
    service_nodes:    # hosts that get service VMs (future)
      hosts: {}

A host can belong to multiple flavor groups.

Step-by-step: adding a new VM type

Using homeassistant as the example:

1. Create the provision role

roles/homeassistant_vm/
├── defaults/main.yml
├── meta/main.yml
└── tasks/main.yml

tasks/main.yml must follow this skeleton:

---
- name: Check if HomeAssistant VM already exists
  ansible.builtin.command:
    cmd: qm status {{ homeassistant_vm_id }}
  register: vm_status
  failed_when: false
  changed_when: false

- name: Set VM existence flag
  ansible.builtin.set_fact:
    vm_exists: "{{ vm_status.rc == 0 }}"

# Upload image, create VM, import disk, attach NICs, start VM
# All creation tasks guarded with: when: not vm_exists | bool

# Auto-start (runs unconditionally to self-heal):
- name: Configure VM to start on boot
  ansible.builtin.command:
    cmd: >-
      qm set {{ homeassistant_vm_id }}
      --onboot 1
      --startup order={{ homeassistant_vm_startup_order }}

# Add to dynamic inventory:
- name: Add HomeAssistant VM to dynamic inventory
  ansible.builtin.add_host:
    name: "{{ homeassistant_vm_name }}"
    groups: homeassistant
    ansible_host: "<bootstrap_ip>"

2. Create the configure role

roles/homeassistant_configure/
├── defaults/main.yml
├── meta/main.yml
└── tasks/main.yml

3. Add VMID to group_vars

# inventory/group_vars/all.yml
homeassistant_vm_id: 200
homeassistant_vm_name: homeassistant
homeassistant_vm_memory: 2048
homeassistant_vm_cores: 2
homeassistant_vm_disk_size: 32G
homeassistant_image_path: images/haos.qcow2

4. Add dynamic group and flavor group to inventory

5. Extend site.yml with provision + configure plays

6. Update Molecule (add flavor group to platforms, add verify assertions)

7. Create VM-specific skill in .cursor/skills/<type>-build/SKILL.md

LXC container provisioning pattern

LXC containers use the shared proxmox_lxc role via include_role. Each service's <type>_lxc role is a thin wrapper:

# roles/pihole_lxc/tasks/main.yml
---
- name: Provision Pi-hole container
  ansible.builtin.include_role:
    name: proxmox_lxc
  vars:
    lxc_ct_id: "{{ pihole_ct_id }}"
    lxc_ct_hostname: pihole
    lxc_ct_dynamic_group: pihole
    lxc_ct_memory: 256
    lxc_ct_cores: 1
    lxc_ct_disk: "4"
    lxc_ct_onboot: true
    lxc_ct_startup_order: 3

The proxmox_lxc role handles: template upload, pct create, networking, device bind mounts, auto-start, container start, readiness wait, and add_host registration. For readiness: use ls / not hostname (BusyBox containers may lack it). For OpenWrt LXC: use --ostype unmanaged. See proxmox-safety rule.

LXC configure connection (API-first, verified 2026-04-18)

Configure roles use ansible.builtin.uri to push config via the NodeManager API over VPN. 9/10 configure roles are 100% API-first (zero SSH). The NM API provides endpoints for container exec, config push, and status queries.

# In site.yml — API-driven configure
- name: Configure Pi-hole
  hosts: dns_nodes
  gather_facts: false
  tasks:
    - name: Set Pi-hole password via NM API
      ansible.builtin.uri:
        url: "http://{{ vpn_ip }}:9001/api/container/pihole/exec"
        method: POST
        body_format: json
        body:
          command: "/usr/local/bin/pihole -a -p {{ pihole_password }}"
      delegate_to: localhost

Exceptions (SSH justified):

• kiosk_configure — deploys the NodeManager itself (bootstrap)
• openwrt_configure — OpenWrt VM has no HTTP API

NEVER add SSH fallback paths. If the NM API is unreachable, the 4-tier system is broken and must be fixed, not worked around.

Legacy pct_remote and add_host patterns remain in provisioning roles for hypervisor-side operations (pct config, pct status, pct create) that require host-level access.

For OpenWrt containers (no Python), use ansible.builtin.raw with commands wrapped in /bin/sh -c '...'. See the openwrt-build skill section "Shell syntax and PATH through pct_remote" for quoting rules.

LXC template management

Templates are stored locally in images/ and uploaded to the Proxmox host during provisioning. NEVER use pveam download — the host may not have internet access or may have unreachable enterprise repos.

See Image management section below for the full pattern.

Image management

ALL VM images and LXC templates are stored locally in the images/ directory (gitignored) and uploaded to the Proxmox host during provisioning. This is a deliberate design decision, not a convenience shortcut.

Why local images

1. No internet dependency on Proxmox host. The host may not have internet access (e.g., after cleanup destroys the router VM). Enterprise repos may be unreachable without a paid subscription. pveam download and wget both fail in this state.
2. Reproducibility. Pinning exact image versions locally ensures every build uses the same image. Remote repos can remove or rename versions.
3. Speed. Local uploads are faster than downloading from the internet, especially on slow WAN links.
4. Future self-hosting. Images can be served from a local NAS or HTTP server for multi-node deployments.

Previous bug: pveam download failed with 400 Parameter verification failed. template: no such template because the template name didn't match the available list. Switching to local hosting eliminated the dependency.

Directory layout

images/                                              (gitignored)
├── openwrt-router-24.10.0-x86-64-combined.img.gz   Router VM (build-images.sh --only router)
├── openwrt-mesh-lxc-24.10.0-x86-64-rootfs.tar.gz   Mesh LXC (build-images.sh --only mesh)
├── pihole-debian-12-*.tar.zst                       Pi-hole LXC template
├── wireguard-debian-12-*.tar.zst                    WireGuard LXC template
├── rsyslog-debian-12-*.tar.zst                      rsyslog LXC template
├── netdata-debian-12-*.tar.zst                      Netdata LXC template
├── homeassistant-debian-12-*.tar.zst                Home Assistant LXC template
├── jellyfin-debian-12-*.tar.zst                     Jellyfin LXC template
├── kodi-debian-12-*.tar.zst                         Kodi LXC template
├── moonlight-debian-12-*.tar.zst                    Moonlight LXC template
├── gaming-fedora-41-*.tar.zst                       Gaming LXC template (Fedora)
└── ...

NEVER commit images to git. The images/ directory is listed in .gitignore. Document the expected image filename and download URL in role defaults and in docs/architecture/.

Custom images via build-images.sh

build-images.sh builds pre-configured images for all services. OpenWrt images use the OpenWrt Image Builder; Debian/Fedora LXC templates are built via pct create + apt/dnf install + vzdump. Use --only <target> for selective rebuilds (~2 min each vs ~15 min for all).

Available targets: router, mesh, pihole, rsyslog, netdata, wireguard, homeassistant, jellyfin, kodi, moonlight, gaming, sunshine, desktop.

Per the project's "Bake, don't configure at runtime" principle (project-structure.mdc), custom images are REQUIRED. Provision roles verify the image exists and hard-fail with an actionable message if missing:

- name: Fail if image is missing
  ansible.builtin.fail:
    msg: "Image not found: {{ image_path }}. Run ./build-images.sh to build it."
  when: not (image_stat.stat.exists | default(false))

Image paths are defined in group_vars/all.yml. When adding a new service, define its image path there and add an existence check in the provision role.

Upload pattern for VMs

- name: Upload image to Proxmox
  ansible.builtin.copy:
    src: "{{ openwrt_image_path }}"
    dest: "/tmp/openwrt-upload"
    mode: "0644"
  when: not vm_exists | bool

- name: Decompress gzip image
  ansible.builtin.command:
    cmd: gunzip -f /tmp/openwrt-upload
  when:
    - not vm_exists | bool
    - openwrt_image_path.endswith('.gz')

# ... qm importdisk, then clean up /tmp file

Upload pattern for LXC templates

- name: Upload LXC template to Proxmox
  ansible.builtin.copy:
    src: "{{ role_path }}/../../{{ lxc_ct_template_path }}"
    dest: "/var/lib/vz/template/cache/{{ lxc_ct_template }}"
    mode: "0644"

ALWAYS use role_path for the source path. Relative paths like ../../images/... break when Molecule runs from non-default scenarios because the working directory is molecule/<scenario>/, not the project root.

Previous bug: proxmox_lxc template upload used a bare relative path. It worked from molecule/default/ but failed from molecule/proxmox-lxc/ with "Could not find or access". Fix was to use {{ role_path }}/../../.

Adding a new image

1. Download the image to images/ (use wget, browser, or pveam locally).
2. Add the path variable to group_vars/all.yml.
3. Reference the variable in the provision role's defaults and tasks.
4. Document the download URL and expected checksum in the role README or architecture doc.
5. Add the filename to Before running tests in the ansible-testing skill (e.g., "Verify LXC template exists: ls images/*.tar.zst").

Per-feature molecule scenarios for LXC

Each LXC service gets a molecule/<type>-lxc/ scenario that:

• Creates/destroys only the test container (VMID 999 for tests)
• Never touches the router VM or other services
• Runs in ~30-60 seconds (vs 4-5 min for full integration)
• Has no initial cleanup phase (assumes baseline exists)

Bridge allocation

The WAN bridge is auto-detected by proxmox_bridges via the host's default route (proxmox_wan_bridge fact). All physical-NIC-backed bridges are exported as proxmox_all_bridges.

Different VM/container types consume bridges differently:

• Router VMs (OpenWrt): ALL bridges — WAN on net0, remaining as LAN ports
• Service VMs: typically ONE LAN bridge — proxmox_all_bridges[1] (first non-WAN)
• LXC containers on LAN hosts (router_nodes, lan_hosts): proxmox_all_bridges[1] (LAN bridge)
• LXC containers on WAN hosts (ordinary Debian services): container NAT bridge vmbr_ct from proxmox_bridges — NOT proxmox_all_bridges[1] unless the role explicitly needs LAN topology; OpenWrt mesh/bridge LXC are exceptions (WAN/backhaul bridges per role)
• Isolated VMs: a dedicated bridge if network isolation is required

VMs that need internet during configure

If the configure role needs to download packages:

1. The VM must have a NIC on a bridge with upstream connectivity.
2. If the VM changes its own network topology mid-configure, use a multi-phase restart pattern (see VM-specific skill for details).
3. For VMs behind a router VM, WAN access comes through the LAN bridge — no special handling needed.

Host-level apt prerequisites

Roles that install packages on the Proxmox HOST (not inside VMs/containers) must handle three prerequisites:

1. Clock sync: Sync via NTP before apt-get update. GPG verification fails with "Not live until" when the clock is behind. ALWAYS check skew first with chronyc -n tracking | awk '/System time/{print $4}'. Only run the full burst+sleep+makestep sequence when skew > 30 seconds. The sync takes ~7s per host; on a 4-node cluster with 3 sync points, unconditional syncs waste ~84s when clocks are accurate.
2. DNS: After cleanup destroys the router VM, /etc/resolv.conf may point to a dead IP (10.10.10.1). Check DNS with getent hosts deb.debian.org and fall back to 8.8.8.8 / 1.1.1.1.
3. Enterprise repos: pve-enterprise.sources and ceph.sources require a subscription. Without it, apt-get update hangs. Rename both to .disabled and add the pve-no-subscription repo. ALWAYS restore them in cleanup.
   • NEVER use sed or Enabled: no in deb822 files — unreliable.
   • ALWAYS rename with mv (e.g., mv pve-enterprise.sources pve-enterprise.sources.disabled).
   • The proxmox_igpu role implements this pattern — reference it.
   • ALWAYS disable BOTH pve-enterprise.sources AND ceph.sources. Missing either one causes apt to hang.

Deployment tracking

The deploy_stamp role writes /etc/ansible/facts.d/vm_builds.fact on Proxmox hosts after each play. On subsequent runs with gather_facts: true, the data is available as ansible_local.vm_builds. Each play appends its entry without overwriting others.

Service config validation

Configure roles that deploy config files into LXC containers SHOULD validate the config before restarting the service. If the config is invalid, the service won't start and the container loses the service until the config is fixed.

Pattern: use an Ansible handler chain where validation runs before restart:

# handlers/main.yml
- name: Validate config
  ansible.builtin.command:
    cmd: <service> --check-config  # e.g., rsyslogd -N1, nginx -t
  listen: _restart_service
  changed_when: false

- name: Restart service
  ansible.builtin.command:
    cmd: systemctl restart <service>
  listen: _restart_service

Handlers with the same listen event run in definition order. If validation fails, the chain stops and the restart never executes.

After flush_handlers, add a health check with retries to confirm the service came up:

- name: Wait for service
  ansible.builtin.command:
    cmd: systemctl is-active <service>
  retries: 5
  delay: 2
  until: result.stdout | trim == 'active'

Previous bug: rsyslog 20-forward.conf deployment had no config validation. An invalid template would have crashed rsyslog on restart, killing log reception for all upstream senders.

Config file ordering for optional runtime configs

When baked image configs need to interoperate with optional runtime configs, use numbered filenames in /etc/<service>.d/ to control processing order:

10-base.conf       — module loads, template definitions (baked)
20-optional.conf   — runtime config deployed by configure role
50-routing.conf    — final routing/filtering (baked)

This pattern is needed when:

• The runtime config needs to intercept messages before the baked config processes them (e.g., forwarding before local storage)
• The baked config uses stop to prevent messages from falling through

Previous bug: rsyslog used a named ruleset for TCP-received messages. Messages in a named ruleset never enter the default ruleset, so the optional forwarding config (in the default ruleset) never saw remote messages. Fix: remove the named ruleset, use a property filter with stop at number 50, and deploy forwarding at number 20.

Diagnostics pattern

Every VM type SHOULD include diagnostic tasks at key milestones in its roles. These run on every build and provide debug context when things fail.

Standard diagnostic milestones for any VM:

1. Post-bootstrap (<type>_vm): VM status, bridge layout, bootstrap IP, dmesg errors
2. Post-configure (<type>_configure): Service status, network state, final config
3. Final report (<type>_configure): Summary of all configured parameters

Rules:

• changed_when: false and failed_when: false — diagnostics MUST NOT break the build
• Register output and display via debug: var: so it appears in logs
• Include dmesg checks — kernel errors are often the root cause when app-level symptoms mislead
• Include protocol-level checks — ICMP ping working does NOT mean TCP/HTTP works

When troubleshooting adds ad-hoc debug tasks, ALWAYS generalize and make them permanent before closing the issue.

Detached/async operations

When a configure role restarts services that sever the SSH connection, use detached scripts (see async-job-patterns rule). Key points:

• Launch returns success ≠ script completed successfully
• ALWAYS verify the expected outcome after the pause (wait_for, service check)
• NEVER restart services synchronously when the restart changes firewall/network rules affecting the current SSH path

Cleanup completeness

When a role deploys a file to the Proxmox host or the controller, ALWAYS add it to playbooks/cleanup.yml (unified cleanup). molecule/default/cleanup.yml is a one-line import of that playbook — do not duplicate removal lists.

Current managed files:

• /etc/network/interfaces.d/ansible-bridges.conf (may be modified in-place to inet dhcp)
• /etc/network/interfaces.d/ansible-proxmox-lan.conf (legacy, removed by converge if present)
• /etc/network/interfaces.d/ansible-temp-lan.conf (test workaround, cleaned up)
• /etc/modprobe.d/blacklist-wifi.conf
• /etc/modprobe.d/vfio-pci.conf
• /etc/ansible/facts.d/vm_builds.fact
• /tmp/openwrt.img (edge case: left behind if build fails mid-upload)
• /var/lib/vz/template/cache/*.tar.zst (LXC templates, uploaded by proxmox_lxc)
• .state/addresses.json (controller, via delegate_to: localhost)

Cleanup MUST destroy both project VMs and containers using explicit VMIDs from group_vars/all.yml. NEVER use blanket qm list / pct list iteration — it destroys non-project resources on shared hosts. Check existence with qm status / pct status before attempting stop + destroy. Current project VMIDs: see group_vars/all.yml and playbooks/cleanup.yml. When adding a new service, add its VMID to playbooks/cleanup.yml and tasks/cleanup_lan_host.yml.

Dynamic group reconstruction

Dynamic groups populated by add_host (e.g., openwrt, pihole) are ephemeral — they exist only within a single ansible-playbook invocation. Any entry point that runs as a separate invocation MUST reconstruct the group before targeting it.

Entry points that need reconstruction:

• Per-feature molecule/*/converge.yml
• Per-feature molecule/*/verify.yml
• playbooks/cleanup.yml (rollback plays)

ALWAYS extract group reconstruction into a reusable task file (e.g., tasks/reconstruct_<type>_group.yml) to avoid duplication. The task file must:

1. Verify the VM/container is running
2. Detect its IP from host-side state (bridge IPs, DHCP leases, etc.)
3. Detect the current auth method (may change after security hardening)
4. Register the host via add_host with the correct connection args

See the ansible-testing skill for the full pattern with auth detection.

Previous bug: per-feature verify assertions targeting the openwrt group ran with zero hosts — all assertions passed silently because Ansible skips plays when the host group is empty. The issue was only caught during plan review, not at runtime, because "0 assertions passed" looks the same as "all assertions passed" in Ansible output.

Rollback conventions

Every feature MUST define a rollback procedure. See .agents/skills/rollback-architecture/SKILL.md for full details. Summary:

• Per-feature rollback uses tags in cleanup.yml: --tags <feature>-rollback -e rollback=true
• Rollback plays are gated by when: rollback | default(false) | bool
• Each feature's project plan milestone includes inline rollback steps
• deploy_stamp records which features are applied; rollback checks this

# cleanup.yml pattern for per-feature rollback
- name: Rollback <feature>
  hosts: <target_group>
  tags: [<feature>-rollback]
  gather_facts: false
  tasks:
    - name: Revert changes
      # ... undo UCI, remove packages, restart services

Cross-cutting infrastructure ownership

Some infrastructure components are deployed by one service but consumed by multiple services. The deploying project OWNS the resource; other projects only ATTACH to it. This prevents duplicate deployment tasks and conflicting configurations.

Current cross-cutting components:

Rules:

• Only the owning project deploys the hookscript (via its provision role).
• Other projects add hookscript: <path> to their VM/container config.
• If the hookscript doesn't exist yet (owning project not deployed), the consumer's provision role MUST skip hookscript attachment with a warning, NOT deploy its own copy.
• Document the owning project explicitly in each consumer's project plan.

Separate hardware topology

Services that run on separate physical hardware (e.g., Gaming Rig on a dedicated machine) have unique characteristics:

• No OpenWrt router — the machine connects directly to the ISP router.
• Build profile may differ from the primary cluster.
• Testing requires the hardware to be physically available.
• Molecule scenarios should be conditional: skip when hardware unavailable rather than hard-fail.

Document the hardware topology separately in the project plan, including:

• Which machine, where it is, how it connects to the network.
• Which build profile it uses.
• How testing works when hardware is unavailable.

VA-API driver portability

Image builds for services that use the iGPU for hardware acceleration (Jellyfin, Kodi, Moonlight) SHOULD include BOTH Intel and AMD VA-API driver packages. At runtime, only the matching driver loads.

Intel: intel-media-va-driver + vainfo AMD: mesa-va-drivers + vainfo

This avoids rebuilding images when a container is moved to different hardware. The configure role reads igpu_vendor (exported by proxmox_igpu) to set LIBVA_DRIVER_NAME appropriately (iHD for Intel, radeonsi for AMD).

Hardware detection: hard-fail by default

NEVER add "graceful skip" for hardware expected on every host. Roles MUST hard-fail when required hardware is missing. The reasoning: silent skips mask fixable problems (BIOS settings, missing drivers) behind warnings that are easy to miss, wasting entire test cycles.

Previous bug: proxmox_pci_passthrough silently skipped passthrough when IOMMU groups were invalid. Root cause was VT-d disabled in BIOS — a 30-second fix masked for an entire test cycle.

Test strategy

• molecule/default/ is the full integration test (rebuilds everything from scratch across 6 nodes).
• Per-feature scenarios (molecule/<feature>/) test incremental changes on top of the baseline.
• Per-feature scenarios assume the baseline exists (all infrastructure + services running). They do NOT rebuild.
• Use molecule converge + molecule verify for day-to-day iteration (preserves baseline).
• Use molecule test only for clean-state validation (CI, pre-commit). It destroys the baseline.
• After molecule test, ALWAYS re-run molecule converge to restore the baseline before working on layered scenarios.
• Cleanup destroys known project VMs/containers by explicit VMIDs from group_vars/all.yml.
• Reproduce production bugs on the test machine first (test.env). Only involve production when the test machine cannot reproduce.
• Use TDD: write the verify assertion first, confirm it fails, implement the fix, confirm it passes.

LXC package management

• ALWAYS use install_recommends: false when installing packages in LXC containers. Many packages Recommend kernel-related metapackages that pull in 70+ MB kernel images, filling the small LXC disk.
• Broken apt in an LXC container means the baseline is wrong. The fix belongs in proxmox_lxc (shared provisioning), NOT in individual configure roles. See the clean-baselines rule.
• The shared proxmox_lxc role removes broken kernel packages from the dpkg database after creating a container. This ensures every container starts with working apt regardless of template issues.
• Previous bug: wireguard-tools Recommends wireguard metapackage which depends on linux-image-rt-amd64. Without install_recommends: false, apt pulled in a 70MB kernel image that filled the 1GB container disk (No space left on device).

Per-host IP indexing for multi-node LXC services

When a service deploys to multiple Proxmox hosts (e.g., rsyslog on all monitoring_nodes), each container needs a unique IP on the shared LAN subnet. Use the host's index in its flavor group: offset + groups['<flavor>'].index(inventory_hostname) together with tasks/lxc_lan_network.yml (LAN hosts).

WAN hosts: container addressing uses tasks/lxc_wan_or_lan_network.yml: per-host NAT bridge (10.99.{container_subnet_id}.{offset}) — no household subnet, no collision with host management IPs. Offsets are unique per service in group_vars/all.yml.

• Ansible sorts group members ALPHABETICALLY. The index depends on the full group composition, which can differ between per-feature and E2E scenarios.
• Spacing: indexed offset blocks MUST NOT overlap (see project-structure rules — kiosk / rsyslog / netdata spacing).
• Verify tasks MUST query the actual container IP from pct config instead of recomputing it. This avoids index drift between scenarios.

Pattern for querying actual IP in verify tasks:

- name: Get container IP from Proxmox config
  ansible.builtin.shell:
    cmd: |
      set -o pipefail
      pct config {{ ct_id }} | grep -oP 'ip=\K[^/,]+'
    executable: /bin/bash
  register: _ct_ip_query
  changed_when: false

Per-feature scenario group membership

Per-feature Molecule scenarios MUST include all groups that affect the role's branching logic. The role's when: conditions check group membership — if a group is missing, the wrong branch executes.

• Previous bug: rsyslog-lxc scenario had home in monitoring_nodes only. The LAN/WAN detection checked router_nodes membership, found it missing, took the WAN path, and failed because ansible_default_ipv4 was undefined (no gather_facts).
• Fix: add router_nodes to the per-feature scenario's platform groups for hosts that need the LAN path.
• This applies to ALL LXC roles with LAN/WAN branching, not just rsyslog.

Systemd sandboxing in LXC containers

Services installed from Debian packages (Netdata, rsyslog, etc.) often ship systemd unit files with sandboxing directives (LogNamespace, ProtectSystem, ProtectHome, ProtectControlGroups, BindReadOnlyPaths, RuntimeDirectory) that require mount namespace creation. Inside LXC containers, these fail with exit code 226 (NAMESPACE).

Fix (two-part):

1. Container feature: ALWAYS add nesting=1 to LXC containers running services with systemd sandboxing. Without it, unshare(CLONE_NEWNS) is forbidden and ANY namespace-requiring directive fails.

2. Systemd drop-in override: Even with nesting=1, some directives like LogNamespace still fail. Deploy a drop-in override in the configure role AND bake it into the image:

# /etc/systemd/system/<service>.service.d/lxc-override.conf
[Service]
LogNamespace=
ProtectSystem=false
ProtectHome=false
ProtectControlGroups=false
BindReadOnlyPaths=

Rules:

• NEVER use empty strings for boolean-like settings (ProtectSystem= doesn't work). Use ProtectSystem=false.
• For LogNamespace, empty string IS correct (it means "no namespace").
• ALWAYS bake the override into the image via build-images.sh. The configure role MUST NOT deploy it — the override is base system config (identical on every container), not host-specific topology. Deploying via pct_remote adds 3 slow tasks (mkdir + copy + daemon_reload) per container. On a 4-node cluster this adds ~3 minutes of pure overhead. Moving the override to the image build saved 38% of per-feature test time.

Previous bug: Netdata service exited 226/NAMESPACE in a privileged LXC container. Root cause: LogNamespace=netdata in the Netdata systemd unit created a journal namespace, which requires CLONE_NEWNS forbidden inside LXC even with nesting=1. Fix: systemd override clearing LogNamespace.

Host metrics via bind mounts

Monitoring agents (Netdata) that read host CPU, memory, disk, and temperature need access to the HOST's /proc and /sys, not the container's.

Requirements:

1. Privileged container (lxc_ct_unprivileged: false). Unprivileged containers use UID mapping that prevents reading host procfs files.
2. Nesting feature (features: nesting=1). Required for systemd sandboxing (see above).
3. Bind mount entries: Pass full Proxmox mount specs to proxmox_lxc:

   lxc_ct_mount_entries:
     - "/proc,mp=/host/proc,ro=1"
     - "/sys,mp=/host/sys,ro=1"
   

4. Image config: Bake /host/proc and /host/sys paths into the monitoring agent's config during image build. The agent reads host metrics, not container-local metrics.

Previous bug: 1GB disk was too small for the Netdata template (314MB compressed, 1013MB extracted). pct create failed mid-extraction. Fixed by increasing to 2GB.

LXC disk sizing

LXC templates compress well (~5:1 for Debian). ALWAYS verify that the rootfs disk is large enough for the EXTRACTED template, not just the compressed size.

Rule of thumb: Set disk to at least 2× the compressed template size, minimum 2 GB for any Debian-based container with monitoring or database services.

Previous bug: Netdata template was 314MB compressed but expanded to ~1013MB. 1GB rootfs caused pct create to fail mid-extraction (tar xpf exit code 2) with no clear error message. Fixed by increasing to 2GB.

Logrotate in LXC containers

When writing logrotate configs baked into LXC images, use root adm as the file ownership — NOT syslog adm. The syslog user may not exist in minimal container templates. The Debian community standard for /etc/logrotate.d/rsyslog uses root adm.

Previous bug: logrotate config with create 0640 syslog adm failed in the rsyslog container because the syslog user didn't exist in the Proxmox Debian 12 standard template. The chown fallback in build-images.sh (chown syslog:syslog || chown root:root) masked this — a graceful fallback that should have been a hard error.

Configure role performance (pct_remote overhead)

Each task in an LXC configure role opens a new paramiko SSH connection to the Proxmox host, then spawns pct exec inside the container. This overhead (15-60 seconds per task) makes LXC configure roles significantly slower than SSH-based configure roles.

Rules:

• MINIMIZE the number of tasks in LXC configure roles. Every task that can be baked into the image MUST be.
• Base system config (systemd overrides, default configs, package configs) is ALWAYS the same across all containers → belongs in the image.
• Host-specific config (IPs, streaming endpoints, peer keys, passwords) varies per container → belongs in the configure role.
• When in doubt, ask: "Does this value change between containers?" If no, bake it into the image.

**What belongs in the image

…(truncated)