Ansible Testing

Ansible Testing

MANDATORY: Test after every code change

ALWAYS run molecule test after modifying roles, playbooks, or molecule config. NEVER consider a task complete until the full test suite passes. When adding or changing behavior, ALWAYS update molecule/default/verify.yml with corresponding assertions before running the test.

Test-first reproduction

When a bug is reported against production, ALWAYS reproduce it on the test machine first. Replicate the production environment in test.env (same env vars, same image) and run molecule test. Only involve the production host when the test machine cannot reproduce the issue.

Previous bug: SSH timeout on production was reproduced by adding WAN_MAC to test.env. Four fix-and-verify cycles completed in 15 minutes without touching production.

TDD iteration pattern

1. Write or update the verify assertion in verify.yml first.
2. molecule test — the assertion should fail (proves the test catches the issue).
3. Implement the fix in the role.
4. molecule test — the assertion should now pass.
5. Update skills/rules with lessons learned.

Use molecule converge for rapid mid-fix iteration. Use molecule test for the final end-to-end proof from a clean state.

Quick start

source .venv/bin/activate
set -a; source test.env; set +a

molecule test          # full clean-state pipeline (destroys baseline)
molecule converge      # run playbook only (preserves baseline)
molecule verify        # run assertions only
molecule cleanup       # reset test host

Baseline workflow

The OpenWrt baseline on the primary host takes ~4 minutes to build. Layered scenarios (mesh1-infra, openwrt-security, etc.) and leaf nodes (mesh1) all depend on it. Prefer keeping the baseline running between test runs.

Day-to-day iteration:

molecule converge                  # build/update baseline (idempotent)
molecule verify                    # run assertions
molecule converge -s mesh1-infra   # run layered scenario (baseline must exist)
molecule verify -s mesh1-infra     # verify layered scenario

Clean-state validation (CI, pre-commit, final proof):

molecule test                      # full pipeline — baseline left running after verify

Molecule pipeline sequence

molecule test runs these phases in order:

1. dependency — install Galaxy requirements
2. cleanup — reset host from previous runs
3. syntax — ansible syntax check
4. prepare — start callhome API server, write .state/callhome_url
5. converge — run playbooks/site.yml
6. verify — run molecule/default/verify.yml

There is NO trailing cleanup or converge — the baseline is left running after verify. NEVER add cleanup or destroy to the end of the test_sequence. That tears down OpenWrt and makes mesh1 permanently unreachable. There is NO lint phase in the Molecule config. Run ansible-lint and yamllint separately.

Architecture

• Driver: default with managed: false (real Proxmox hardware, not Docker)
• Platforms: 6 nodes — home (primary), ai, mesh2, bridge-1, bridge-2 (directly reachable), mesh1 (LAN satellite via ProxyCommand)
• Platform groups: home gets proxmox + all primary flavor groups (including wifi_nodes); ai gets proxmox, vpn_nodes; mesh2 gets proxmox, vpn_nodes, wifi_nodes; mesh1 gets proxmox, lan_hosts, vpn_nodes, wifi_nodes
• Provisioner: playbooks/site.yml (phased: primary hosts → LAN bootstrap → services)
• Cleanup: two-play cleanup — proxmox:!lan_hosts for primary, router_nodes for LAN hosts via SSH
• Config: molecule/default/molecule.yml

6-node topology

ISP Router (192.168.86.x supernet)
  |
Switch
  |            |                  |
Home          AI Node          Mesh2
(primary)     192.168.86.220   192.168.86.211
  |
  |-- OpenWrt VM (10.10.10.1)
  |     |
  |     LAN bridge (10.10.10.x)
  |       |
  |     Mesh1 (10.10.10.210)

• home, ai, mesh2: directly reachable on the supernet (no ProxyCommand)
• mesh1: behind home's OpenWrt, reachable via ProxyCommand through home
• home, mesh1, ai, mesh2 are in vpn_nodes — WireGuard containers deploy on all 4 in parallel
• mesh1 and mesh2 are also in wifi_nodes — OpenWrt Mesh LXC deploys on both
• home is the only router_nodes member (runs OpenWrt)
• mesh1 is the only lan_hosts member (requires OpenWrt to be running)

Parallelism within plays

With 4 nodes in vpn_nodes, Ansible's linear strategy runs VPN tasks on all 4 hosts concurrently. Bridge nodes (bridge-1, bridge-2) run only WiFi bridge plays. No molecule-level parallelization is needed — the parallelism is automatic. Each host gets its own WireGuard container provisioned and configured simultaneously. Similarly, the wifi_nodes:!router_nodes play runs mesh LXC provisioning on mesh1 and mesh2 in parallel.

Phased site.yml

site.yml runs in three phases to respect host reachability dependencies:

1. Phase 1 (Primary hosts): proxmox:!lan_hosts — backup, infra, OpenWrt VM, OpenWrt configure
2. Phase 2 (LAN satellites): After OpenWrt creates the LAN, bootstrap LAN hosts from router_nodes, then run backup + infra on lan_hosts
3. Phase 3 (Services): Flavor groups that span both primary and LAN hosts (e.g., vpn_nodes includes home + mesh1) — runs in parallel across both hosts

This ordering is correct for both test and production. LAN hosts are only reachable after OpenWrt provisions the LAN bridge.

Baseline testing model

The baseline is the state after molecule/default converges successfully: all 15 services deployed across 6 nodes (home, mesh1, ai, mesh2, bridge-1, bridge-2), fleet heartbeat active, all hosts reachable. The core network baseline (OpenWrt router, bridges, PCI, iGPU) must succeed before any service plays run. All per-feature molecule scenarios start from this baseline and only converge/revert their own changes.

CRITICAL: The OpenWrt baseline stays up. Only tear down the specific containers being tested. Full molecule test is reserved for final validation only.

Scenario Hierarchy
├── molecule/default/              Full integration (home, mesh1, ai, mesh2, bridge-1, bridge-2 — 6-node)
│   ├── converge.yml               imports site.yml (phased: primary → LAN → services)
│   ├── verify.yml                 Multi-play: common infra, router, WireGuard, LAN host, mesh LXC
│   └── cleanup.yml               One-line import of playbooks/cleanup.yml (unified cleanup)
│
├── molecule/openwrt-security/     Per-feature (assumes baseline exists)
│   ├── converge.yml               runs only security plays via tags
│   ├── verify.yml                 security-specific assertions only
│   └── cleanup.yml                runs security rollback only
│
├── molecule/wireguard-lxc/        Per-feature (WireGuard standalone)
│   ├── converge.yml               WireGuard provision + configure
│   ├── verify.yml                 WireGuard-specific assertions
│   └── cleanup.yml                destroy container + unload module
│
├── molecule/mesh1-infra/          Lightweight infra-only on mesh1
│   └── ...                        (partially redundant with default, kept for quick iteration)
│
└── ...

Primary workflow: Per-feature scenarios are the main test loop:

1. molecule converge (once) — build the full baseline with all 6 nodes
2. molecule converge -s wireguard-lxc + molecule verify -s wireguard-lxc — iterate
3. Each per-feature scenario tears down only its own containers, verifies, then cleans up
4. Baseline (all infrastructure + services) is assumed to exist and left running

Final validation only: molecule test runs the full clean-state pipeline. The test_sequence ends at verify — there is NO trailing cleanup or converge. Use only before committing or for CI.

Full molecule test takes 4-5 minutes. Per-feature scenarios take 30-60 seconds. During development, iterate with per-feature scenarios. Run full integration before committing.

Two kinds of per-feature scenarios

There are two distinct patterns for per-feature scenarios:

1. Layered feature scenarios (e.g., openwrt-security, openwrt-vlans): assume the baseline exists (router VM running), converge only their tagged plays, verify only their assertions, clean up only their changes.

2. Standalone role scenarios (e.g., proxmox-lxc, proxmox-igpu): test a single shared infrastructure role in isolation. They converge the role, verify its output, and clean up any artifacts. They do NOT depend on the baseline — they can run against a bare Proxmox host.

Standalone scenarios are the right pattern for shared roles that run on the Proxmox host before any VMs are created. Layered scenarios are for features that build on top of existing VM/container state.

Previous learning: the proxmox-lxc and proxmox-igpu scenarios were developed independently of the default integration test. This allowed rapid iteration on iGPU driver/vendor issues (Intel and AMD; 4+ fix cycles in one session) without waiting for the full 4-minute default test each time.

Per-feature scenario setup

Each per-feature scenario needs its own molecule.yml that shares the platform config with default but uses a different test sequence:

# molecule/openwrt-security/molecule.yml
dependency:
  name: galaxy
  options:
    requirements-file: ${MOLECULE_PROJECT_DIRECTORY}/requirements.yml

driver:
  name: default
  options:
    managed: false

platforms:
  - name: home
    groups:
      - proxmox
      - router_nodes

provisioner:
  name: ansible
  config_options:
    defaults:
      roles_path: ${MOLECULE_PROJECT_DIRECTORY}/roles
      collections_paths: ${MOLECULE_PROJECT_DIRECTORY}/collections
      host_key_checking: false
      retry_files_enabled: false
  env:
    HOME_API_TOKEN: ${HOME_API_TOKEN}
    PRIMARY_HOST: ${PRIMARY_HOST}
    MESH_KEY: ${MESH_KEY}
  inventory:
    links:
      host_vars: ../../inventory/host_vars/
      group_vars: ../../inventory/group_vars/

verifier:
  name: ansible

scenario:
  test_sequence:
    - dependency
    - syntax
    - converge
    - verify
    - cleanup

No initial cleanup phase — the baseline must already exist. If it doesn't, converge will fail fast (can't SSH to OpenWrt) with a clear error.

Per-feature converge pattern

Per-feature converge playbooks run only their tagged plays. They MUST populate the openwrt dynamic group first since the baseline's add_host state doesn't persist across molecule runs.

The reconstruction logic MUST detect the current SSH auth method. After security hardening (M1), SSH uses key auth. Before M1 (or after rollback), SSH uses password auth with empty password. Extract this into a reusable task file to avoid duplication across converge, verify, and cleanup.

# molecule/openwrt-security/converge.yml
---
- name: Reconstruct openwrt dynamic group from baseline
  hosts: router_nodes
  gather_facts: true
  tasks:
    - name: Verify VM 100 is running
      ansible.builtin.command:
        cmd: qm status 100
      register: _vm_status
      changed_when: false
      failed_when: "'running' not in _vm_status.stdout"

    - name: Include reusable group reconstruction
      ansible.builtin.include_tasks: ../../tasks/reconstruct_openwrt_group.yml

- name: Apply security hardening
  hosts: openwrt
  gather_facts: false
  tasks:
    - name: Include security hardening tasks
      ansible.builtin.include_role:
        name: openwrt_configure
        tasks_from: security.yml

Reusable group reconstruction task file

Extract the group reconstruction logic into tasks/reconstruct_openwrt_group.yml at the project root. This file is consumed by:

• Per-feature converge.yml (molecule scenarios)
• Per-feature verify.yml (molecule scenarios)
• playbooks/cleanup.yml (rollback plays)

The task file MUST:

1. Detect the OpenWrt LAN IP from Proxmox bridge state
2. Detect whether key auth or password auth is active
3. Build the correct ansible_ssh_common_args for the detected auth method
4. Register the host via add_host with appropriate args

Auth detection heuristic:

• If OPENWRT_SSH_PRIVATE_KEY env var is set AND deploy_stamp shows security hardening was applied → use key auth
• Otherwise → use password auth (empty password, sshpass)

# tasks/reconstruct_openwrt_group.yml (simplified)
---
- name: Detect OpenWrt LAN IP from Proxmox LAN bridge
  ansible.builtin.shell:
    cmd: |
      set -o pipefail
      wan_br=$(ip -o route show default | awk '{print $5}' | head -1)
      for br in /sys/class/net/vmbr*/; do
        brname=$(basename "$br")
        [ "$brname" = "$wan_br" ] && continue
        ip -o -4 addr show dev "$brname" | awk '{print $4}' | cut -d/ -f1 | head -1
        break
      done
    executable: /bin/bash
  register: _proxmox_lan_ip
  changed_when: false

- name: Compute OpenWrt LAN IP
  ansible.builtin.set_fact:
    _openwrt_ip: >-
      {{ _proxmox_lan_ip.stdout.split('.')[0:3] | join('.') }}.1

- name: Detect SSH auth method
  ansible.builtin.set_fact:
    _use_key_auth: >-
      {{ (lookup('env', 'OPENWRT_SSH_PRIVATE_KEY') | length > 0) and
         (ansible_local.vm_builds.plays.openwrt_security is defined) }}

- name: Add OpenWrt to dynamic group
  ansible.builtin.add_host:
    name: openwrt-router
    groups: openwrt
    ansible_host: "{{ _openwrt_ip }}"
    ansible_user: root
    ansible_ssh_common_args: >-
      -o ProxyJump=root@{{ ansible_host }}
      -o StrictHostKeyChecking=no
      -o UserKnownHostsFile=/dev/null
      -o ConnectTimeout=10
      -o ServerAliveInterval=15
      -o ServerAliveCountMax=4
      {% if _use_key_auth %}
      -i {{ lookup('env', 'OPENWRT_SSH_PRIVATE_KEY') }}
      {% endif %}

Per-feature verify pattern

Per-feature verify playbooks also run as a separate ansible-playbook invocation, so the dynamic group is empty. The verify MUST reconstruct the group before running assertions that target the VM.

# molecule/openwrt-security/verify.yml
---
- name: Reconstruct openwrt dynamic group
  hosts: router_nodes
  gather_facts: true
  tasks:
    - name: Include reusable group reconstruction
      ansible.builtin.include_tasks: ../../tasks/reconstruct_openwrt_group.yml

- name: Verify security hardening
  hosts: openwrt
  gather_facts: false
  tasks:
    - name: Check banIP is installed
      ansible.builtin.raw: opkg list-installed | grep banip
      register: _banip
      changed_when: false
    - name: Assert banIP installed
      ansible.builtin.assert:
        that: _banip.rc == 0

Previous bug: per-feature verify assertions targeting the openwrt group ran with zero hosts, silently passing all assertions. The group was empty because add_host from converge doesn't persist into verify.

Per-feature cleanup pattern

Per-feature cleanup runs only the feature's rollback tag. The cleanup.yml playbook includes its own group reconstruction play (tagged with all rollback tags), so no reconstruction is needed here:

# molecule/openwrt-security/cleanup.yml
---
- name: Rollback security hardening
  ansible.builtin.import_playbook: ../../playbooks/cleanup.yml
  tags: [openwrt-security-rollback]

Running per-feature tests

# Establish baseline (once, or when baseline is stale)
molecule converge

# Iterate on a feature
molecule test -s openwrt-security     # converge + verify + cleanup
molecule converge -s openwrt-security # apply only (for debugging)
molecule verify -s openwrt-security   # check only (after manual fixes)

# Full integration (before commit)
molecule test

Standalone role scenario setup

Standalone scenarios test a single role without any baseline dependency. Use these for shared infrastructure roles (proxmox_lxc, proxmox_igpu, future shared roles). They converge the role directly, verify facts and artifacts, then clean up host state changes.

# molecule/proxmox-igpu/converge.yml
---
- name: Test proxmox_igpu role
  hosts: proxmox
  gather_facts: true
  roles:
    - proxmox_igpu

Key differences from layered scenarios:

1. No baseline dependency — no need to populate dynamic groups first.
2. Cleanup restores host state — enterprise repos, config files, etc. The cleanup is role-specific, not a full cleanup.yml --tags call.
3. VMID 999 for throwaway resources — standalone LXC tests use VMID 999 to avoid collisions with real service VMIDs.
4. Include cleanup in test_sequence — standalone scenarios SHOULD include cleanup at the end (unlike layered scenarios which inherit it).

Fact scoping across Molecule phases

Facts set during converge are NOT available in verify. Molecule runs converge and verify as separate Ansible invocations with independent fact caches.

Impact: If a role exports facts via set_fact (e.g., proxmox_igpu exports igpu_available, igpu_render_device, etc.), those facts will be undefined in verify.yml.

Fix: For read-only roles, re-include the role in verify:

# molecule/proxmox-igpu/verify.yml
---
- name: Verify proxmox_igpu role
  hosts: proxmox
  gather_facts: true
  roles:
    - proxmox_igpu    # re-run to populate facts (idempotent)
  tasks:
    - name: Assert igpu_available fact is set
      ansible.builtin.assert:
        that: igpu_available is defined

For roles with side effects, use shell commands in verify to check state directly instead of relying on Ansible facts.

Previous bug: proxmox-igpu verify failed with "igpu_available is not defined" because the fact was only set during converge. Re-including the role (which is idempotent) fixed it.

Molecule file path resolution

role_path and relative paths behave differently depending on which scenario is running. The working directory during a Molecule run is the scenario directory (molecule/<scenario>/), not the project root.

Impact: A task referencing ../../images/template.tar.zst works from molecule/default/ but breaks from molecule/proxmox-lxc/ because the relative path resolves differently.

Fix: ALWAYS use role_path or MOLECULE_PROJECT_DIRECTORY for paths that need to resolve relative to the project root:

# GOOD — resolves correctly regardless of which scenario runs
src: "{{ role_path }}/../../{{ lxc_ct_template_path }}"

# BAD — breaks when called from non-default scenarios
src: "../../images/{{ lxc_ct_template }}"

Previous bug: proxmox_lxc template upload failed with "Could not find or access" when run from the proxmox-lxc scenario because the relative path resolved from molecule/proxmox-lxc/ instead of the project root.

Before running tests

1. Source test env: set -a; source test.env; set +a
2. Verify access: curl -sf http://localhost:$WEBUI_PORT/api/fleet/health (if base state up) or ansible home -m ping (first run)
3. Build custom images (required): ./build-images.sh
4. Verify images exist: ls images/openwrt-router-*.img.gz images/openwrt-mesh-lxc-*-rootfs.tar.gz images/debian-*.tar.zst
5. If previous run left host in bad state, power-cycle the machine

Molecule platform groups

The molecule platform config MUST include all flavor groups that any play in site.yml targets. When a new flavor group is added to inventory/hosts.yml, it MUST also be added to molecule/default/molecule.yml:

platforms:
  - name: home
    groups:
      - proxmox
      - router_nodes
      - vpn_nodes
      - dns_nodes
      - wifi_nodes
      - monitoring_nodes
      - service_nodes
      - media_nodes
      - desktop_nodes
  - name: mesh1
    groups:
      - proxmox
      - lan_hosts
      - vpn_nodes
      - wifi_nodes
  - name: ai
    groups:
      - proxmox
      - vpn_nodes
  - name: mesh2
    groups:
      - proxmox
      - vpn_nodes
      - wifi_nodes

All 4 hosts are tested in the default scenario. home runs all services. ai and mesh2 are directly reachable on the supernet (no ProxyCommand). mesh1 is behind OpenWrt (requires ProxyCommand). All 4 are in vpn_nodes — WireGuard deploys on all 4 in parallel within the same play.

Without this, provision plays targeting flavor groups will skip hosts.

Cleanup requirements for repeatable runs

The cleanup playbook MUST restore the host to a clean state using service-specific cleanup — destroy only known project VMs and containers by explicit VMID from group_vars/all.yml.

1. Destroy project VMs by explicit VMID (check existence first with qm status, then stop + destroy). Current VMIDs: OpenWrt (100).
2. Destroy project containers by explicit VMID (check with pct status, then stop + destroy). Current VMIDs: WireGuard (101), Pi-hole (102), Mesh WiFi (103), WiFi Bridge (104), Home Assistant (200), Jellyfin (300), Kodi (301), Moonlight (302), Kiosk (401), Netdata (500), rsyslog (501), Gaming LXC (601).
3. Unbind all devices from vfio-pci. Without this, WiFi hardware is invisible.
4. Remove modprobe blacklist files (/etc/modprobe.d/blacklist-wifi.conf, /etc/modprobe.d/vfio-pci.conf).
5. Rebind WiFi PCI devices via sysfs: tasks/sysfs_wifi_rebind.yml (NEVER modprobe -r).
6. Rescan PCI bus: echo 1 > /sys/bus/pci/rescan.
7. Tear down stale bridges (skip vmbr0 management bridge).
8. ifup --all --force to restore interfaces.

Steps 3-6 are critical. Without them, the next run cannot detect WiFi hardware.

NEVER use blanket qm list / pct list iteration to destroy VMs and containers. This destroys non-project resources on shared hosts, is slower than explicit VMIDs, and provides no benefit. Use explicit VMIDs defined in group_vars/all.yml. Check existence with qm status / pct status before attempting stop + destroy.

Previous bug: molecule cleanup used qm list | awk and pct list | awk to discover and destroy ALL VMs/containers. This was unsafe on shared test hosts and forced a full rebuild of every resource on every test run.

NEVER delete LXC templates from /var/lib/vz/template/cache/ in molecule cleanup. The proxmox_lxc role checks pveam list and skips upload when the template is already cached. Deleting templates forces re-upload of ~820MB across 4 hosts on every run. Template deletion is only appropriate in playbooks/cleanup.yml behind [full-restore, clean] tags.

Previous bug: molecule cleanup deleted all cached templates. Each subsequent molecule test re-uploaded pihole (205MB), rsyslog (143MB), netdata (315MB), wireguard (143MB), and openwrt-mesh (14MB) to every host. Removing the template deletion saved ~7 minutes on a 6-node test run.

NEVER restore the host config from backup archive in molecule cleanup. The explicit file removal tasks already remove all ansible-managed files. The backup restore is redundant and adds ~15s per host. Backup restore is only appropriate in playbooks/cleanup.yml behind [full-restore] tag.

Make update-initramfs conditional on PCI passthrough config having been present. Check stat on /etc/modprobe.d/vfio-pci.conf before removing it. Only run update-initramfs when the file existed. This saves ~20s per host when PCI passthrough wasn't configured.

Verify completeness requirements

Every configured feature MUST have a corresponding assertion in verify.yml. "Is it running?" is not enough. A router VM that is running but has no DHCP or has a colliding subnet is a production outage.

Minimum assertion categories per VM type:

Previous bug: DHCP was configured but never verified. A broken DHCP config would pass all tests and only be caught when clients couldn't get addresses.

Extending verify for new VM types

Add assertions to molecule/default/verify.yml per VM type. Verify from the Proxmox host using qm, sshpass, or SSH ProxyJump. Avoid running Ansible directly against VMs in verify — use raw commands instead.

- name: Verify HomeAssistant VM
  hosts: proxmox
  gather_facts: false
  tasks:
    - name: Check VM is running
      ansible.builtin.command:
        cmd: qm status {{ homeassistant_vm_id }}
      register: ha_status
      changed_when: false

    - name: Assert VM is running
      ansible.builtin.assert:
        that: "'running' in ha_status.stdout"
        fail_msg: "HomeAssistant VM is not running"

LXC container verify checklist

LXC services need deeper verification than "container is running." The following categories should be tested for every LXC service:

The no-leak test is critical for services that use rsyslog's stop directive to prevent remote messages from polluting local logs. Without it, the stop directive could silently break and double-log everything.

Previous bug: rsyslog used a named ruleset for TCP input. Messages in the named ruleset never reached the default ruleset, so forwarding config deployed at number 20 never saw remote messages. The no-leak test catches this class of routing error.

Hard-fail over graceful degradation

NEVER add "graceful skip" logic for hardware expected on every host. Silent skips mask fixable problems (wrong BIOS settings, missing drivers).

• iGPU: REQUIRED. proxmox_igpu hard-fails if absent. Supports both Intel (i915) and AMD (amdgpu). Never assume Intel-only.
• WiFi + VT-d/IOMMU: REQUIRED for passthrough. proxmox_pci_passthrough hard-fails if IOMMU is not active or groups are invalid.
• NIC count: OK to handle dynamically — hardware legitimately varies.
• Previous bug: graceful skip of IOMMU masked a disabled VT-d BIOS setting on mesh1 for an entire test cycle. A 30-second BIOS fix was hidden behind "WARNING: skipping passthrough."

Cleanup completeness

CRITICAL — cleanup must NEVER destroy access credentials:

• NEVER remove /root/.ssh/authorized_keys from any host. SSH keys are operator prerequisites. Removing them permanently locks out remote nodes — potentially thousands of miles away with no physical console access.
• NEVER remove Proxmox API tokens (pveum user token remove). API tokens are operator-created, stored in .env, and required for all Ansible runs.
• NEVER remove ANY credential the operator created manually. Apply this test to EVERY cleanup task: "Did the converge/playbook create this?" If no → do not touch it.
• Previous bug: mesh1-infra cleanup removed both authorized_keys and the API token from a LAN satellite node, permanently locking out all SSH and API access.

When a role writes a new file to the Proxmox host, ALWAYS add it to the removal list in:

• playbooks/cleanup.yml (unified cleanup — primary hosts)
• tasks/cleanup_lan_host.yml (shared LAN host cleanup — included by unified cleanup)

molecule/default/cleanup.yml is a one-line import of playbooks/cleanup.yml — there is only ONE cleanup to maintain.

When a role writes a local state file (e.g., .state/addresses.json), ALWAYS add a delegate_to: localhost cleanup task to remove it.

Current managed files:

• Host config: ansible-bridges.conf, ansible-proxmox-lan.conf (legacy), ansible-temp-lan.conf (test workaround)
• Module config: blacklist-wifi.conf, vfio-pci.conf, wireguard.conf
• Apt repos: pve-no-subscription.sources (added by igpu), enterprise repos (renamed to .disabled, restored on cleanup)
• VM images: /tmp/openwrt-upload* (temporary during import)
• Templates: /var/lib/vz/template/cache/* — ONLY in playbooks/cleanup.yml behind [full-restore, clean] tags. NEVER in molecule cleanup (see above).
• Hookscripts: /var/lib/vz/snippets/mesh-wifi-phy-*.sh
• Facts: vm_builds.fact
• Local: .state/addresses.json, .env.generated, test.env.generated

Previous bug: ansible-proxmox-lan.conf was deployed by openwrt_configure but not removed by cleanup, causing stale LAN management IPs on subsequent test runs.

Test performance optimization

The full 6-node molecule test takes ~13-14 minutes. Most time goes to template uploads, NTP sync, pct_remote overhead, and SSH round trips in verify. Apply these rules to avoid wasting time:

Template caching

• NEVER delete templates in molecule cleanup. Keep them cached on Proxmox hosts. The proxmox_lxc role's pveam list check skips upload when cached. Template deletion forces re-upload of ~820MB across 4 hosts.
• Template deletion is only valid in production cleanup behind [full-restore, clean] tags.

NTP sync

• ALWAYS check clock skew BEFORE running the full NTP burst+sleep+makestep sequence. Use chronyc -n tracking | awk '/System time/{print $4}' to get skew in seconds. Only sync when skew > 30s.
• The NTP sync takes ~7s per host (sleep 6 + burst). With 4 hosts × 3 sync points (site.yml primary, site.yml LAN, proxmox_igpu), that's 84s wasted when clocks are accurate.

pct_remote task count

• Each pct_remote task opens a new paramiko SSH connection → pct exec pipeline. Each task takes 15-60 seconds depending on payload.
• MINIMIZE the number of tasks in configure roles. Base system config that is identical across all containers belongs in the image, NOT the configure role.
• Previous bug: netdata_configure deployed a systemd override (3 tasks: mkdir, copy, daemon_reload) via pct_remote. With 4 containers, this added ~12 minutes. Moving the override to the image saved 38% of the per-feature test time.

apt cache

• Set cache_valid_time: 86400 (24h) for apt tasks, not 3600 (1h). Test machines rarely have stale packages. The shorter interval triggers apt-get update on every run when the cache expires between test cycles.

Selective image rebuilds

• Use ./build-images.sh --host <ip> --only <target> to rebuild a single image (mesh, router, pihole, rsyslog, netdata, wireguard). Full rebuilds take ~15 min; selective rebuilds take ~2-3 min.
• Every service MUST have a custom image with ALL packages baked in. WireGuard was the last service converted (from stock Debian template to custom image). ZERO configure roles should install packages at runtime.

Verify phase: consolidate pct config reads

• NEVER call pct config <id> multiple times for the same container. Each call is an SSH round trip (~1-2s). Read the full config once, register it, then assert against the registered output using Jinja filters.
• Pattern: ansible.builtin.command: cmd: pct config {{ ct_id }} → register as _ct_cfg → assert "'onboot: 1' in _ct_cfg.stdout", _ct_cfg.stdout is regex('startup:.*order=3'), etc.
• For IP extraction from cached config: use {{ _ct_cfg.stdout | regex_search('ip=([^/,]+)', '\\1') | first }}.
• Previous bug: verify.yml had 20 individual pct config calls (3-6 per container). Consolidating to 6 (one per container type) eliminated 14 SSH round trips per host × 4 hosts = 56 unnecessary SSH connections.

Verify phase: batch pct exec calls per container

• NEVER run multiple individual pct exec calls against the same container when the checks are independent. Batch them into a single pct exec with /bin/sh -c '...' and key=value output for easy parsing.
• Pattern: combine health checks into one shell script that outputs KEY=value lines. Parse with 'KEY=value' in result.stdout or result.stdout | regex_search('KEY=(\d+)', '\\1') | first.
• Example (WireGuard — 7 checks → 1 pct exec):

  - name: Run WireGuard container health checks (single pct exec)
    ansible.builtin.shell:
      cmd: >-
        pct exec {{ ct_id }} -- /bin/sh -c '
        echo "IFACE=$(ip link show wg0 >/dev/null 2>&1 && echo ok || echo missing)";
        echo "SVC=$(systemctl is-enabled wg-quick@wg0 2>/dev/null || echo unknown)";
        echo "NAT=$(iptables -t nat -C POSTROUTING -o wg0 -j MASQUERADE 2>/dev/null && echo present || echo missing)"
        '
    register: wg_health
    changed_when: false
  
  - name: Assert wg0 exists
    ansible.builtin.assert:
      that: "'IFACE=ok' in wg_health.stdout"
  

• Keep tasks with retry logic (e.g., curl with retries/until) as separate pct exec calls since they need per-attempt control.
• Keep timing-dependent tests (log reception after pause) as separate calls since they depend on wall-clock ordering.
• Previous savings: batching eliminated ~43 individual SSH calls in the default verify, saving ~60-80 seconds across 4 hosts.

Verify phase: fleet API as primary (no SSH fallback)

• For container liveness checks (service running, health status), use the fleet readiness API (/api/fleet/ready, /api/container/{id}/ready) as the ONLY path. If the API is unavailable, the 4-tier heartbeat system is broken — fix it, don't fall back to SSH.
• The _fleet_api_ready fact confirms the API is up. If it's not, the heartbeat system is broken — fail and fix, don't fall back to SSH.
• New services added to verify MUST use the fleet API. See the manager-api-pattern skill for the full pattern.
• SSH-only operations (hypervisor config, L3 integration, QEMU GA, OpenWrt deep checks) remain unchanged — the fleet API does not replace these.

Verify phase: merge plays with same hosts target

• When two verify plays target the same hosts: group with the same gather_facts: setting, merge them into one play. Each play has startup overhead (SSH connection setup, fact gathering if enabled, variable resolution).
• Example: rsyslog and Netdata both target monitoring_nodes with gather_facts: false. Merging them into one play eliminates one play startup per host.

Wait/pause tuning

• OpenWrt detached restart scripts take ~18s (8s explicit sleep + ~10s service restarts). The post-script pause should be 20s, not 30s.
• OpenWrt VM first boot SSH: delay: 10, timeout: 120 (not 15/180). Typical first boot is 30-60s.
• LXC container networking: delay: 3 (not 4). DHCP usually completes on first probe.
• WiFi PHY detection: delay: 3 (not 5). PHY appears within seconds or indicates a real problem.
• Apt retry delay: delay: 10 (not 15). Transient apt failures recover quickly.
• Verify-phase SSH waits: reduce delay and timeout for services that are already confirmed running from converge. The verify wait is a safety check, not a first-boot wait.

Common failures

Ansible syntax pitfalls with raw: heredocs

When using ansible.builtin.raw: | with shell heredocs (e.g., cat << 'EOF'), the Ansible argument parser may fail on content that looks like Jinja2:

• `$

…(truncated)