Molecule Verify

Molecule Verify

Rules

1. Every configured feature MUST have assertion in verify.yml.
2. Assert IP non-empty and no collision BEFORE functional tests.
3. SINGLE pct config call per container, then assert on cached output.
4. Use ansible.builtin.uri → NM/SM API for runtime status checks. pct exec is ONLY for hypervisor-side operations that have no API equivalent.
5. NEVER use failed_when: false on client sends — let errors surface.
6. Per-feature verify MUST reconstruct dynamic groups (separate invocation).
7. Use the fleet readiness API as the sole path for container liveness: /api/fleet/ready, /api/container/{id}/ready, /api/config/self. NEVER add SSH fallback paths — if the API is unreachable, the 4-tier system is broken and must be fixed.
8. Hard-failure messages MUST include INVESTIGATE prompts with specific diagnostic steps (curl commands, journalctl checks, relay chain tracing).
9. All API calls in verify use vpn_ip for NM communication (VPN-only post-bootstrap). NEVER use ansible_host for API queries.

Completeness requirements

Per VM:

Per LXC:

IP validation

- name: Read container config
  ansible.builtin.command:
    cmd: pct config {{ ct_id }}
  register: _ct_cfg
  changed_when: false

- name: Extract IP
  ansible.builtin.set_fact:
    _ct_ip: >-
      {{ _ct_cfg.stdout | regex_search('ip=([^/,]+)', '\1') | first }}

- name: Assert IP valid
  ansible.builtin.assert:
    that:
      - _ct_ip is defined
      - _ct_ip | length > 0

Batch pct config

ansible.builtin.command:
  cmd: pct config {{ ct_id }}
register: _ct_cfg

- name: Assert onboot
  ansible.builtin.assert:
    that: "'onboot: 1' in _ct_cfg.stdout"

Batch pct exec

ansible.builtin.shell:
  cmd: >-
    pct exec {{ ct_id }} -- /bin/sh -c '
    echo "IFACE=$(ip link show wg0 >/dev/null 2>&1 && echo ok || echo missing)";
    echo "SVC=$(systemctl is-enabled wg-quick@wg0 2>/dev/null || echo unknown)"
    '
  register: health
  changed_when: false

Client sends

# Let errors surface immediately
ansible.builtin.shell:
  cmd: logger --tcp {{ ct_ip }} 514 --server.host test

# Reserve failed_when: false for commands whose rc is checked by assertion

Service management

Prefer ansible.builtin.systemd over command: systemctl:

ansible.builtin.systemd:
  name: rsyslog
  state: restarted

Multi-node patterns

For 4-node tests, avoid IP recomputation. Read config once:

ansible.builtin.command:
  cmd: pct config {{ ct_id }}
register: _ct_cfg

ansible.builtin.set_fact:
  _ct_ip: >-
    {{ _ct_cfg.stdout | regex_search('ip=([^/,]+)', '\1') | first }}

Fact scoping

Facts from converge NOT available in verify. For roles with facts, re-include:

- name: Verify proxmox_igpu role
  hosts: proxmox
  gather_facts: true
  roles:
    - proxmox_igpu  # re-run to populate facts

No-leak testing

Verify remote messages don't pollute local logs:

- name: Send from remote
  ansible.builtin.shell:
    cmd: logger --tcp {{ remote_ip }} 514 "TEST-LEAK-{{ ansible_date_time.epoch }}"

- name: Verify no leak
  ansible.builtin.shell:
    cmd: grep "TEST-LEAK-{{ ansible_date_time.epoch }}" /var/log/syslog
  register: _leak
  failed_when: _leak.rc == 0

Verify task conventions

1. Fail-fast IP validation. Assert IP non-empty and no collision with host IP BEFORE functional tests. "IP is empty" is clear; "log not found" 10 tasks later is not.

2. No failed_when: false on client sends. Let logger --tcp, dig, curl fail immediately on connection error. "Connection refused" is far more useful than downstream "expected output not found."

3. set -o pipefail only on pipelines. NEVER add pipefail to single-command tasks. It adds noise and obscures the convention that pipefail signals a pipeline.

4. Use ansible.builtin.systemd over command: systemctl for restarts. Use command only for status checks and validation.

Jinja regex_search in assert

regex_search without a capture group returns a STRING match (not boolean). In assert: that: blocks, ALWAYS append is not none:

# BAD — returns string, assert sees "Conditional was derived from type str"
- _output | regex_search('RADIOS=phy')

# GOOD — explicit boolean
- _output | regex_search('RADIOS=phy') is not none

grep -c with find -exec in containers

When using find ... -exec grep -c 'pattern' {} + inside a container (via pct exec), grep -c outputs filename:count when multiple files match, but just count for a single file. Jinja2's | int filter returns 0 for filename:count strings, silently breaking assertions.

ALWAYS use grep -rch with awk sum instead:

# BAD — breaks when multiple files match
pct exec {{ ct_id }} -- find /var/log/remote/ -name '*.log' -exec grep -c 'pattern' {} + 2>/dev/null | head -1

# GOOD — numeric output regardless of file count
pct exec {{ ct_id }} -- grep -rch 'pattern' /var/log/remote/ 2>/dev/null | awk '{sum+=$1} END {print sum+0}'

Previous bug: rsyslog separation test used find -exec grep -c. On mesh1, multiple log files existed under different hostname directories. grep -c output /var/log/remote/home/e2e_daemon_a.log:11. Jinja2 | int converted this to 0, failing the assertion.

Host-to-container TCP fallback pattern

When verifying TCP-based services in containers, host-to-container TCP may fail on specific hosts (e.g., mesh1) even when ICMP ping succeeds and br_netfilter is disabled. Add a fallback path:

- name: Send log via host TCP
  ansible.builtin.shell:
    cmd: logger --tcp -n {{ container_ip }} -P 514 "test message"
  register: _send
  failed_when: false
  retries: 5
  delay: 5
  until: _send.rc == 0

- name: Diagnose TCP failure
  ansible.builtin.shell:
    cmd: |
      echo "=== nft ruleset ==="
      nft list ruleset 2>/dev/null | head -40
      echo "=== raw TCP ==="
      timeout 3 bash -c "echo test > /dev/tcp/{{ container_ip }}/514" 2>&1 && echo "ok" || echo "fail"
    executable: /bin/bash
  when: _send.rc != 0

- name: Fallback via pct exec
  ansible.builtin.shell:
    cmd: >-
      pct exec {{ ct_id }} --
      logger --tcp -n 127.0.0.1 -P 514 "test message"
  when: _send.rc != 0

Previous bug: mesh1 host consistently failed logger --tcp to its rsyslog container (10.10.10.14) despite ping succeeding and br_netfilter disabled. Root cause unresolved (likely nftables or conntrack). Fallback via pct exec with localhost TCP verifies rsyslog works without host-level network dependency.

Windows VM verification via QEMU Guest Agent

For Windows VMs without reliable SSH (no sshpass, LAN-only IPs), use qm guest exec instead of SSH connectivity tests:

- name: Test in-VM command via Guest Agent
  ansible.builtin.command:
    cmd: qm guest exec {{ vm_id }} --timeout 15 -- cmd /c "echo vm-ok"
  register: _exec_test
  changed_when: false
  failed_when: false

- name: Assert in-VM command works
  ansible.builtin.assert:
    that: "'vm-ok' in _exec_test.stdout"

For per-host VMID scenarios, compute the effective VMID at the start of verify:

- name: Set per-host VM ID
  ansible.builtin.set_fact:
    gaming_vm_id: "{{ (gaming_vm_id | int) + groups['gaming_nodes'].index(inventory_hostname) }}"

Previous bug: verify used sshpass for SSH testing. sshpass wasn't installed on mesh1. Switched to Guest Agent which bypasses network entirely.

DNS service verification (Pi-hole FTL)

ALWAYS check BOTH TCP and UDP listeners for DNS services. dig defaults to UDP. ss -tlnp only checks TCP — a service can bind TCP:53 but miss UDP:53 due to startup race or partial initialization.

- name: Wait for DNS on TCP+UDP port 53
  ansible.builtin.shell:
    cmd: >-
      pct exec {{ ct_id }} -- /bin/sh -c '
      tcp=$(ss -tlnp 2>/dev/null | grep -c ":53 ");
      udp=$(ss -ulnp 2>/dev/null | grep -c ":53 ");
      [ "$tcp" -gt 0 ] && [ "$udp" -gt 0 ] && echo BOTH || echo MISSING
      '
  retries: 6
  delay: 5
  until: "'BOTH' in _ftl_listen.stdout"

- name: Restart FTL if listeners are incomplete
  ansible.builtin.command:
    cmd: >-
      pct exec {{ ct_id }} -- /bin/sh -c
      'systemctl restart pihole-FTL && sleep 3'
  when: "'BOTH' not in _ftl_listen.stdout"

ALWAYS wrap DNS resolution tests in block/rescue. The retries/until pattern makes a task fatal when all retries expire — any diagnostic tasks after it never execute. Use block: for the test and rescue: for diagnostics + recovery (FTL restart + retry).

ALWAYS add a network connectivity pre-check before DNS tests: verify the container can ping its gateway and an external IP. This distinguishes "FTL is broken" from "container has no network."

Previous bug: Pi-hole DNS verify timed out in E2E tests. ss -tlnp | grep :53 passed (TCP listener present) but dig @127.0.0.1 timed out (UDP listener missing). FTL had TCP bound but UDP wasn't ready. Adding UDP check + automatic FTL restart fixed the intermittent failure.

Previous bug: DNS test used retries: 5 with until:. After all retries expired, the task failed fatally. Diagnostic tasks defined below it never executed. Switching to block/rescue ensures diagnostics always fire, and the rescue block restarts FTL + retries, recovering from transient upstream timeout.

Avoid regex_search on pct exec output

NEVER use regex_search with capture groups to parse pct exec output in set_fact or assert blocks. pct exec output passes through multiple shell layers (SSH → Proxmox host bash → lxc-attach → container shell) and may contain unexpected formatting, ANSI escapes, or extra whitespace that breaks regex matching.

Safe pattern: Use string-contains checks with in operator:

# BAD — crashes with 'NoneType' if regex returns None
- set_fact:
    val: "{{ output | regex_search('KEY=(\\S+)', '\\1') | first }}"

# GOOD — simple, robust, no crash risk
- assert:
    that: "'KEY=expected' in output"

# GOOD — for negation checks
- assert:
    that: "'KEY=0' not in output"

Previous bug: Batched rsyslog/gaming verify tasks used regex_search('MATCH=(\\S+)', '\\1') | first to extract values from pct exec output. The regex returned None on all 4 nodes, crashing set_fact with 'NoneType' object has no attribute 'group'. Replaced with string-contains checks.

Display-exclusive container assertions

When containers share a display device (iGPU DRI render node), a hookscript may stop one container when another starts (e.g., Kiosk stopped when Desktop LXC starts). Assert container config separately from running state:

- name: Assert container config is correct
  ansible.builtin.assert:
    that:
      - "'onboot: 1' in _cfg.stdout"
      - _cfg.stdout is regex('startup:.*order=6')

- name: Check if competing container is running
  ansible.builtin.command:
    cmd: pct status {{ desktop_ct_id }}
  register: _desktop_status
  changed_when: false
  failed_when: false

- name: Assert running OR stopped by hookscript
  ansible.builtin.assert:
    that: >-
      'running' in _cfg.stdout or
      ('stopped' in _cfg.stdout and 'running' in (_desktop_status.stdout | default('')))

Previous bug: Kodi and Kiosk assertions required 'running' state, but the display-exclusive hookscript correctly stopped them when the Desktop LXC started. Fix: accept 'stopped' when the competing container is running.

Multi-instance container IP from API

When the same container type deploys to multiple hosts (e.g., rsyslog on 4 nodes), the fleet API returns a single IP (last check-in) for all instances sharing the same container_id. ALWAYS use pct config for per-host container IPs in verify:

- name: Extract per-host container IP
  ansible.builtin.set_fact:
    _ct_ip: "{{ _ct_cfg.stdout | regex_search('ip=([^/,]+)', '\\1') | first }}"

Previous bug: rsyslog verify used API-returned IP for cross-service log tests. All 4 hosts got the same IP (10.10.10.14 — Home Assistant's IP), causing logger TCP to fail on 3 of 4 hosts. Fix: always use pct config for IP extraction.

systemd service state from callhome

callhome.py uses systemctl list-units which reports state as running (not active from systemctl is-active). Assertions against API systemd_services data MUST accept both values:

- _api.json.systemd_services.get('service', '') in ['active', 'running']

CRITICAL: Retries are a code smell — fix root causes, don't add wait time

NEVER use retries to paper over timing or infrastructure issues. Retries mask root causes and waste minutes per test cycle — compounding across every developer and every run.

The retry audit rule: Every retries: in verify.yml MUST have a comment explaining (1) what exact propagation delay it accounts for, and (2) the expected time. If you can't state both, the retry is hiding a bug.

Known propagation times (after relay interval fix):

• Container → NodeManager heartbeat: 5s (startup burst)
• NM → CM relay: 10s
• CM → SM relay: 10s
• Total container → SM: ~25s worst case
• Services already running by verify start: most checks should pass immediately

Hard rules:

• NEVER use retries > 6 for any check. If it doesn't work after 60s, it's broken.
• NEVER use delay > 10 for any check. Short polling finds fast successes.
• NEVER add retries when a service fails to start. Diagnose WHY it fails.
• REMOVE retries when the root cause is fixed. Zero retries is the ideal.
• NEVER add debug/diagnostic tasks as permanent verify fixtures. Fix the root cause, then remove the debug task.

Previous catastrophe (2026-04-14): kiosk-display-proxy on mesh1 was crash-looping with 957 restarts because a legacy kiosk-vnc-proxy from before the KasmVNC migration was still holding port 6080. Instead of diagnosing the root cause, retries were repeatedly increased (from 3×10s to 10×10s = 100 seconds). Debug tasks were added to capture systemctl show output. The SM heartbeat check used retries: 12, delay: 30 (6 MINUTES). The CM check was another 6 minutes. The verify phase alone burned 12+ minutes on retries that were NEVER going to succeed because the underlying issue was a port conflict, not timing.

Fix: (1) Added legacy VNC proxy cleanup before deploying new display proxies — service now starts on first attempt with zero retries. (2) Reduced relay heartbeat interval from 30s to 10s — SM/CM heartbeats arrive within seconds of verify starting, not minutes. (3) SM and CM checks: retries: 6, delay: 10 (60s max vs 360s before). Both now succeed on the FIRST attempt.

The real cost: These retries were silently burning 12+ minutes on EVERY molecule test run. Over dozens of test cycles, that's HOURS of wasted time. And the retries never surfaced the root cause — they just delayed the inevitable failure.

Common failures

• 0 assertions ran → dynamic group empty (add reconstruction)
• Rollback targets 0 hosts → empty group in cleanup
• igpu_available not defined → re-include role in verify
• Wrong IP for WAN hosts → use pct config extraction
• Per-feature scenario missing router_nodes group → LAN/WAN detection takes wrong path
• Register result used as string → ALWAYS use _var.stdout | trim, not _var directly. Register results are dicts, not strings. Previous bug: _jellyfin_verify_ip.split('.') failed because _jellyfin_verify_ip was a register result (dict), not the stdout string.
• set -o pipefail on non-pipeline commands → use ansible.builtin.command for single commands without pipes. Pipefail on non-pipeline commands adds noise and obscures the convention that pipefail signals a pipeline.
• config_files[path].keys() returns a method, not a list → use config_files[path]['keys'] to access the stored key list from callhome extensions.