Proxmox Host Safety

Proxmox Host Safety Validation

Pre-flight checklist

Before running ANY command or playbook against a Proxmox host, validate:

1. Network-killing commands (BLOCK these)

These commands will sever SSH and make the host unreachable:

# DANGEROUS - will kill your connection
- ifdown --all
- ifdown --all --force
- systemctl stop networking
- systemctl restart networking   # can drop and fail to restore
- ip link delete vmbr0           # destroys management bridge
- ip link set vmbr0 down         # kills management path

Safe alternatives:

# Safe - additive, brings up new interfaces without tearing down existing
- ifup --all --force

# Safe - reload that preserves running interfaces
- ifreload -a

# Safe - tear down a SPECIFIC non-management bridge
- ip link set vmbr5 down && ip link delete vmbr5

2. Bridge teardown safety

When removing bridges during cleanup:

• Get the management bridge from the host's default route device (do NOT assume vmbr0).
• NEVER tear down the management bridge.
• Iterate over stale bridges and skip the management one:

- name: Tear down stale bridges (skip management)
  ansible.builtin.shell:
    cmd: |
      mgmt_br=$(ip -o route show default | awk '{print $5}' | head -1)
      for br in $(ip -br link show type bridge | awk '{print $1}'); do
        [ "$br" = "$mgmt_br" ] && continue
        ip link set "$br" down
        ip link delete "$br"
      done
  changed_when: true

2b. WAN bridge ordering

NEVER hardcode bridge-to-role mappings (e.g., vmbr0 = WAN). The WAN bridge is detected at runtime via the host's default route. openwrt_vm orders bridges so the WAN bridge maps to net0/eth0; all others become LAN. Override with openwrt_wan_bridge in host_vars if needed.

Previous bug: hardcoded vmbr0 = WAN made Proxmox GUI unreachable when the modem was plugged into the NIC behind vmbr0, because leaf nodes on the LAN bridge had no route to the management IP on the WAN bridge.

3. LVM operations on root volumes

• Do NOT create LVM snapshots of the Proxmox root volume (pve/root). Merging snapshots on a live root volume is unreliable and can leave the system in a stuck merge state requiring reboot.
• Use file-based config backups (tar) and vzdump for VMs instead.

4. Reboot awareness

• If a playbook changes GRUB, initramfs, or kernel modules, a reboot may be needed.
• Set pci_passthrough_allow_reboot: true in host vars to allow automated reboots.
• After reboot, wait for SSH to come back with wait_for_connection.

4b. Hardware detection: hard-fail, not graceful skip

• iGPU: every modern Intel CPU has one. proxmox_igpu MUST hard-fail if absent.
• WiFi + VT-d/IOMMU: required for PCI passthrough. proxmox_pci_passthrough MUST hard-fail if IOMMU is not active after reboot or groups are invalid.
• NIC count: OK to handle dynamically (hardware legitimately varies).
• NEVER add "graceful skip" for hardware expected on every host. Silent skips mask fixable BIOS settings (VT-d disabled) behind warnings that are easy to miss.
• Previous bug: proxmox_pci_passthrough silently skipped WiFi passthrough when IOMMU groups were invalid on mesh1. Root cause was VT-d disabled in BIOS — a 30-second fix masked for an entire test cycle.

5. Cleanup completeness

When ANY role deploys a file to the Proxmox host, ALWAYS add it to the removal list in BOTH cleanup playbooks (molecule/default/cleanup.yml AND playbooks/cleanup.yml).

Current ansible-managed files that must be cleaned:

• /etc/network/interfaces.d/ansible-bridges.conf (bridge config, may be modified to inet dhcp)
• /etc/network/interfaces.d/ansible-proxmox-lan.conf (legacy LAN management IP, superseded)
• /etc/network/interfaces.d/ansible-temp-lan.conf (test workaround, cleaned up)
• /etc/modprobe.d/blacklist-wifi.conf (WiFi driver blacklist)
• /etc/modprobe.d/vfio-pci.conf (PCI passthrough config)
• /etc/ansible/facts.d/vm_builds.fact (deploy stamp tracking)
• /etc/apt/sources.list.d/pve-no-subscription.sources (added by proxmox_igpu)
• /tmp/openwrt-router-*.img* (left behind if build fails mid-upload)
• /var/lib/vz/template/cache/debian-*.tar.zst (LXC templates)
• Enterprise repos: restore pve-enterprise.sources.disabled → .sources and ceph.sources.disabled → .sources

Local state files that must be cleaned (via delegate_to: localhost):

• .state/addresses.json (cached host IPs)

Previous bug: ansible-proxmox-lan.conf was deployed but not cleaned up, leaving stale LAN management IPs across test runs.

6. Test machine protocol

Before running destructive operations (cleanup, VM destroy):

1. Confirm the target is the test machine (check PROXMOX_HOST env var).
2. Verify a backup exists (check for manifest.json in backup dir).
3. Use the cleanup.sh wrapper which enforces env file sourcing.

SSH stability for OpenWrt connections

After OpenWrt network restart, the LAN IP changes (e.g., 192.168.1.1 → 10.10.10.1). The bootstrap SSH connection will hang forever unless ConnectTimeout is set.

Required SSH args for baseline (password auth) OpenWrt connections:

ansible_ssh_common_args: >-
  -o ProxyJump=root@{{ ansible_host }}
  -o StrictHostKeyChecking=no
  -o UserKnownHostsFile=/dev/null
  -o PubkeyAuthentication=no
  -o ConnectTimeout=10
  -o ServerAliveInterval=15
  -o ServerAliveCountMax=4

After security hardening (M1), SSH switches to key auth. Replace PubkeyAuthentication=no with -i <key_path> and remove sshpass:

ansible_ssh_common_args: >-
  -o ProxyJump=root@{{ ansible_host }}
  -o StrictHostKeyChecking=no
  -o UserKnownHostsFile=/dev/null
  -o ConnectTimeout=10
  -o ServerAliveInterval=15
  -o ServerAliveCountMax=4
  -i {{ lookup('env', 'OPENWRT_SSH_PRIVATE_KEY') }}

The group reconstruction task file (tasks/reconstruct_openwrt_group.yml) auto-detects which auth method is active by checking deploy_stamp state and the OPENWRT_SSH_PRIVATE_KEY env var.

• ConnectTimeout=10: Prevents infinite hang when LAN IP changes.
• ServerAliveInterval=15: Prevents connection drop during local Ansible tasks (set_fact sequences) that don't generate SSH traffic.
• NEVER retry SSH to the old bootstrap address after LAN reconfiguration.

PCI device cleanup after VM destruction

Devices bound to vfio-pci do NOT auto-revert when the VM is destroyed. Without cleanup, the next run can't detect WiFi hardware:

# 1. Unbind all vfio-pci devices
for dev in /sys/bus/pci/drivers/vfio-pci/0000:*/; do
  addr=$(basename "$dev")
  echo "$addr" > /sys/bus/pci/drivers/vfio-pci/unbind
done

# 2. Remove blacklist and vfio config files
rm -f /etc/modprobe.d/blacklist-wifi.conf /etc/modprobe.d/vfio-pci.conf

# 3. Rebind WiFi via sysfs (NEVER modprobe -r)
# Use tasks/sysfs_wifi_rebind.yml or inline:
for phy in /sys/class/ieee80211/phy*; do
  PCI=$(basename $(readlink -f "$phy/device"))
  DRV=$(basename $(readlink -f "$phy/device/driver"))
  echo "$PCI" > /sys/bus/pci/drivers/$DRV/unbind
  echo "" > /sys/bus/pci/devices/$PCI/driver_override
done

# 4. Rescan PCI bus + explicit bind
echo 1 > /sys/bus/pci/rescan
sleep 1
# Explicitly bind back (rescan alone won't auto-bind loaded modules)
echo "$PCI" > /sys/bus/pci/drivers/$DRV/bind

All steps are required. The explicit bind (step 4) is critical -- echo 1 > /sys/bus/pci/rescan alone is insufficient because the kernel won't auto-bind drivers that were explicitly unbound.

GPU driver cleanup — SEPARATE from WiFi PCI cleanup

GPU driver unload (i915/amdgpu) is NOT the same as WiFi driver unload. WiFi modprobe -r iwlwifi/iwlmvm is BANNED on ALL hardware. On AMD APUs, WiFi module unload destabilizes the NBIO via PCIe reset, killing USB ethernet hours later. Use tasks/sysfs_wifi_rebind.yml (sysfs unbind + PCI rescan + explicit bind) instead — universally safe on Intel and AMD. GPU modprobe -r amdgpu on a single-GPU AMD host causes a kernel panic.

Rules:

• NEVER run modprobe -r amdgpu or modprobe -r i915 in broad-scope plays (hosts: proxmox*). PCI bus rescan after vfio-pci unbind is sufficient for E2E cleanup.
• ONLY run GPU driver unload in per-feature cleanup (e.g., sunshine-vm, gaming-rollback) gated on VGA controller count >= 2.
• tests/test_host_safety.py catches modprobe -r amdgpu/i915 in broad-scope plays without VGA guards. Run pytest tests/ to catch this at dev time.

Previous bug: E2E cleanup ran modprobe -r amdgpu on ALL hosts including ai (single AMD GPU, USB ethernet). Kernel panicked, host crashed. Required physical power-on 3000 miles away.

Host recoverability

Every host MUST declare wol_capable (true/false) in inventory/host_vars/. USB ethernet adapters do NOT support WoL. Hosts with wol_capable: false (e.g., ai) cannot be recovered remotely after a crash or shutdown.

Rules:

• NEVER run operations that could crash non-WoL hosts (modprobe -r sole GPU, shutdown, poweroff)
• scripts/wol.sh MUST NOT include non-WoL hosts — enforced by tests/test_wol.py
• tests/test_host_safety.py enforces GPU cleanup safety — static linter catches the bug class at pytest time

Decision tree

Is this command touching network interfaces?
├── YES → Does it tear down ALL interfaces?
│   ├── YES → BLOCK. Use targeted teardown instead.
│   └── NO → Is it tearing down vmbr0?
│       ├── YES → BLOCK.
│       └── NO → SAFE. Proceed.
└── NO → Is it modifying LVM on root?
    ├── YES → BLOCK. Use tar + vzdump instead.
    └── NO → Does it unload GPU drivers (modprobe -r amdgpu/i915)?
        ├── YES → Is the play broad-scope (hosts: proxmox*)?
        │   ├── YES → BLOCK. Use PCI bus rescan instead.
        │   └── NO → Is VGA count >= 2?
        │       ├── YES → SAFE. Proceed.
        │       └── NO → BLOCK. Kernel panic on single-GPU AMD.
        └── NO → SAFE. Proceed.