# Openwrt Build > OpenWrt VM provisioning and configuration patterns. Use when modifying openwrt_vm or openwrt_configure roles, debugging OpenWrt network issues, working with UCI, opkg, firewall zones, WAN/LAN bridge ordering, bootstrap connectivity, or the two-phase restart pattern. - Skill: `krmorehead/openwrt-build` (Agent Skill) - Install (CLI): `npx skillmds@latest add krmorehead/openwrt-build` - Raw SKILL.md: https://api.skillmd.com/api/skills/krmorehead/openwrt-build/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Coding & Dev Tools - Author: krmorehead (https://skillmd.com/u/krmorehead) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/krmorehead/openwrt-build --- # OpenWrt Build Patterns ## Context OpenWrt is a router VM — it consumes ALL Proxmox bridges (WAN + every LAN port) and controls network topology for the entire host. This makes it fundamentally different from service VMs that attach to a single LAN bridge. These patterns are specific to OpenWrt and should not be applied to other VM types. ## Rules 1. OpenWrt gets ALL bridges: WAN on `net0`/`eth0`, remaining bridges as LAN ports. Most other VMs need only ONE LAN bridge. 2. The WAN bridge is auto-detected by `proxmox_bridges` via the host's default route. NEVER hardcode a bridge as WAN. Override with `openwrt_wan_bridge` in `host_vars` only if auto-detection fails. 3. After ANY network restart that changes interface assignments, ALWAYS restart the firewall before attempting outbound connections. Firewall zone bindings go stale when interfaces change. 4. Network operations (`wget`, DNS lookups) MUST have `retries` + `delay`. DNS, DHCP, and firewall state take seconds to settle after a restart. 5. Detached restart scripts MUST restart services in order: `firewall` → `dnsmasq` → `network` → `firewall` → `dropbear`. The first firewall/dnsmasq restart prepares for the topology change; the second firewall restart rebinds zones after interface changes. 6. NEVER restart the firewall synchronously over SSH when WAN zone rules have changed. The firewall applies WAN zone rules (input REJECT) to the current SSH path, killing the connection. ALWAYS use detached scripts with `ignore_unreachable: true`. 7. Per the project's "Bake, don't configure at runtime" principle: all packages are in the custom image. Configure roles NEVER run `opkg install`. To add a package, update `build-images.sh` and rebuild. 8. The `WAN_MAC` env variable is optional. NEVER apply it at the Proxmox NIC level during VM creation (`qm set --net0 macaddr=...`). ALWAYS go through the MAC conflict detection flow during the final configure phase. If no conflict is detected, apply via UCI (`uci set network.wan.macaddr`). If a conflict IS detected, defer the MAC to `/etc/openwrt_wan_mac_deferred` on the VM. An init script auto-applies it on the next boot when the conflict is gone. 9. Duplicate MAC addresses on the same L2 segment cause IPv6 DAD failures, corrupt uclient/libubox state, and cause `wget`/`opkg` segfaults — even when ICMP ping works. Consumer routers use sequential MACs across ports — the WAN MAC and LAN MAC often share the same OUI and differ by ±1 in the last byte. 10. BusyBox ash does NOT support `set -o pipefail`. NEVER add pipefail to `ansible.builtin.raw` tasks or `{{ openwrt_ssh }}` commands that run on OpenWrt. Pipefail is required for all host-side `ansible.builtin.shell` tasks — see the `proxmox-safety` rule. 11. BusyBox `ip neigh show` does NOT support IP filter arguments like full iproute2. ALWAYS use `/proc/net/arp` with `awk` to look up gateway MACs on OpenWrt. Similarly, avoid `ip -o`, `grep -oP`, and `grep -E` on OpenWrt. 12. BusyBox `tr -d '[:space:]'` deletes colons (`:`) because BusyBox treats `[:space:]` as a character set containing `[`, `:`, `s`, `p`, `a`, `c`, `e`, `]` — NOT as a POSIX character class. ALWAYS use explicit chars: `tr -d ' \t\n\r'`. 13. BusyBox `nc` does NOT support `-w` (timeout) flag. Use `(echo QUIT | nc HOST PORT) ` for selective rebuilds. Two **OpenWrt** images are produced via the OpenWrt Image Builder: 1. **Mesh LXC rootfs** (`openwrt-mesh-lxc-*-rootfs.tar.gz`): - WiFi packages pre-installed (`wpad-mesh-openssl`, `iw`, `kmod-iwlwifi`, `kmod-mt76`, `kmod-ath9k`, `kmod-ath10k-ct`) - `iw` included for namespace-aware WiFi detection via netlink - Firewall stripped (`-firewall4`, `-nftables`) - No routing (`-dnsmasq`, `-ppp`, `-odhcpd-ipv6only`) - UCI defaults: `eth0` on DHCP, no WAN, no IPv6, HTTP opkg feeds 2. **Router VM image** (`openwrt-router-*-combined.img.gz`): - WiFi mesh packages pre-installed - Security packages (`banip`), DNS packages (`https-dns-proxy`), mesh steering (`dawn`), diagnostics (`curl`, `ip-full`, `tcpdump`) - No UCI defaults baked in — `openwrt_configure` handles all config dynamically based on detected topology Build: `./build-images.sh` (downloads Image Builder once, caches in `.image-builder-cache/`). Use `--clean` to force re-download. All other service images (Debian/Fedora LXC templates) are built via `pct create` + package install + `vzdump`. Each is ~2 min to build. Per the project's design principles (`project-structure.mdc`): custom images are REQUIRED, there is no fallback, and configure roles NEVER run `opkg install` or `apt install`. To add a package, add it to `build-images.sh` and rebuild. ### Shell syntax and PATH through pct_remote The `community.proxmox.proxmox_pct_remote` connection plugin builds: `/usr/sbin/pct exec -- ` and sends it as a single string via SSH to the Proxmox host. The HOST's bash interprets the entire string before `pct exec` runs. This means: 1. **Semicolons split at host level.** `cmd1; cmd2` becomes two separate commands on the Proxmox host — only `cmd1` runs inside the container. 2. **Pipes split at host level.** `cmd1 | cmd2` — `cmd1` runs inside the container, `cmd2` runs on the HOST (filtering stdout from the container). This happens to work for text processing but is fragile. 3. **`export` is NOT a binary.** `lxc-attach` tries to exec the first word of the command as a binary. `export` is a shell builtin — it fails with `lxc-attach: Failed to exec "export"`. 4. **PATH is not set inside the container.** `lxc-attach`'s `execvp` uses the default path (`/bin:/usr/bin`), which misses `/sbin` and `/usr/sbin` where OpenWrt puts `uci`, `wifi`, `iw`. **Solution: wrap all commands in `/bin/sh -c '...'`.** The single quotes protect the payload from the host bash. Inside the container, busybox ash provides its default `PATH=/sbin:/usr/sbin:/bin:/usr/bin`. ```yaml # BAD — semicolons split at host level, export is not a binary - ansible.builtin.raw: >- export PATH="/usr/sbin:/usr/bin:/sbin:/bin:$PATH"; opkg update # BAD — for loops break (host bash tries to exec "for") - ansible.builtin.raw: >- for mod in iwlwifi ath9k; do modprobe "$mod" 2>/dev/null; done # GOOD — sh -c wraps everything in a container-side shell - ansible.builtin.raw: >- /bin/sh -c 'opkg update' # GOOD — complex commands with semicolons inside sh -c - ansible.builtin.raw: >- /bin/sh -c 'opkg list-installed 2>/dev/null | grep -c wpad-mesh || true' # GOOD — multi-command chains use && inside sh -c - ansible.builtin.raw: >- /bin/sh -c 'uci set wireless.mesh0=wifi-iface && uci set wireless.mesh0.device="radio0" && uci commit wireless' ``` **Quoting rules for sh -c through pct_remote:** - Outer single quotes protect the entire payload from host bash - Inside, use double quotes for values: `uci set foo.bar="value"` - NEVER nest single quotes — use double quotes or drop quotes for simple alphanumeric values - `&&` and `||` inside single quotes are interpreted by container ash - `[ ... ] && echo x || echo y` WITHOUT sh -c is OK — `[` is exec'd by lxc-attach, `&&`/`||` chain at host level (works for simple checks) ## Feature integration via task files Post-baseline features (security, VLANs, DNS, mesh) are implemented as separate task files within `roles/openwrt_configure/tasks/`: ``` roles/openwrt_configure/tasks/ ├── main.yml # Baseline configuration (WAN, LAN, DHCP, firewall) ├── security.yml # M1: SSH hardening, banIP ├── vlans.yml # M2: VLAN segmentation ├── dns.yml # M3: Encrypted DNS (https-dns-proxy) └── mesh.yml # M4: WDS WiFi backhaul + Dawn steering ``` Each feature gets TWO plays in `site.yml`: 1. A configure play targeting the `openwrt` dynamic group with `include_role` using `tasks_from: .yml` 2. A `deploy_stamp` play targeting `router_nodes` (Proxmox host) to record the feature was applied Both plays share a tag (e.g., `openwrt-security`) so they can be run independently via `--tags`. This pattern avoids re-running baseline tasks when iterating on a feature and enables per-feature molecule scenarios that converge only the relevant task file. ## SSH auth transition After security hardening (M1), OpenWrt switches from password auth (empty password) to key-only auth. This is a critical ordering problem: 1. **Deploy key**: copy the public key to OpenWrt via `raw` (password auth still works) 2. **Verify key auth**: test SSH with the key to confirm it works 3. **Disable password auth**: `uci set dropbear.@dropbear[0].PasswordAuth='off'` 4. **Re-register `openwrt` host**: `add_host` with `-i ` in SSH args Steps 1-4 MUST happen in this exact order within a single play. If step 3 runs before step 2 confirms key auth works, the VM becomes unreachable. The key path comes from `OPENWRT_SSH_PRIVATE_KEY` env var (optional, defined in role `defaults/main.yml`). When not set, security hardening skips SSH lockdown and only installs banIP. **Rollback** must reverse this completely: re-enable password auth in dropbear, clear the root password in `/etc/shadow` (restore empty-password baseline), and remove the authorized key. ## VLAN implementation (virtual environment) On physical OpenWrt routers, VLANs use DSA or swconfig for port-based tagging. In a Proxmox VM, OpenWrt has no physical switch — only virtual NICs (`eth0`, `eth1`, etc.) backed by Proxmox bridges. VLANs in the virtual environment use 802.1Q VLAN devices on bridge ports: ``` eth1 (LAN bridge port) ├── eth1.10 (IoT VLAN) ├── eth1.20 (Guest VLAN) └── eth1.30 (Management VLAN) ``` Proxmox bridges pass tagged frames by default. No `bridge-vlan-aware` or trunk configuration is needed on the Proxmox side — the VLAN tagging happens entirely within OpenWrt. NEVER use DSA or swconfig configurations for virtual OpenWrt deployments. ## Encrypted DNS integration `https-dns-proxy` on OpenWrt auto-configures dnsmasq on install: it adds itself as the upstream DNS server and restarts dnsmasq. No manual dnsmasq configuration is needed for the basic DoH setup. The configure task only needs to: 1. Install `https-dns-proxy` (with retries per rule 4) 2. Optionally configure specific DoH providers via UCI 3. Verify DNS resolution works through the proxy ## WAN/LAN bridge ordering `openwrt_vm` orders bridges so the WAN bridge is always `net0`/`eth0`: ```yaml _ordered_bridges: [_wan_bridge] + (proxmox_all_bridges | difference([_wan_bridge]) | sort) ``` Previous bug: alphabetical bridge sorting made `vmbr0` always WAN. When the modem was on `vmbr0`, the Proxmox GUI became unreachable from LAN nodes. ## Two-phase restart pattern OpenWrt needs two network restarts because the LAN IP changes mid-run: **Phase 1** (WAN + LAN ports, keep default LAN IP): 1. Configure WAN device, LAN bridge ports via UCI 2. `uci commit` → detached script (firewall → dnsmasq → network → firewall → dropbear) 3. Pause 30s for services to stabilize 4. Migrate bootstrap IP from WAN bridge to LAN bridge on Proxmox 5. Wait for SSH on LAN bridge, wait for WAN default route 6. Restart firewall again (zone rebinding after interface change) 7. Install packages (opkg) while connectivity works **Phase 2** (final LAN IP + DHCP + WAN MAC): 1. Set final LAN IP, netmask, DHCP params, and WAN MAC (if configured) via UCI 2. `uci commit` → detached script (firewall → dnsmasq → network → firewall → dropbear) 3. Pause 30s, clean up bootstrap IP The split is necessary because changing the LAN IP in Phase 1 would break SSH mid-configure. ## Detached restart scripts The detached script pattern survives SSH disconnects caused by network restarts: ```bash printf '#!/bin/sh\nsleep 1\n/etc/init.d/firewall restart\n/etc/init.d/dnsmasq restart\nsleep 2\n/etc/init.d/network restart\nsleep 5\n/etc/init.d/firewall restart\n/etc/init.d/dropbear restart\nrm -f /tmp/_restart_net.sh\n' \ > /tmp/_restart_net.sh && chmod +x /tmp/_restart_net.sh && \ start-stop-daemon -S -b -x /tmp/_restart_net.sh ``` Previous bugs: - Script omitted `firewall restart` → stale zones → `opkg update` got `EPERM`. - Firewall + dnsmasq were restarted synchronously before the detached script → SSH connection killed because WAN zone rules (input REJECT) were applied to the bootstrap SSH path. ## Bootstrap connectivity To reach OpenWrt at its default `192.168.1.1` during initial setup: 1. Add a temporary IP (`192.168.1.2`) to the WAN bridge on Proxmox 2. SSH through ProxyJump via the Proxmox host 3. After Phase 1 network restart, OpenWrt's LAN moves to non-WAN bridges 4. Remove bootstrap IP from WAN bridge, add to LAN bridge 5. After Phase 2, clean up bootstrap IP entirely Previous bug: we excluded `vmbr0` and tried to connect through `vmbr1` (which had no IP in OpenWrt's default config). ## Proxmox LAN management IP When OpenWrt is the primary router, the Proxmox host needs a predictable IP on the LAN bridge so the GUI is reachable from leaf nodes: 1. Compute LAN IP from OpenWrt's LAN subnet + offset (default `.2`) 2. `ip addr add` on the LAN bridge (immediate, current session) 3. Upgrade the LAN bridge in `ansible-bridges.conf` from `inet manual` to `inet dhcp` 4. Add a DHCP static reservation on OpenWrt mapping the Proxmox host's LAN bridge MAC to the computed IP 5. On reboot: the DHCP client on the LAN bridge requests an IP, OpenWrt always assigns the reserved one 6. Remove any stale LAN-subnet IPs from non-LAN bridges to prevent routing conflicts 7. Remove any separate `ansible-proxmox-lan.conf` (superseded by bridges.conf DHCP) 8. Write `.state/addresses.json` with both the management IP and the new LAN IP 9. Probe original management IP — if unreachable (topology changed), update `ansible_host` via `add_host` NEVER use a separate config file with `iface inet dhcp` — it conflicts with the `inet manual` stanza in the bridges config and `ifreload -a` won't start the DHCP client. ALWAYS modify the bridge stanza in-place. NEVER leave stale IPs on non-LAN bridges in the same subnet as the LAN bridge. Two routes for the same /24 on different bridges causes the kernel to use the wrong interface, breaking all LAN VM connectivity. ## Auto-subnet selection To avoid collisions between the WAN subnet and the OpenWrt LAN subnet: 1. Detect the upstream gateway prefix from the Proxmox host's default route 2. Pass it to OpenWrt via `add_host` as `upstream_wan_prefix` 3. Iterate `openwrt_lan_subnet_candidates` and pick the first whose prefix differs from WAN ## State file for cross-run IP discovery `build.py` probes `PRIMARY_HOST` before running Ansible. If unreachable, it reads `.state/addresses.json` for cached alternative IPs. This handles cable-swap scenarios where the original management IP is no longer routable. The state file is written by `openwrt_configure` and cleaned by both cleanup playbooks. It is gitignored. ## WAN MAC conflict detection Before applying a cloned WAN MAC, the build runs a three-layer conflict check: 1. **Exact MAC in ARP table** (`/proc/net/arp`): catches direct L2 duplicates 2. **EUI-64 in IPv6 neighbor table** (`ip -6 neigh`): catches SLAAC address collisions where the MAC itself isn't visible but its derived IPv6 address is 3. **Gateway OUI match**: if the WAN gateway's MAC shares the first 3 bytes (OUI) with the cloned MAC, the devices are almost certainly from the same router — a conflict waiting to happen If any check triggers, the MAC is saved to `/etc/openwrt_wan_mac_deferred` and NOT applied to UCI. The build also deploys `/etc/init.d/wan_mac_apply` — an OpenWrt init script (START=99) that runs on every boot. It re-runs the same three-layer conflict detection and, if the conflict is gone (old router removed), applies the MAC via UCI and restarts the network automatically. No manual intervention required. Previous bug: WAN MAC `08:B4:B1:1A:63:08` was applied while the old router (LAN MAC `08:B4:B1:1A:63:09`, same OUI) was still on the segment. IPv6 SLAAC generated the same global address → DAD failure → corrupted network stack → `wget` EPERM, `opkg` failures, DNS timeouts. ICMP still worked, making the root cause non-obvious without `dmesg` diagnostics. The deferred file is cleaned at the start of each run (`rm -f`) for idempotency. The VM destruction during cleanup also removes it. The init script self-cleans by removing the deferred file after applying the MAC. ## OpenWrt Mesh LXC (satellite nodes) Mesh satellite nodes (`wifi_nodes:!router_nodes`) run OpenWrt in a **privileged LXC container** instead of a VM. This allows WiFi management via WDS AP/STA without requiring PCIe passthrough (IOMMU/VT-d). The container receives the host's WiFi PHY via `iw phy set netns ` (network namespace move). Key differences from the VM pattern: - No routing — mesh containers are NOT routers. They run WDS STA only. - Uses the OpenWrt rootfs tarball (`openwrt-*-rootfs.tar.gz`), not the VM disk image. - Must be privileged (`unprivileged: false`) for the PHY namespace move. - Must set `--ostype unmanaged` because Proxmox cannot auto-detect OpenWrt. - Container readiness uses `ls /` (not `hostname`, which is absent in BusyBox). - `lxc_ct_skip_debian_cleanup: true` to avoid dpkg operations on OpenWrt. - A Proxmox hookscript re-moves the WiFi PHY after container restarts. WiFi PHY handling: 1. Load common WiFi kernel modules (`iwlwifi`, `ath9k`, etc.) on the host. 2. Detect PHYs in `/sys/class/ieee80211/`. 3. If no PHYs found, hard-fail. All `wifi_nodes` are expected to have WiFi. Missing WiFi usually means stale vfio-pci bindings from a previous run or missing firmware. 4. Move PHYs into the container's network namespace. 5. Deploy hookscript for persistence across reboots. IMPORTANT: Detect WiFi radios inside LXC containers with `iw phy` (netlink), NOT `ls /sys/class/ieee80211/` (sysfs). LXC containers bind-mount the host's sysfs, which doesn't reflect network-namespace-specific entries like WiFi PHYs. `iw phy` queries the kernel via netlink and correctly sees PHYs moved into the container's network namespace. The `iw` package must be pre-installed in the custom image or via `opkg install iw`. Previous bug: `ls /sys/class/ieee80211/` inside the container returned empty despite a successful `iw phy set netns` — sysfs showed the host's view. IMPORTANT: NEVER `modprobe` WiFi modules inside the container via `pct_remote`. `modprobe` inside the container runs on the HOST kernel (containers share the kernel). If the module reloads, the new PHY appears in the HOST namespace, not the container namespace — effectively un-doing the PHY namespace move. Modules MUST be loaded on the host BEFORE the container is created and the PHY moved. Previous bug: `modprobe iwlwifi` inside the container via `pct_remote` caused the PHY to revert to the host namespace. WiFi detection inside the container then found zero radios despite a successful namespace move. IMPORTANT: `proxmox_pci_passthrough` cleans stale vfio bindings on non-router hosts. If WiFi was previously bound to vfio-pci, the role removes `blacklist-wifi.conf` and `vfio-pci.conf`, unbinds devices, and reloads drivers. Without this cleanup, WiFi PHYs are invisible to the mesh role. Previous bug: mesh1 WiFi was bound to vfio-pci from a prior test cycle. `/sys/class/ieee80211/` was empty despite the hardware being present. IMPORTANT: After the WiFi PHY is namespace-moved into the container, OpenWrt does NOT auto-generate `/etc/config/wireless`. The configure role MUST run `wifi config` inside the container to generate the wireless configuration from detected hardware BEFORE any `uci set wireless.radio*` commands. Without this step, `uci set wireless.radio0.disabled=0` fails with `uci: Invalid argument` because the `radio0` section doesn't exist. Previous bug: `uci set wireless.radio0.disabled=0` failed on both mesh1 and mesh2. The PHY was detected by `iw phy` (found `phy0`), but the UCI wireless config had no matching `radio0` section because the PHY was moved into the namespace after the container booted. Container networking follows host topology: - LAN hosts (`router_nodes`, `lan_hosts`) → OpenWrt LAN subnet, LAN bridge. - WAN hosts → `proxmox_wan_bridge`, `ansible_default_ipv4` subnet, DNS `8.8.8.8`. - IP offset +200 for WAN containers to avoid collisions with LAN containers. ## Permanent diagnostics The OpenWrt build includes diagnostic tasks at two key milestones: **Bootstrap diagnostics** (`openwrt_vm`, after SSH bootstrap): - VM status, bridge layout, bootstrap IP presence, dmesg errors **Phase 1 diagnostics** (`openwrt_configure`, after WAN route + firewall restart): - WAN route, WAN IP, LAN IP, DNS resolvers, firewall status, dmesg errors **Final diagnostics** (`openwrt_configure`, end of build): - VM status, onboot/startup config, LAN bridge IP, management config presence These run on EVERY build. When a build fails, the diagnostic output from the last successful milestone narrows the failure window. ## Detached script verification Detached scripts report "success" when they launch — NOT when they complete. NEVER trust the launch result. ALWAYS verify the expected outcome after the pause: 1. `wait_for` on SSH port (proves dropbear restarted) 2. `wait_for` on WAN default route (proves network restarted) 3. Firewall restart task (proves firewall can be restarted = it's running) If the detached script fails silently, these verification steps catch it. Previous bug: detached script launched successfully but the firewall restart inside it failed. The pause completed, and subsequent tasks got EPERM because the firewall zones were stale. The verification pattern (`wait_for` + explicit firewall restart) would have caught or self-healed this.