openwrt-build

name	openwrt-build
description	OpenWrt VM provisioning and configuration patterns. Use when modifying openwrt_vm or openwrt_configure roles, debugging OpenWrt network issues, working with UCI, opkg, firewall zones, WAN/LAN bridge ordering, bootstrap connectivity, or the two-phase restart pattern.

OpenWrt Build Patterns

Context

OpenWrt is a router VM — it consumes ALL Proxmox bridges (WAN + every LAN port) and controls network topology for the entire host. This makes it fundamentally different from service VMs that attach to a single LAN bridge. These patterns are specific to OpenWrt and should not be applied to other VM types.

Rules

1. OpenWrt gets ALL bridges: WAN on net0/eth0, remaining bridges as LAN ports. Most other VMs need only ONE LAN bridge.
2. The WAN bridge is auto-detected by proxmox_bridges via the host's default route. NEVER hardcode a bridge as WAN. Override with openwrt_wan_bridge in host_vars only if auto-detection fails.
3. After ANY network restart that changes interface assignments, ALWAYS restart the firewall before attempting outbound connections. Firewall zone bindings go stale when interfaces change.
4. Network operations (wget, DNS lookups) MUST have retries + delay. DNS, DHCP, and firewall state take seconds to settle after a restart.
5. Detached restart scripts MUST restart services in order: firewall → dnsmasq → network → firewall → dropbear. The first firewall/dnsmasq restart prepares for the topology change; the second firewall restart rebinds zones after interface changes.
6. NEVER restart the firewall synchronously over SSH when WAN zone rules have changed. The firewall applies WAN zone rules (input REJECT) to the current SSH path, killing the connection. ALWAYS use detached scripts with ignore_unreachable: true.
7. Per the project's "Bake, don't configure at runtime" principle: all packages are in the custom image. Configure roles NEVER run opkg install. To add a package, update build-images.sh and rebuild.
8. The WAN_MAC env variable is optional. NEVER apply it at the Proxmox NIC level during VM creation (qm set --net0 macaddr=...). ALWAYS go through the MAC conflict detection flow during the final configure phase. If no conflict is detected, apply via UCI (uci set network.wan.macaddr). If a conflict IS detected, defer the MAC to /etc/openwrt_wan_mac_deferred on the VM. An init script auto-applies it on the next boot when the conflict is gone.
9. Duplicate MAC addresses on the same L2 segment cause IPv6 DAD failures, corrupt uclient/libubox state, and cause wget/opkg segfaults — even when ICMP ping works. Consumer routers use sequential MACs across ports — the WAN MAC and LAN MAC often share the same OUI and differ by ±1 in the last byte.
10. BusyBox ash does NOT support set -o pipefail. NEVER add pipefail to ansible.builtin.raw tasks or {{ openwrt_ssh }} commands that run on OpenWrt. Pipefail is required for all host-side ansible.builtin.shell tasks — see the proxmox-safety rule.
11. BusyBox ip neigh show does NOT support IP filter arguments like full iproute2. ALWAYS use /proc/net/arp with awk to look up gateway MACs on OpenWrt. Similarly, avoid ip -o, grep -oP, and grep -E on OpenWrt.
12. BusyBox tr -d '[:space:]' deletes colons (:) because BusyBox treats [:space:] as a character set containing [, :, s, p, a, c, e, ] — NOT as a POSIX character class. ALWAYS use explicit chars: tr -d ' \t\n\r'.
13. BusyBox nc does NOT support -w (timeout) flag. Use (echo QUIT | nc HOST PORT) </dev/null for TCP port checks. NEVER use echo | nc -w 3 on OpenWrt.
14. When checking for the default route in scripts on OpenWrt, NEVER filter by device name (ip route show default dev eth0). OpenWrt's netifd may use interface aliases (e.g., wan, eth0.2) that differ from the physical device name. Use ip route show default without a device filter.

Custom images (build-images.sh)

build-images.sh builds pre-configured images for all 15 services. This section covers the OpenWrt-specific images. For the full list of targets, see ./build-images.sh --help or the image management skill.

Available targets: router, mesh, pihole, rsyslog, netdata, wireguard, homeassistant, jellyfin, kodi, moonlight, gaming, sunshine, desktop. Use --only <target> for selective rebuilds.

Two OpenWrt images are produced via the OpenWrt Image Builder:

1. Mesh LXC rootfs (openwrt-mesh-lxc-*-rootfs.tar.gz):

   • WiFi packages pre-installed (wpad-mesh-openssl, iw, kmod-iwlwifi, kmod-mt76, kmod-ath9k, kmod-ath10k-ct)
   • iw included for namespace-aware WiFi detection via netlink
   • Firewall stripped (-firewall4, -nftables)
   • No routing (-dnsmasq, -ppp, -odhcpd-ipv6only)
   • UCI defaults: eth0 on DHCP, no WAN, no IPv6, HTTP opkg feeds

2. Router VM image (openwrt-router-*-combined.img.gz):

   • WiFi mesh packages pre-installed
   • Security packages (banip), DNS packages (https-dns-proxy), mesh steering (dawn), diagnostics (curl, ip-full, tcpdump)
   • No UCI defaults baked in — openwrt_configure handles all config dynamically based on detected topology

Build: ./build-images.sh (downloads Image Builder once, caches in .image-builder-cache/). Use --clean to force re-download.

All other service images (Debian/Fedora LXC templates) are built via pct create + package install + vzdump. Each is ~2 min to build.

Per the project's design principles (project-structure.mdc): custom images are REQUIRED, there is no fallback, and configure roles NEVER run opkg install or apt install. To add a package, add it to build-images.sh and rebuild.

Shell syntax and PATH through pct_remote

The community.proxmox.proxmox_pct_remote connection plugin builds: /usr/sbin/pct exec <vmid> -- <cmd> and sends it as a single string via SSH to the Proxmox host. The HOST's bash interprets the entire string before pct exec runs. This means:

1. Semicolons split at host level. cmd1; cmd2 becomes two separate commands on the Proxmox host — only cmd1 runs inside the container.
2. Pipes split at host level. cmd1 | cmd2 — cmd1 runs inside the container, cmd2 runs on the HOST (filtering stdout from the container). This happens to work for text processing but is fragile.
3. export is NOT a binary. lxc-attach tries to exec the first word of the command as a binary. export is a shell builtin — it fails with lxc-attach: Failed to exec "export".
4. PATH is not set inside the container. lxc-attach's execvp uses the default path (/bin:/usr/bin), which misses /sbin and /usr/sbin where OpenWrt puts uci, wifi, iw.

Solution: wrap all commands in /bin/sh -c '...'. The single quotes protect the payload from the host bash. Inside the container, busybox ash provides its default PATH=/sbin:/usr/sbin:/bin:/usr/bin.

# BAD — semicolons split at host level, export is not a binary
- ansible.builtin.raw: >-
    export PATH="/usr/sbin:/usr/bin:/sbin:/bin:$PATH";
    opkg update

# BAD — for loops break (host bash tries to exec "for")
- ansible.builtin.raw: >-
    for mod in iwlwifi ath9k; do modprobe "$mod" 2>/dev/null; done

# GOOD — sh -c wraps everything in a container-side shell
- ansible.builtin.raw: >-
    /bin/sh -c 'opkg update'

# GOOD — complex commands with semicolons inside sh -c
- ansible.builtin.raw: >-
    /bin/sh -c
    'opkg list-installed 2>/dev/null | grep -c wpad-mesh || true'

# GOOD — multi-command chains use && inside sh -c
- ansible.builtin.raw: >-
    /bin/sh -c
    'uci set wireless.mesh0=wifi-iface &&
    uci set wireless.mesh0.device="radio0" &&
    uci commit wireless'

Quoting rules for sh -c through pct_remote:

• Outer single quotes protect the entire payload from host bash
• Inside, use double quotes for values: uci set foo.bar="value"
• NEVER nest single quotes — use double quotes or drop quotes for simple alphanumeric values
• && and || inside single quotes are interpreted by container ash
• [ ... ] && echo x || echo y WITHOUT sh -c is OK — [ is exec'd by lxc-attach, &&/|| chain at host level (works for simple checks)

Feature integration via task files

Post-baseline features (security, VLANs, DNS, mesh) are implemented as separate task files within roles/openwrt_configure/tasks/:

roles/openwrt_configure/tasks/
├── main.yml          # Baseline configuration (WAN, LAN, DHCP, firewall)
├── security.yml      # M1: SSH hardening, banIP
├── vlans.yml         # M2: VLAN segmentation
├── dns.yml           # M3: Encrypted DNS (https-dns-proxy)
└── mesh.yml          # M4: WDS WiFi backhaul + Dawn steering

Each feature gets TWO plays in site.yml:

1. A configure play targeting the openwrt dynamic group with include_role using tasks_from: <feature>.yml
2. A deploy_stamp play targeting router_nodes (Proxmox host) to record the feature was applied

Both plays share a tag (e.g., openwrt-security) so they can be run independently via --tags.

This pattern avoids re-running baseline tasks when iterating on a feature and enables per-feature molecule scenarios that converge only the relevant task file.

SSH auth transition

After security hardening (M1), OpenWrt switches from password auth (empty password) to key-only auth. This is a critical ordering problem:

1. Deploy key: copy the public key to OpenWrt via raw (password auth still works)
2. Verify key auth: test SSH with the key to confirm it works
3. Disable password auth: uci set dropbear.@dropbear[0].PasswordAuth='off'
4. Re-register openwrt host: add_host with -i <key_path> in SSH args

Steps 1-4 MUST happen in this exact order within a single play. If step 3 runs before step 2 confirms key auth works, the VM becomes unreachable.

The key path comes from OPENWRT_SSH_PRIVATE_KEY env var (optional, defined in role defaults/main.yml). When not set, security hardening skips SSH lockdown and only installs banIP.

Rollback must reverse this completely: re-enable password auth in dropbear, clear the root password in /etc/shadow (restore empty-password baseline), and remove the authorized key.

VLAN implementation (virtual environment)

On physical OpenWrt routers, VLANs use DSA or swconfig for port-based tagging. In a Proxmox VM, OpenWrt has no physical switch — only virtual NICs (eth0, eth1, etc.) backed by Proxmox bridges.

VLANs in the virtual environment use 802.1Q VLAN devices on bridge ports:

eth1 (LAN bridge port)
├── eth1.10 (IoT VLAN)
├── eth1.20 (Guest VLAN)
└── eth1.30 (Management VLAN)

Proxmox bridges pass tagged frames by default. No bridge-vlan-aware or trunk configuration is needed on the Proxmox side — the VLAN tagging happens entirely within OpenWrt.

NEVER use DSA or swconfig configurations for virtual OpenWrt deployments.

Encrypted DNS integration

https-dns-proxy on OpenWrt auto-configures dnsmasq on install: it adds itself as the upstream DNS server and restarts dnsmasq. No manual dnsmasq configuration is needed for the basic DoH setup.

The configure task only needs to:

1. Install https-dns-proxy (with retries per rule 4)
2. Optionally configure specific DoH providers via UCI
3. Verify DNS resolution works through the proxy

WAN/LAN bridge ordering

openwrt_vm orders bridges so the WAN bridge is always net0/eth0:

_ordered_bridges: [_wan_bridge] + (proxmox_all_bridges | difference([_wan_bridge]) | sort)

Previous bug: alphabetical bridge sorting made vmbr0 always WAN. When the modem was on vmbr0, the Proxmox GUI became unreachable from LAN nodes.

Two-phase restart pattern

OpenWrt needs two network restarts because the LAN IP changes mid-run:

Phase 1 (WAN + LAN ports, keep default LAN IP):

1. Configure WAN device, LAN bridge ports via UCI
2. uci commit → detached script (firewall → dnsmasq → network → firewall → dropbear)
3. Pause 30s for services to stabilize
4. Migrate bootstrap IP from WAN bridge to LAN bridge on Proxmox
5. Wait for SSH on LAN bridge, wait for WAN default route
6. Restart firewall again (zone rebinding after interface change)
7. Install packages (opkg) while connectivity works

Phase 2 (final LAN IP + DHCP + WAN MAC):

1. Set final LAN IP, netmask, DHCP params, and WAN MAC (if configured) via UCI
2. uci commit → detached script (firewall → dnsmasq → network → firewall → dropbear)
3. Pause 30s, clean up bootstrap IP

The split is necessary because changing the LAN IP in Phase 1 would break SSH mid-configure.

Detached restart scripts

The detached script pattern survives SSH disconnects caused by network restarts:

printf '#!/bin/sh\nsleep 1\n/etc/init.d/firewall restart\n/etc/init.d/dnsmasq restart\nsleep 2\n/etc/init.d/network restart\nsleep 5\n/etc/init.d/firewall restart\n/etc/init.d/dropbear restart\nrm -f /tmp/_restart_net.sh\n' \
  > /tmp/_restart_net.sh && chmod +x /tmp/_restart_net.sh && \
  start-stop-daemon -S -b -x /tmp/_restart_net.sh

Previous bugs:

• Script omitted firewall restart → stale zones → opkg update got EPERM.
• Firewall + dnsmasq were restarted synchronously before the detached script → SSH connection killed because WAN zone rules (input REJECT) were applied to the bootstrap SSH path.

Bootstrap connectivity

To reach OpenWrt at its default 192.168.1.1 during initial setup:

1. Add a temporary IP (192.168.1.2) to the WAN bridge on Proxmox
2. SSH through ProxyJump via the Proxmox host
3. After Phase 1 network restart, OpenWrt's LAN moves to non-WAN bridges
4. Remove bootstrap IP from WAN bridge, add to LAN bridge
5. After Phase 2, clean up bootstrap IP entirely

Previous bug: we excluded vmbr0 and tried to connect through vmbr1 (which had no IP in OpenWrt's default config).

Proxmox LAN management IP

When OpenWrt is the primary router, the Proxmox host needs a predictable IP on the LAN bridge so the GUI is reachable from leaf nodes:

1. Compute LAN IP from OpenWrt's LAN subnet + offset (default .2)
2. ip addr add on the LAN bridge (immediate, current session)
3. Upgrade the LAN bridge in ansible-bridges.conf from inet manual to inet dhcp
4. Add a DHCP static reservation on OpenWrt mapping the Proxmox host's LAN bridge MAC to the computed IP
5. On reboot: the DHCP client on the LAN bridge requests an IP, OpenWrt always assigns the reserved one
6. Remove any stale LAN-subnet IPs from non-LAN bridges to prevent routing conflicts
7. Remove any separate ansible-proxmox-lan.conf (superseded by bridges.conf DHCP)
8. Write .state/addresses.json with both the management IP and the new LAN IP
9. Probe original management IP — if unreachable (topology changed), update ansible_host via add_host

NEVER use a separate config file with iface <bridge> inet dhcp — it conflicts with the inet manual stanza in the bridges config and ifreload -a won't start the DHCP client. ALWAYS modify the bridge stanza in-place.

NEVER leave stale IPs on non-LAN bridges in the same subnet as the LAN bridge. Two routes for the same /24 on different bridges causes the kernel to use the wrong interface, breaking all LAN VM connectivity.

Auto-subnet selection

To avoid collisions between the WAN subnet and the OpenWrt LAN subnet:

1. Detect the upstream gateway prefix from the Proxmox host's default route
2. Pass it to OpenWrt via add_host as upstream_wan_prefix
3. Iterate openwrt_lan_subnet_candidates and pick the first whose prefix differs from WAN

State file for cross-run IP discovery

build.py probes PRIMARY_HOST before running Ansible. If unreachable, it reads .state/addresses.json for cached alternative IPs. This handles cable-swap scenarios where the original management IP is no longer routable.

The state file is written by openwrt_configure and cleaned by both cleanup playbooks. It is gitignored.

WAN MAC conflict detection

Before applying a cloned WAN MAC, the build runs a three-layer conflict check:

1. Exact MAC in ARP table (/proc/net/arp): catches direct L2 duplicates
2. EUI-64 in IPv6 neighbor table (ip -6 neigh): catches SLAAC address collisions where the MAC itself isn't visible but its derived IPv6 address is
3. Gateway OUI match: if the WAN gateway's MAC shares the first 3 bytes (OUI) with the cloned MAC, the devices are almost certainly from the same router — a conflict waiting to happen

If any check triggers, the MAC is saved to /etc/openwrt_wan_mac_deferred and NOT applied to UCI. The build also deploys /etc/init.d/wan_mac_apply — an OpenWrt init script (START=99) that runs on every boot. It re-runs the same three-layer conflict detection and, if the conflict is gone (old router removed), applies the MAC via UCI and restarts the network automatically. No manual intervention required.

Previous bug: WAN MAC 08:B4:B1:1A:63:08 was applied while the old router (LAN MAC 08:B4:B1:1A:63:09, same OUI) was still on the segment. IPv6 SLAAC generated the same global address → DAD failure → corrupted network stack → wget EPERM, opkg failures, DNS timeouts. ICMP still worked, making the root cause non-obvious without dmesg diagnostics.

The deferred file is cleaned at the start of each run (rm -f) for idempotency. The VM destruction during cleanup also removes it. The init script self-cleans by removing the deferred file after applying the MAC.

OpenWrt Mesh LXC (satellite nodes)

Mesh satellite nodes (wifi_nodes:!router_nodes) run OpenWrt in a privileged LXC container instead of a VM. This allows WiFi management via WDS AP/STA without requiring PCIe passthrough (IOMMU/VT-d). The container receives the host's WiFi PHY via iw phy <phy> set netns <pid> (network namespace move).

Key differences from the VM pattern:

• No routing — mesh containers are NOT routers. They run WDS STA only.
• Uses the OpenWrt rootfs tarball (openwrt-*-rootfs.tar.gz), not the VM disk image.
• Must be privileged (unprivileged: false) for the PHY namespace move.
• Must set --ostype unmanaged because Proxmox cannot auto-detect OpenWrt.
• Container readiness uses ls / (not hostname, which is absent in BusyBox).
• lxc_ct_skip_debian_cleanup: true to avoid dpkg operations on OpenWrt.
• A Proxmox hookscript re-moves the WiFi PHY after container restarts.

WiFi PHY handling:

1. Load common WiFi kernel modules (iwlwifi, ath9k, etc.) on the host.
2. Detect PHYs in /sys/class/ieee80211/.
3. If no PHYs found, hard-fail. All wifi_nodes are expected to have WiFi. Missing WiFi usually means stale vfio-pci bindings from a previous run or missing firmware.
4. Move PHYs into the container's network namespace.
5. Deploy hookscript for persistence across reboots.

IMPORTANT: Detect WiFi radios inside LXC containers with iw phy (netlink), NOT ls /sys/class/ieee80211/ (sysfs). LXC containers bind-mount the host's sysfs, which doesn't reflect network-namespace-specific entries like WiFi PHYs. iw phy queries the kernel via netlink and correctly sees PHYs moved into the container's network namespace. The iw package must be pre-installed in the custom image or via opkg install iw. Previous bug: ls /sys/class/ieee80211/ inside the container returned empty despite a successful iw phy set netns — sysfs showed the host's view.

IMPORTANT: NEVER modprobe WiFi modules inside the container via pct_remote. modprobe inside the container runs on the HOST kernel (containers share the kernel). If the module reloads, the new PHY appears in the HOST namespace, not the container namespace — effectively un-doing the PHY namespace move. Modules MUST be loaded on the host BEFORE the container is created and the PHY moved. Previous bug: modprobe iwlwifi inside the container via pct_remote caused the PHY to revert to the host namespace. WiFi detection inside the container then found zero radios despite a successful namespace move.

IMPORTANT: proxmox_pci_passthrough cleans stale vfio bindings on non-router hosts. If WiFi was previously bound to vfio-pci, the role removes blacklist-wifi.conf and vfio-pci.conf, unbinds devices, and reloads drivers. Without this cleanup, WiFi PHYs are invisible to the mesh role. Previous bug: mesh1 WiFi was bound to vfio-pci from a prior test cycle. /sys/class/ieee80211/ was empty despite the hardware being present.

IMPORTANT: After the WiFi PHY is namespace-moved into the container, OpenWrt does NOT auto-generate /etc/config/wireless. The configure role MUST run wifi config inside the container to generate the wireless configuration from detected hardware BEFORE any uci set wireless.radio* commands. Without this step, uci set wireless.radio0.disabled=0 fails with uci: Invalid argument because the radio0 section doesn't exist. Previous bug: uci set wireless.radio0.disabled=0 failed on both mesh1 and mesh2. The PHY was detected by iw phy (found phy0), but the UCI wireless config had no matching radio0 section because the PHY was moved into the namespace after the container booted.

Container networking follows host topology:

• LAN hosts (router_nodes, lan_hosts) → OpenWrt LAN subnet, LAN bridge.
• WAN hosts → proxmox_wan_bridge, ansible_default_ipv4 subnet, DNS 8.8.8.8.
• IP offset +200 for WAN containers to avoid collisions with LAN containers.

Permanent diagnostics

The OpenWrt build includes diagnostic tasks at two key milestones:

Bootstrap diagnostics (openwrt_vm, after SSH bootstrap):

• VM status, bridge layout, bootstrap IP presence, dmesg errors

Phase 1 diagnostics (openwrt_configure, after WAN route + firewall restart):

• WAN route, WAN IP, LAN IP, DNS resolvers, firewall status, dmesg errors

Final diagnostics (openwrt_configure, end of build):

• VM status, onboot/startup config, LAN bridge IP, management config presence

These run on EVERY build. When a build fails, the diagnostic output from the last successful milestone narrows the failure window.

Detached script verification

Detached scripts report "success" when they launch — NOT when they complete. NEVER trust the launch result. ALWAYS verify the expected outcome after the pause:

1. wait_for on SSH port (proves dropbear restarted)
2. wait_for on WAN default route (proves network restarted)
3. Firewall restart task (proves firewall can be restarted = it's running)

If the detached script fails silently, these verification steps catch it.

Previous bug: detached script launched successfully but the firewall restart inside it failed. The pause completed, and subsequent tasks got EPERM because the firewall zones were stale. The verification pattern (wait_for + explicit firewall restart) would have caught or self-healed this.