| name | image-management-patterns |
| description | Image management and local storage patterns for VM and LXC images. Use when managing VM images, handling custom image builds, or planning image storage strategies. |
Image Management Rules
Local Image Storage Requirement
- ALL VM images and LXC templates are stored locally in the
images/ directory (gitignored) and uploaded to the Proxmox host during provisioning. This is a deliberate design decision, not a convenience shortcut.
Why Local Images
-
No internet dependency on Proxmox host: The host may not have internet access (e.g., after cleanup destroys the router VM). Enterprise repos may be unreachable without a paid subscription.
-
Reproducibility: Pinning exact image versions locally ensures every build uses the same image. Remote repos can remove or rename versions.
-
Speed: Local uploads are faster than downloading from the internet, especially on slow WAN links.
-
Future self-hosting: Images can be served from a local NAS or HTTP server for multi-node deployments.
Directory Layout
-
images/ (gitignored)
├── pihole.version Sidecar: "1.0.0"
├── pihole-1.0.0-debian-12-amd64.tar.zst Built image
├── router.version Sidecar: "1.0.0"
├── openwrt-router-1.0.0-x86-64-combined.img.gz Built image
├── mesh.version Sidecar: "1.0.0"
├── openwrt-mesh-1.0.0-x86-64-rootfs.tar.gz Built image
└── ...per-service .version + image file...
-
NEVER commit images to git. The images/ directory is listed in .gitignore.
The .version sidecar files are also gitignored — they are created by
build-images.sh and read by Ansible via lookup('file') in
group_vars/all.yml. Document the expected image filename and download URL
in role defaults and in docs/architecture/.
Image Versioning (Sidecar Pattern)
7b. Each image target has a images/<service>.version sidecar file containing
a single semver string (e.g., 1.0.0). This replaces the old centralized
manifest.json which suffered from race conditions during parallel builds.
- `build-images.sh` reads the current version from the sidecar, bumps the
patch number, builds the image with the new version in the filename,
bakes the version into `/etc/image_version` inside the image, and writes
the new version back to the sidecar.
- `group_vars/all.yml` reads each sidecar via `lookup('file')` to derive
template paths and version variables for Ansible roles.
- At runtime, containers report their baked `/etc/image_version` via
heartbeat to the Node Manager, which stores it in `HostStateStore`.
- The Node Manager exposes `GET /api/images/versions` for version queries.
- The SuperManager aggregates all Node Managers via `GET /api/fleet/versions`.
Version flow: sidecar → image filename → `/etc/image_version` → heartbeat →
Node Manager → SuperManager.
Previous bug: centralized `manifest.json` with file locking caused race
conditions when building images in parallel across 6 hosts. Each host
read-modify-wrote the same file. Replaced with per-image sidecars that
are independent and lock-free.
NEVER Patch Running Containers (CRITICAL)
-
When a container is missing a package, binary, or config baked into the
image: NEVER install/fix it on the running container via pct exec,
apt install, opkg install, or manual file edits.
ALWAYS follow this workflow:
- Update the build script (
build-images.sh) or image source files
- Rebuild the image IN PARALLEL across hosts:
./build-images.sh --host <ip> --only <target>
- Redeploy:
molecule converge (or molecule test for clean state)
Runtime patches are fragile — they're lost on container recreation, not
tracked in version control, and create drift between the image definition
and the running state. The image IS the source of truth.
Previous bug (2026-04-09): Router VM was missing openssl-util and
batman_trigger.sh. Initial instinct was to opkg install on the
running VM. The correct fix: add openssl-util to ROUTER_PACKAGES
in build-images.sh, rebuild the router image, and redeploy via
molecule converge. Same bug occurred earlier with mesh containers
missing openssl-util — same fix pattern.
NEVER Add Legacy Image Fallbacks (CRITICAL)
8b. NEVER add conditional "if baked content not found, deploy at runtime"
logic in configure roles. This violates the bake-not-configure principle.
If the image lacks something, the answer is ALWAYS to rebuild the image.
There is no such thing as an "old image" in normal operation — images
are always rebuilt before running molecule test.
The `proxmox_lxc` role has a version-mismatch detection system that
auto-rebuilds containers when the image version changes. This means
a fresh image with a version bump automatically produces fresh containers
on the next converge. No fallback code needed.
Previous bug (2026-04-11): Agent modified 8 services in build-images.sh,
then ran molecule test WITHOUT rebuilding images. HA configure failed
because `homeassistant-compose.service` didn't exist in the old image.
Agent added a runtime fallback (check if file exists, deploy via heredoc
if not). User correctly rejected it — the fix was to rebuild the image,
not to add dead fallback code that violates architecture.
Parallel Image Builds (REQUIRED)
8c. You have 6 Proxmox hosts. Build images IN PARALLEL across them.
build-images.sh --host <ip> --only <target> builds a single image
type on a single host. Run multiple simultaneously:
```bash
./scripts/build-images.sh --host $PRIMARY_HOST --only router &
./scripts/build-images.sh --host $AI_HOST --only pihole &
./scripts/build-images.sh --host $MESH_2_HOST --only wireguard &
wait
```
Each image build needs one host as a build environment (creates a temp
container, installs packages, captures template). Different images can
build on different hosts simultaneously. The only constraint is that
each host can only run one build at a time.
Custom Images via Image Builder
-
build-images.sh uses the OpenWrt Image Builder to create pre-configured images with packages pre-installed and UCI defaults baked in. This eliminates runtime opkg install and resolves firewall/networking conflicts in LXC containers.
-
Per the project's "Bake, don't configure at runtime" principle, custom images are REQUIRED. Provision roles verify the image exists and hard-fail with an actionable message if missing:
- name: Fail if image is missing
ansible.builtin.fail:
msg: "Image not found: {{ image_path }}. Run ./build-images.sh to build it."
when: not (image_stat.stat.exists | default(false))
Upload Pattern for VMs
-
- name: Upload image to Proxmox
ansible.builtin.copy:
src: "{{ openwrt_image_path }}"
dest: "/tmp/openwrt-upload"
mode: "0644"
when: not vm_exists | bool
- name: Decompress gzip image
ansible.builtin.command:
cmd: gunzip -f /tmp/openwrt-upload
when:
- not vm_exists | bool
- openwrt_image_path.endswith('.gz')
Adding a New Image
- Process for adding a new image:
- Download the image to
images/ (use wget, browser, or pveam locally)
- Add the path variable to
group_vars/all.yml
- Reference the variable in the provision role's defaults and tasks
- Document the download URL and expected checksum in the role README or architecture doc
- Add the filename to
Before running tests in the ansible-testing skill
VA-API Driver Portability
-
Image builds for services that use the iGPU for hardware acceleration (Jellyfin, Kodi, Moonlight) SHOULD include BOTH Intel and AMD VA-API driver packages. At runtime, only the matching driver loads.
-
Intel: intel-media-va-driver + vainfo
AMD: mesa-va-drivers + vainfo
-
This avoids rebuilding images when a container is moved to different hardware. The configure role reads igpu_vendor to set LIBVA_DRIVER_NAME appropriately.
From-Source Compilation in Image Builds
-
When upstream packages don't provide x86-64 Debian binaries (e.g., moonlight-embedded only has armhf packages for Raspberry Pi), compile from source inside a build container. Pattern:
- Install runtime deps FIRST (
apt-get install). Apt marks these as manually installed.
- Install build-only deps (cmake, gcc, -dev packages) in a SECOND
apt-get install.
- Clone, compile,
make install.
apt-get purge build-only packages, then apt-get autoremove. Runtime deps survive because they're marked manual.
-
ALWAYS include runtime versions of FFmpeg / SDL2 / etc. in step 1. If libavcodec59 is only pulled in as a transitive dependency of libavcodec-dev, autoremove deletes it after purging the -dev package.
- Previous bug:
libavcodec59 and libavutil57 were autoremoved because they were only transitive deps of the -dev packages. Fix: add them to the first (runtime) apt-get install block.
-
For LXC containers without X11, pass -DENABLE_X11=OFF (or equivalent cmake/configure flag) when compiling applications that optionally link X11/EGL/GLES. Without this, the binary links against libraries that get autoremoved.
- Previous bug: moonlight-embedded linked
libEGL.so.1 at compile time. After autoremove, the binary failed to load. Fix: cmake -DENABLE_X11=OFF eliminates the dependency entirely.
-
Verification in build scripts MUST use full paths for binaries installed to /usr/local/bin/. pct exec uses a restricted PATH (/sbin:/bin:/usr/sbin:/usr/bin) that excludes /usr/local/bin/.
- Previous bug:
which moonlight via pct exec failed because /usr/local/bin/ isn't in PATH. Fix: test -x /usr/local/bin/moonlight.
-
Allocate sufficient build container resources for compilation: 4GB disk (2GB too small for build deps + FFmpeg), 1024MB RAM, 2 cores.
Windows VM Images
-
Windows images are built via unattended install on the Proxmox host: create temp VM, boot Tiny11 ISO + virtio-win ISO + answer ISO, wait for Guest Agent + post-install marker, export disk via qemu-img convert to qcow2. Build VMID 992 is reserved for this.
-
Windows qcow2 images are 8-18 GB. Upload to /var/tmp/ (real disk), NOT /tmp/ (tmpfs, typically ~7.8 GB). Same applies to qemu-img convert output during the build.
-
EFI disks on LVM-thin storage MUST use format=raw. The qcow2 format is unsupported and causes build failures.
-
Post-install validation: the build script MUST hard-fail if the post-install marker file is missing after timeout. Proceeding without it produces an incomplete image (no SSH server, no Guest Agent, no software) that fails silently during deployment.
