| name | molecule-performance |
| description | Molecule test performance optimization patterns. Template caching, NTP sync, pct_remote overhead, apt cache, selective rebuilds. |
Molecule Performance Optimization
The full 4-node molecule test takes ~13-14 minutes. Most time goes to template uploads, NTP sync, pct_remote overhead, and SSH round trips in verify. Apply these rules to avoid wasting time on every test run.
Template Caching
NEVER delete templates in molecule cleanup. Keep them cached on Proxmox hosts. Template deletion forces re-upload of ~820MB across 4 hosts on every run.
Template deletion is only valid in production cleanup behind [full-restore, clean] tags.
NTP Sync Optimization
Check clock skew BEFORE running the full NTP burst sequence. Use chronyc -n tracking | awk '/System time/{print $4}' to get skew in seconds.
Only sync when skew > 30s. The NTP sync takes ~7s per host. With 4 hosts × 3 sync points, that's 84s wasted when clocks are accurate.
pct_remote Task Minimization
Each pct_remote task opens a new SSH connection and takes 15-60 seconds. MINIMIZE tasks in configure roles. Base system config that is identical across all containers belongs in the image, NOT the configure role.
Example: Moving a systemd override from configure role (3 tasks via pct_remote) to the image saved 38% of per-feature test time.
Fleet API Is the PRIMARY Health Check (No SSH Fallback)
The fleet readiness API (/api/fleet/ready, /api/container/{id}/ready)
is the SOLE mechanism for verifying container and service health. The
heartbeat system provides continuous liveness monitoring through the 4-tier
chain: Container → NodeManager → ClusterManager → SuperManager.
- NEVER use
pct exec ... systemctl is-active to check service health
- NEVER add SSH fallback paths for when the fleet API is unavailable
- If the fleet API reports a service as not ready, the service IS not ready
- If the fleet API itself is unreachable, the heartbeat watchdog kills the
entire Ansible run within 5 seconds — there is nothing to fall back TO
SSH/pct exec remains relevant ONLY for:
- Hypervisor-side checks (
pct config, qm config) — host access required
- Cross-service L3 integration (send log, check DNS) — proves connectivity
- Post-failure diagnostics (
journalctl inside containers) — debugging only
See the manager-api-pattern skill for the fleet health verification pattern.
Verify Phase Optimization
Consolidate pct config reads:
ansible.builtin.command:
cmd: pct config {{ ct_id }}
register: _ct_cfg
ansible.builtin.assert:
that: "'onboot: 1' in _ct_cfg.stdout"
Use fleet API for service health (not pct exec):
ansible.builtin.uri:
url: "{{ callhome_server_local }}/api/fleet/ready?services=wireguard"
return_content: true
register: _fleet_ready
ansible.builtin.uri:
url: "{{ callhome_server_local }}/api/container/wireguard/ready"
return_content: true
register: _wg_health
Batch pct exec for hypervisor-side config validation only:
ansible.builtin.command:
cmd: pct config {{ ct_id }}
register: _ct_cfg
Wait/Pause Tuning
- OpenWrt detached restart scripts: 20s pause (not 30s)
- OpenWrt VM first boot:
delay: 10, timeout: 120 (not 15/180)
- LXC container networking:
delay: 3 (not 4)
- WiFi PHY detection:
delay: 3 (not 5)
- Verify-phase SSH waits: reduce delay/timeout for services already confirmed running from converge
Apt Cache Configuration
Set cache_valid_time: 86400 (24h) for apt tasks, not 3600 (1h). Test machines rarely have stale packages.
Selective Image Rebuilds
Use ./build-images.sh --host <ip> --only <target> to rebuild a single image. Full rebuilds take ~15 min; selective rebuilds take ~2-3 min.
Every service MUST have a custom image with ALL packages baked in. ZERO configure roles should install packages at runtime.
Build images IN PARALLEL across 6 hosts. Never serialize image builds when you have 6 available machines. The fast iteration loop is: fix build script → rebuild 1 image (~2 min) → per-feature test (~5 min) → loop until clean → full E2E once.
Host-Level Services Are Deployable Infrastructure
Proxmox hosts are bakeable targets. Host-level systemd units (manager-api-proxy.service, supermanager-relay.service), iptables rules, and kernel parameters are all infrastructure that Ansible deploys during provisioning. They are tested the same way as container services — if a host service breaks, it shows up in molecule verify.
NEVER use nohup ... & for persistent host services. Background processes die when SSH sessions close. ALWAYS deploy systemd units with Restart=always.
Previous bug (2026-04-12): socat relays for 4-tier heartbeat chain were started with nohup socat &. Both died when Ansible's SSH ControlMaster session closed mid-E2E, causing 0 of 6 hosts on the SuperManager. Fix: systemd units persisted through the entire test and survived reboots.
pct_remote gather_facts Avoidance
NEVER use gather_facts: true on plays targeting pct_remote dynamic groups unless the configure role actually references ansible_* facts. The pct_remote connection plugin pipes the setup module's JSON through pct exec and SSH. Hosts with many block devices (LVM thin pool, loop devices, device-mapper) generate ansible_devices output that exceeds the buffer, causing "Module result deserialization failed: No end of json char found".
ALWAYS default to gather_facts: false for pct_remote configure plays. If specific facts are needed, use gather_subset: ['!hardware'] to skip the device-heavy hardware facts.
Previous bug: wireguard-ai via pct_remote — the ai host had 15+ block devices, producing ~12KB of ansible_devices JSON. The full setup module output (~20KB) was truncated by pct exec, causing JSON deserialization failure. Other hosts with fewer devices succeeded.
serial Throttling for Multi-Container pct_remote
When a configure play targets 4+ pct_remote containers, each task opens a NEW paramiko SSH connection. With 4 containers × 8-10 tasks, that's 32-40 SSH connections opening/closing in rapid succession through the same SSH server. This overwhelms sshd and causes "Connection reset by peer", "Broken pipe", and "SSH session not active" errors.
ALWAYS add serial: 2 to configure plays that target 4+ pct_remote containers:
- name: Configure WireGuard
hosts: wireguard
gather_facts: false
serial: 2
tags: [wireguard]
roles:
- wireguard_configure
Previous bug: WireGuard configure ran 4 containers in parallel. Two containers (wireguard-home, wireguard-mesh1) both routed SSH through home's sshd. The rapid connection churn caused "Connection reset by peer (104)" on both. Fix: serial: 2 processes containers in batches, halving simultaneous SSH load.
Service Health Uses Fleet API, Not pct exec
NEVER use pct exec ... systemctl is-active <service> to verify container
health. The heartbeat system already monitors every service continuously.
Use the fleet readiness API instead:
- name: Verify fleet health
hosts: localhost
connection: local
tasks:
- name: Query fleet readiness
ansible.builtin.uri:
url: "{{ callhome_server_local }}/api/fleet/ready?services=netdata"
return_content: true
register: _fleet_ready
retries: 12
delay: 3
until: (_fleet_ready.json.all_ready | default(false))
The pct exec pattern for health checks has multiple problems:
- Opens new SSH connections per container (15-60s overhead each)
- Can cause paramiko hangs that
serial and timeout cannot prevent
- Duplicates what the heartbeat system already knows
Previous bug: Configure Netdata play with pct_remote hung indefinitely on
paramiko SSH handshake — even with serial: 1, timeout = 60, and
host_key_auto_add = True. The fleet API would have returned the answer
in <1 second.
Play Merging
When two verify plays target the same hosts: group with the same gather_facts: setting, merge them into one play to eliminate startup overhead.
Large VM Image Caching
Windows qcow2 images (19 GB) take 10-25 min to upload per host via SSH. Cache the image on the Proxmox host:
- name: Check if image is already cached
ansible.builtin.stat:
path: "{{ upload_dest }}"
get_checksum: false
register: _remote_image
- name: Upload image (skip if cached)
ansible.builtin.copy:
src: "{{ local_image }}"
dest: "{{ upload_dest }}"
when: not (_remote_image.stat.exists | default(false))
Do NOT delete the cached image after import. Only production cleanup removes it.
Previous bug: 43-minute test run spent 25 minutes (60%) on image upload. Caching eliminates this on subsequent runs.
Per-Feature Scenario Scope
Per-feature scenarios should test on a single host (e.g., home only). Multi-node coverage belongs in the full E2E test. This halves test time for services with expensive setup (large images, slow imports).
