automatedlab-proxmox

name	automatedlab-proxmox
compatibility	Requires the AutomatedLab PowerShell module, Proxmox VE API access, and the QEMU guest agent on the lab virtual machines.
description	Diagnoses AutomatedLab Windows provisioning and remoting failures on Proxmox/QEMU from API, guest-agent, Windows Setup, and WinRM evidence. Separates deploy-time AppX cleanup from unusable generalized templates. USE FOR: AutomatedLab Proxmox, Install-Lab Proxmox hang, Windows could not start the installation process, IMAGE_STATE_UNDEPLOYABLE, IMAGE_STATE_COMPLETE, pre-sysprepped template, generalized template, specialized golden image, BiExportStoreAlterationsToEfi, c000000d, qm template, Sysprep AppX 0x80073cf2, guest-exec-status, qmpstatus, New-LabPSSession port is closed, NetworkAdapterBindingCorrected, stale AutomatedLabCore/AutomatedLabWorker. DO NOT USE FOR: Hyper-V lab deployment (use automatedlab-deployment), generic WinRM diagnosis (use winrm-troubleshooting), generic job monitoring (use long-running-job-monitor).

AutomatedLab Proxmox troubleshooting

Diagnose Proxmox-backed AutomatedLab failures from correlated evidence, then make the smallest guarded change and verify it against a live Windows guest.

When to use

• Install-Lab stalls or emits only progress dots on Proxmox.
• Windows Server succeeds but a Windows client fails Sysprep or OOBE.
• A clone's first boot shows an Install Windows dialog saying "Windows could not start the installation process," or the supplied template may be pre-generalized or stuck in a failed Sysprep state.
• New-LabPSSession reports a closed port during or after a guest restart.
• QEMU guest-exec starts a process but its final result is unknown.
• A patched AutomatedLab module appears installed but a live shell runs old code.
• A Proxmox task reports a warning even though the VM may have reached its intended state.

Outcome

Produce a timestamped root-cause account, a focused regression guard, and live evidence that the affected Windows guest reaches the required protocol and final state without destroying diagnostic evidence.

Dependencies and safety

• Require access to the AutomatedLab source and installed module paths.
• Require read access to the Proxmox API and QEMU guest agent.
• Use an alternate channel when WinRM is unavailable: QEMU guest agent, Proxmox API, console, or another hypervisor-side control plane.
• Preserve failed VMs and Windows setup logs until the cause is established.
• Never start a supplied Proxmox template directly. Inspect its configuration without power-on, then boot a throwaway clone for image-state evidence.
• Never classify a VM from orchestrator output alone.
• Use long-running-job-monitor for the monitoring mechanics and status format.

Workflow

1. Pin the failing run

Record the exact command, process start time, module paths and versions, lab name, VM IDs, guest addresses, and log paths. Determine whether the shell was already running when module files were replaced.

2. Build a multi-plane timeline

Collect timestamped state from each independent plane:

1. Orchestrator: process liveness, current AutomatedLab phase, warning/error stream, and terminal marker.
2. Hypervisor: Proxmox task result, VM status, QEMU qmpstatus, uptime, and restart time.
3. Guest agent: availability, command PID, guest-exec-status, exit code, and signal state.
4. Template image: API configuration plus a throwaway clone's ImageState and Sysprep/Panther logs; never power on the original template.
5. Windows setup: Panther, Sysprep, setup activity, and setup error logs.
6. Required protocol: TCP 5985/5986, Test-WSMan, SSH, HTTP, or another service the next orchestration step actually consumes.

Do not infer one plane from another. A running QEMU process does not prove Windows readiness; ping does not prove WinRM readiness.

3. Classify before changing code

• Process alive with target state or a real phase milestone changing: WORKING.
• Target state and real phase milestone unchanged beyond the phase threshold: STALLED, even when synthetic heartbeats continue. Collect evidence before stopping only the orchestrator process.
• QEMU command PID returned but no final status: process creation only; poll guest-exec-status.
• VM restarted and ping returned but the required port is closed: current boot is not ready; keep remoting-dependent work behind a readiness gate.
• AutomatedLab reached its own Sysprep and logs 0x80073cf2: classify this as the deploy-time per-user AppX mismatch handled by guarded cleanup.
• A clone enters OOBE before answer-file injection, shows "Windows could not start the installation process," or reports IMAGE_STATE_UNDEPLOYABLE: classify the supplied template as pre-generalized or incompletely generalized. This is a template-build failure, not an AppX-cleanup failure.
• Windows setup logs contain the first concrete failure: fix that guest-side cause rather than adding retries to the orchestrator.

When template deployability is in doubt:

1. Read the powered-off template configuration through the Proxmox API.
2. Clone it to a throwaway VM and boot only the clone.
3. Use the QEMU guest agent to read Setup\State\ImageState and the Sysprep Panther logs.
4. Destroy the probe clone after evidence capture and confirm the original template remained untouched.

For command/result semantics and restart ordering, read references/evidence-and-readiness.md.

4. Fix the smallest owning path

Move to the nearest function that directly controls the failing ordering, state transition, or command result. Avoid suppressing warnings, blanket retries, and catch-and-continue behavior.

Maintain these invariants:

• A successful QEMU guest-exec request means process creation, not success.
• AutomatedLab owns /generalize /oobe; its Proxmox template must be a specialized, non-sysprepped golden image with a usable ImageState.
• A deliberate restart invalidates every readiness observation from the prior boot.
• Remoting-dependent repair runs only after readiness for the current boot.
• Initialization state is a bitmask; add flags with bitwise OR.
• A Proxmox task warning is nonfatal only when a fresh state query proves the intended postcondition.
• Reload changed dependency modules explicitly in long-lived PowerShell sessions.
• Capability guards verify the required behavior, not a correlated proxy.

For Windows Sysprep/AppX diagnosis, read references/windows-sysprep.md. For build, reload, and installed-artifact checks, read references/module-development.md.

5. Guard the behavior

Write focused tests that fail without the change. For restart/network repair, cover at least:

• call order: start -> wait -> repair;
• preservation of existing initialization flags;
• idempotent skip when repair is already recorded;
• serialized metadata forms returned by the live provider.

Do not weaken nonblocking -Wait:$false semantics implicitly. If a no-wait path remains exposed to the race, record it as a residual risk or design a separate contract change.

6. Verify live

Use a controlled restart of one existing test guest before a full deployment:

1. Back up VM metadata.
2. Clear only the flag needed to force the target path.
3. Restart through the real AutomatedLab command with readiness enabled.
4. Observe the interval where VM-running or ping may precede the required protocol.
5. Require a zero exit marker and absence of the original warning.
6. Confirm all expected metadata flags remain set.
7. Open a direct session and verify the remote service state.
8. Rebuild from current source and compare the artifact hash with the installed module that passed the live test.

For a template-build failure, replace the restart test with a fresh throwaway clone probe, rebuild the template without pre-running Sysprep, then require AutomatedLab's own Sysprep gate to reach IMAGE_STATE_COMPLETE.

Edge cases

• Metadata may serialize a flags enum as comma-separated names. Cast it back to the enum before integer masking; never cast the combined string directly to Int32.
• A control-channel disconnect is not guest-process failure. Reconnect through an independent plane and read the remote process result.
• If the full repository build is unavailable, derive the concatenation order from the repository build task. Do not assume an order from memory.
• Keep plaintext lab credentials out of reusable scripts and skill examples.
• Preserve unrelated user edits and running VMs while testing a module patch.

Anti-rationalization

Rationalization	Reality
"The VM is running, so Windows is ready."	Hypervisor power state does not prove guest protocol readiness.
"Ping works, so WinRM is broken."	Ping commonly returns before TCP 5985 after reboot. Probe the protocol and timeout boundary.
"guest-exec returned a PID, so Sysprep succeeded."	A PID proves only process creation. Require terminal status and exit data.
"The tags and hardware are correct, so the template is fine."	Template discovery is not deployability. Read ImageState from a throwaway clone.
"Import-Module -Force loaded the patch."	A parent reload can retain already-loaded dependency modules. Verify command source and behavior.
"The warning is harmless."	Prove the intended postcondition with a fresh target query before accepting it.

Red flags

• Repair or remoting starts immediately after a deliberate restart.
• A task warning is ignored without a fresh VM/QEMU state query.
• A metadata assignment replaces a flags enum instead of OR-ing a flag.
• A live test reports success without a direct protocol session.
• Installed and source module hashes differ after the claimed verification.
• The supplied template is powered on instead of a disposable probe clone.
• The failed VM is deleted before setup and guest-agent evidence is collected.

Verification

Before reporting done, require:

• focused tests green in an isolated PowerShell process;
• changed PowerShell files parse with zero errors;
• no new ScriptAnalyzer findings relative to the repository baseline;
• live controlled restart exits zero;
• original warning absent from captured output;
• direct protocol session succeeds after the restart;
• final VM metadata preserves all prior flags;
• installed artifact hash matches the current source build;
• when a template is implicated, its probe clone records ImageState and Panther evidence before teardown, while the original remains untouched.

Evals

Real-session regression prompts and deterministic pass criteria live in notes-evals.md.