chapar-incus-ci-triage

name	chapar-incus-ci-triage
description	Triage Chapar GitHub Actions Incus Spack-environment CI failures, including hpcsim Rocky 9/Rocky 10 builds and future environment workflows. Use when an Incus Spack build fails, succeeds on one Rocky version but not the other, or when inspecting self-hosted runner/build logs.

Chapar Incus CI Triage

Use this playbook for Incus-backed GitHub Actions that build Chapar Spack environments. The current workflow is .github/workflows/incus-spack-build.yml for envs/hpcsim, but future workflows may target other envs/<name> environments. Identify the environment before changing config.

First Principles

• Do not guess from the red/green summary alone. Identify the failed workflow, job, step, commit, inputs, OS, and target environment.
• If the user asks about a currently running or stuck build and artifact roots are locally mounted, use chapar-ci-artifact-watch first to inspect durable logs and output files directly instead of waiting for GitHub web logs.
• Compare Rocky 9 and Rocky 10 when one succeeds and the other fails.
• Prefer durable CI artifacts/logs under the configured run root over GitHub web log summaries when available.
• Do not change spack/ or generated lock/cache directories.
• Keep fixes scoped to the failure class: workflow bootstrap, mount/resource validation, Spack setup, concretization, install, buildcache, module refresh, or promotion.

Required Inspection

git status --short
git log --oneline -5
gh run list --limit 10 \
  --json databaseId,workflowName,displayTitle,status,conclusion,createdAt,updatedAt,headSha,headBranch,event

For the target run:

gh run view <run-id> --json jobs,conclusion,displayTitle,headSha,headBranch,workflowName,event

Look for:

• Which workflow failed.
• Which matrix job failed, e.g. Build hpcsim on rocky10.
• Which environment is targeted, e.g. ENV_PATH=envs/hpcsim.
• Which step failed: bootstrap, mount verification, or the environment build.
• Whether failure duration suggests timeout. Common clues:
  • ~1 hour: old generic CHAPAR_CONCRETIZE_TIMEOUT=3600 guardrail.
  • ~3 hours: Rocky 9/Rocky 10 minimum or older Rocky-wide minimum.
  • ~24 hours: workflow-level timeout or a stuck install.

Logs and Run Outputs

For the generic environment workflow, ci/container-build.sh writes durable logs under:

${CHAPAR_ENV_ROOT}/${OS_NAME}/runs/${RUN_ID}/logs/build.log
${CHAPAR_ENV_ROOT}/${OS_NAME}/runs/${RUN_ID}/commit.txt
${CHAPAR_ENV_ROOT}/${OS_NAME}/runs/${RUN_ID}/spack-version.txt
${CHAPAR_ENV_ROOT}/${OS_NAME}/runs/${RUN_ID}/spack-commit.txt
${CHAPAR_ENV_ROOT}/${OS_NAME}/runs/${RUN_ID}/concrete-envs/

Defaults from the hpcsim workflow are:

ENV_NAME=hpcsim
ENV_PATH=envs/hpcsim
CHAPAR_ENV_ROOT=/resources/chapar/hpcsim
CHAPAR_BUILDCACHE_ROOT=/resources/chapar/cache/buildcache
CHAPAR_CCACHE_ROOT=/resources/chapar/cache/ccache

Future environment workflows should have analogous run roots and must be read from their workflow/build script rather than assumed. If GitHub log download is unavailable, say so and use structured job metadata plus durable build logs if mounted locally.

Failure Classification

Bootstrap Rocky dependencies

Inspect the bootstrap script used by the workflow, currently ci/bootstrap-rocky.sh for hpcsim. Fix only missing RPM/repo setup needed by current CI. Avoid adding heavyweight runtime packages unless the target environment needs them.

Do not fix hpcsim builder failures by adding OS CUDA Toolkit, OS Intel oneAPI compiler/MPI, or GitHub CLI RPM repos/packages. hpcsim CUDA, Intel MPI, and any future Intel compiler dependency should come from Spack unless the user explicitly changes that policy.

Verify resources mount

Inspect the workflow inputs and preparation script. hpcsim release-mode builds use ci/prepare-hpcsim-root.sh; generic Spack-mode builds create the run/cache directories directly. Resources, buildcache, and ccache roots must be NFS (nfs or nfs4) and writable for the OS-specific run/cache directories.

Concretization timeout/failure

Use chapar-spack-solve-debug with the target ENV_PATH. Preserve Chapar policy:

• Do not pin dependency minor/patch versions just to make the solver faster.
• Keep CUDA/GDR transport support for GPU environments.
• Put hpcsim root specs and package constraints in envs/hpcsim/spack.yaml.
• Keep package-wide Python policy to allowed minor versions only; do not force python+tkinter globally.

Install/build failure

Find the package and concrete hash in build.log. Check whether the package is a root or dependency, which compiler/provider it used, and whether the failure is OS-specific.

Buildcache failure

Use chapar-buildcache. Do not auto-import legacy caches.

Module refresh failure

Use chapar-release-helper for hpcsim release-helper failures. Modules must be generated only for explicit environment roots and stay hashless {name}/{version}.

Fix Workflow

1. Inspect current branch and local changes.
2. Identify the target environment and failure class.
3. Make the smallest project-config/CI/release-helper change that addresses the classified failure.
4. Validate locally when possible:

bash -n ci/container-build.sh
bash -n ci/bootstrap-rocky.sh
bash -n envs/hpcsim/release.sh
git diff --check

5. If committing, use chapar-commit.
6. When asked to run CI, dispatch only the affected OS/environment unless a push already triggered the needed workflow.