add-linux-bootstrap-platform

# Adding a New Linux Distro/Version as a ponyup Bootstrap-Test Target ## Scope Use this skill when adding a Linux distro/version where ponyup needs to provide both `arm64` and `amd64` bootstrap testing (e.g. Alpine 3.24, Ubuntu 26.04 LTS). The procedure encodes the **multi-platform bootstrap-tester pattern** — a single `.ci-dockerfiles/<distro><version>-bootstrap-tester/` directory containing one `Dockerfile` and one `build-and-push.bash` that builds and publishes a multi-arch image (linux/amd64 + linux/arm64) under `ghcr.io/ponylang/ponyup-ci-<distro><version>-bootstrap-tester`. **Out of scope:** - macOS, Windows, any non-Linux platform. - Adding a new OS or new CPU architecture. - Removing a platform, e.g. dropping an old release. Different procedure, not covered here. Per our policy of supporting only the two newest releases of a distro, adding a newer release usually pairs with dropping the now-oldest by hand. ## Guiding facts - **Adds only; the policy is newest-two.** This skill only adds a bootstrap-test target. Our policy is to support the two newest releases of a distro, so adding a newer release usually pairs with dropping the now-oldest — but that drop is a separate, by-hand procedure (not covered here). Don't drop anything as part of this work. - **Both arches by default.** The multi-platform image builds linux/amd64 and linux/arm64 from one Dockerfile. Don't restrict arches without a concrete reason. - **Multi-arch image gets a date-stamp tag** (`YYYYMMDD`). - **Tier 1 vs tier 2 split is an ASK.** Skill describes the current convention as context (latest x86-64 of each distro family in `pr.yml` / tier 1; everything else, including all arm64, in `ponyup-tier2.yml` / tier 2) but does not enforce it. The user picks per arch. - **SSL choice is an ASK.** Skill describes precedent (Alpine entries use `SSL=libressl` + `libressl-dev`; Ubuntu entries use `SSL=3.0.x` + `libssl-dev`) but does not pick. The user picks based on what the new distro packages and what we want to test against. The choice has **three downstream consistency requirements** — they must all agree, or the bootstrap test fails at link time deep in CI: 1. The Dockerfile installs the matching dev package. 2. The `SSL=` value in the bootstrap test job in `pr.yml`/`ponyup-tier2.yml` matches. 3. `.ci-scripts/test-bootstrap.sh` passes the same value through to `ponyup`. - **Mental model.** One multi-arch bootstrap-tester image is published to GHCR with a date-stamp tag. Two CI bootstrap jobs (one per arch) reference that one image by exact tag from their tier file — Docker pulls the matching arch on each runner. The job exercises `ponyup-init.sh` end-to-end against the new platform. ## File inventory New files (a single bootstrap-tester directory with two files, plus a release note): - `.ci-dockerfiles/<distro><version>-bootstrap-tester/Dockerfile` - `.ci-dockerfiles/<distro><version>-bootstrap-tester/build-and-push.bash` - `.release-notes/<distro><version>.md` Existing files to edit: - `.github/workflows/build-bootstrap-tester-image.yml` — one dropdown option + one job block - `.github/workflows/pr.yml` and/or `.github/workflows/ponyup-tier2.yml` — bootstrap test jobs (one per arch; tier file for each is asked) - `ponyup-init.sh` — distro detection case Conditional edit (only if the canonical-example promotion is selected): - `README.md`, `cmd/cli.pony`, `cmd/main.pony`, `cmd/cloudsmith.pony` — bump the canonical example platform string. (`cmd/packages.pony` has a *bare* distro reference in a docstring, not the full canonical-example form; treat separately if you want to refresh it. The grep recipes in step 13 catch both cases.) Conditional edit (only if the new platform is the latest Alpine): - `test/main.pony` — bump the `x86_64-linux-alpine<version>` test fixture ## Before you begin - **Branch base:** start from a fresh `main`. `git checkout main && git pull`. - **Tools:** Docker daemon running (`docker info` succeeds), `gh` authenticated, `git` configured to push to origin. If the Docker daemon isn't running, ask the user to start it — the agent typically can't start daemons itself. - **`gh` token scopes needed:** `workflow` (for dispatching builds) and `read:packages` (for querying GHCR tags). Run `gh auth status` and confirm the `Token scopes:` line includes both. If missing: `gh auth refresh -s workflow,read:packages`. A 403 from the GHCR API in step 9 means missing scope, not a missing image. - **Placeholders in this document:** every `<distro>`, `<version>`, `<tag>`, etc. is a placeholder for substitution. The angle brackets are not part of any command. Example for Alpine 3.24: `<distro>` → `alpine`, `<version>` → `3.24`. - **Long-running waits:** steps 8 and 18 use `gh run watch --exit-status` and `gh pr checks --watch --exit-status`. The build wait is typically 20–40 minutes (multi-arch QEMU) and PR CI is 10+ minutes — both can exceed the agent's Bash tool timeout. Run them with `run_in_background: true` and check status periodically, or poll non-blocking variants (`gh run view <id> --json status,conclusion`, `gh pr checks <num>`). ## Procedure The **structural exemplar** for the directory layout, `build-and-push.bash`, `build-bootstrap-tester-image.yml` job block, and CI bootstrap job entries is the most recent Alpine multi-arch bootstrap-tester. As of writing, that's `alpine3.24-bootstrap-tester`. Before starting, glance at `.ci-dockerfiles/` and pick whichever Alpine multi-arch bootstrap-tester is newest (`alpineX.Y-bootstrap-tester`); use it in place of `alpine3.24-bootstrap-tester` throughout this document. ### 1. Branch ```bash git checkout main && git pull git checkout -b add-<distro>-<version>-bootstrap-platform ``` ### 2. ASK USER: SSL choice Before writing the Dockerfile, surface the SSL decision to the user: > "What `SSL=` value should the new bootstrap test use, and which dev package should the Dockerfile install? Existing precedent: Alpine entries use `SSL=libressl` + `libressl-dev`; Ubuntu entries use `SSL=3.0.x` + `libssl-dev`. The right choice depends on what the new distro packages and what you want to test against." Surface the precedent as context. Don't pick for the user. Wait for the answer before proceeding to step 3. The choice flows through three places that must agree: 1. The Dockerfile's installed dev package (step 3). 2. The `SSL=` value in each bootstrap job in `pr.yml`/`ponyup-tier2.yml` (step 11). 3. `.ci-scripts/test-bootstrap.sh` passes the value through; no edit needed there if the job sets `SSL=` correctly. A mismatch surfaces as a link-time failure deep in CI. ### 3. Create the bootstrap-tester directory ```bash mkdir -p .ci-dockerfiles/<distro><version>-bootstrap-tester ``` **Dockerfile.** Copy the most recent **same-distro-family** Dockerfile and adjust two things: - `FROM <distro>:<version>` — bump the version. - The SSL dev package — match the SSL choice from step 2. Don't enumerate the package list yourself; copy it. Each distro family has accumulated specifics that aren't obvious from the conceptual list (e.g. Ubuntu Dockerfiles install `lsb-release` so `ponyup-init.sh`'s `lsb_release -d` detection works inside the container, plus `ca-certificates` and a `git config --global --add safe.directory` line). Don't trim packages without a concrete reason; the smoke test in step 4 will surface what's actually missing. For Alpine adds, `alpine3.24-bootstrap-tester/Dockerfile` is the same-distro exemplar. For Ubuntu adds, `ubuntu24.04-bootstrap-tester/Dockerfile` is. (The cross-distro structural exemplar — for shape only, not contents — is the most recent Alpine bootstrap-tester.) If adding a distro that isn't Alpine or Ubuntu, verify the base image is multi-arch first: ```bash docker manifest inspect <distro>:<version> | jq '.manifests[].platform' ``` Both `linux/amd64` and `linux/arm64` must appear. **build-and-push.bash.** Copy from the structural-exemplar bootstrap-tester (`alpine3.24-bootstrap-tester/build-and-push.bash`) and change only the `NAME` and the `BUILDER` prefix. The shape: ```bash #!/bin/bash set -o errexit set -o nounset # # *** You should already be logged in to GHCR when you run this *** # NAME="ghcr.io/ponylang/ponyup-ci-<distro><version>-bootstrap-tester" TODAY=$(date +%Y%m%d) DOCKERFILE_DIR="$(dirname "$0")" BUILDER="<distro><version>-builder-$(date +%s)" docker buildx create --use --name "${BUILDER}" docker buildx build --provenance false --sbom false --platform linux/arm64,linux/amd64 --pull --push -t "${NAME}:${TODAY}" "${DOCKERFILE_DIR}" docker buildx rm "${BUILDER}" ``` Make the script executable: `chmod +x .ci-dockerfiles/<distro><version>-bootstrap-tester/build-and-push.bash`. ### 4. Local smoke test Before pushing or dispatching anything, build the Dockerfile locally for amd64 explicitly: ```bash docker build --pull --platform linux/amd64 .ci-dockerfiles/<distro><version>-bootstrap-tester/ ``` `--platform linux/amd64` makes the smoke test deterministic regardless of host architecture. `--pull` forces a fresh base image rather than a stale local cache. This is a smoke test only — don't run `build-and-push.bash` locally (that would push to GHCR with today's date stamp). If the build fails because Docker isn't running or the package manager can't reach the network, that's an environment issue — surface it to the user and stop. The agent typically can't start daemons or fix host networking on its own. If the build fails inside the image (package gone, repo URL invalid, base image not yet published, etc.): **STOP**. Surface the error to the user. Don't retry. Don't "fix" by tweaking the package list without confirmation — that's a design discussion. The remote multi-platform build in step 7 cross-builds arm64 from x86-64 via QEMU. Local x86-64 success is a strong signal but not a guarantee for arm64; the GHA dispatch is the real arm64 verification. ### 5. Update `build-bootstrap-tester-image.yml` Two edits, both mechanical: **a. Add one dropdown option** (alphabetical across the whole list): ```yaml options: ... - <distro><version>-bootstrap-tester ... ``` **b. Add one job block** by copying the structural-exemplar job (`alpine3_24-bootstrap-tester`) and substituting the version. Place the new job alphabetically among the sibling jobs. The job map key replaces `.` with `_` (YAML map keys can't contain `.`): a version of `26.04` becomes `26_04` in the job key only — the `if:` filter value, the `name:` field, and the `bash` script path all keep the dot form. Reference shape: ```yaml <distro><version_underscore>-bootstrap-tester: if: ${{ github.event.inputs.builder-name == '<distro><version>-bootstrap-tester' }} runs-on: ubuntu-latest name: <distro><version>-bootstrap-tester steps: - name: Checkout uses: actions/checkout@v6.0.2 - name: Set up Docker Buildx uses: docker/setup-buildx-action@v4 with: version: v0.23.0 - name: Login to GitHub Container Registry uses: docker/login-action@v4 with: registry: ghcr.io username: ${{ github.repository_owner }} password: ${{ secrets.GITHUB_TOKEN }} - name: Build and push run: bash .ci-dockerfiles/<distro><version>-bootstrap-tester/build-and-push.bash ``` `runs-on: ubuntu-latest` is correct — the multi-platform build cross-compiles arm64 via QEMU on an x86-64 runner. Don't change it. **Dropdown ↔ directory invariant:** the dropdown option string is consumed by `bash .ci-dockerfiles/<option>/build-and-push.bash`. The option string must exactly match the bootstrap-tester directory name. A typo is silent failure (workflow runs `bash` against a non-existent path; only signal is workflow logs). ### 6. Commit and push the branch (push is a prerequisite for dispatch) The workflow_dispatch in step 7 runs the workflow file from the branch on origin, so the branch must be pushed before dispatch. A WIP commit is fine here — it'll be squashed in step 17. ```bash git add .ci-dockerfiles .github/workflows/build-bootstrap-tester-image.yml git commit -m "WIP: add <distro> <version> bootstrap tester" git push -u origin add-<distro>-<version>-bootstrap-platform ``` ### 7. Dispatch the builder workflow from the branch GHA `workflow_dispatch` runs the workflow file from the chosen ref, so the new dropdown option becomes available when targeting the branch. ```bash gh workflow run build-bootstrap-tester-image.yml \ --ref add-<distro>-<version>-bootstrap-platform \ -f builder-name=<distro><version>-bootstrap-tester ``` If dispatch returns 403, the `gh` token lacks `workflow` scope — see "Before you begin." ### 8. Wait for the build to finish The multi-platform build takes ~20–40 minutes (arm64 via QEMU is the slow leg) — well beyond the agent's foreground Bash timeout. Capture the run id first: ```bash RUN_ID=$(gh run list --workflow=build-bootstrap-tester-image.yml \ --branch=add-<distro>-<version>-bootstrap-platform \ --limit=1 --json databaseId --jq '.[].databaseId') ``` Then either: - Run `gh run watch "$RUN_ID" --exit-status` with `run_in_background: true` and wait for the completion notification, or - Poll non-blocking with `gh run view "$RUN_ID" --json status,conclusion --jq '{status, conclusion}'` periodically until `status == "completed"`. If the build fails: **STOP**. Read the failed logs with `gh run view "$RUN_ID" --log-failed` and surface the failure to the user. A common cause is a package or repo change for the new distro — that's a discussion, not something to silently work around. ### 9. Discover the published tag The successful build pushes `ghcr.io/ponylang/ponyup-ci-<distro><version>-bootstrap-tester:YYYYMMDD`. Query GHCR: ```bash gh api -X GET /orgs/ponylang/packages/container/ponyup-ci-<distro><version>-bootstrap-tester/versions \ --jq '[.[].metadata.container.tags[]] | map(select(test("^[0-9]{8}$"))) | max' ``` The `select(test("^[0-9]{8}$"))` filter restricts to date-form tags (rejects `latest` or other future tags); `max` picks the most recent. If the API returns 403, the `gh` token lacks `read:packages` — see "Before you begin." A 403 means missing scope, not a missing image. ### 10. ASK USER: tier placement > "Where should each bootstrap job land — `pr.yml` (tier 1, runs on every PR) or `ponyup-tier2.yml` (tier 2, runs less often)? Pick independently for x86-64 and arm64. Current convention as context: latest x86-64 of each distro family in tier 1; everything else (including all arm64) in tier 2. But this is your call." Surface the convention as context. Don't pick for the user. If the user picks tier 1 for arm64, surface that there's no arm64 Linux bootstrap precedent in `pr.yml` today, then proceed with their choice. ### 11. Add bootstrap test jobs to the chosen tier file(s) You're adding **two jobs total — one per arch**. Pick a same-arch exemplar from the chosen tier file: | Where you're adding | Exemplar to copy | Job shape | |---|---|---| | Tier 1 x86-64 | `x86-64-alpine3_24-bootstrap` (`pr.yml`) | `runs-on: ubuntu-latest` + `container:` block | | Tier 1 arm64 | *no precedent today* (see note below) | — | | Tier 2 x86-64 | `x86-64-ubuntu24_04-bootstrap` (`ponyup-tier2.yml`) | `runs-on: ubuntu-latest` + `container:` block | | Tier 2 arm64, **Alpine add** | `arm64-alpine3_24-bootstrap` (`ponyup-tier2.yml`) | `runs-on: ubuntu-24.04-arm` + manual `docker pull` / `docker run` |