create-freebsd-dev-env

# Create a FreeBSD dev/test VM for ponyc Stand up a local FreeBSD VM that matches the ponyc tier-3 CI job, so you can build and test ponyc on FreeBSD without GitHub Actions. FreeBSD's tier-3 job exercises paths the clang-on-Linux jobs never do: the embedded-LLD `--static` link recipe, the native sanitizer link path, the dtrace/illumos provider path, and FreeBSD's base libc++/libunwind. So this VM is the authoritative local environment for **FreeBSD-platform** ponyc issues. (It is clang-based — FreeBSD's base compiler is clang, not gcc. For **gcc-specific** issues use the DragonFly skill instead.) Run the repo-relative commands (`rsync` of the source) from your ponyc checkout root. The VM itself lives in a separate directory (`$VMDIR`). **Each fenced `sh` block below is self-contained.** It re-sets `VMDIR`/`FREEBSD_VERSION` (and, where needed, the `SSH` command) at its top, because an agent runs each block as a fresh shell — environment variables and the working directory do NOT carry from one block to the next. Do not "optimize" by setting them once and dropping them from later blocks: an empty `$SSH` would make a `$SSH /bin/sh <<'EOF' ... EOF` heredoc run **on your host** (silently) instead of in the VM. Keep every block whole. Pick your `VMDIR`/ `FREEBSD_VERSION` once and use the same values in every block. ## Gotchas that will bite you (read first) FreeBSD is much simpler to bring up than DragonFly (cloud-init injects the ssh key, so there is no VGA-console automation), but these still matter: - **Per-block fresh shells.** See above: re-establish `VMDIR`/`FREEBSD_VERSION`/`SSH` in every block; never let `$SSH` be empty in front of a `/bin/sh` heredoc. - **You log in as `freebsd`, not `root`.** The cloud-init seed installs your key for the `freebsd` user and sets root's password to `ciroot`. Build and test run **as `freebsd`** (no root needed). Root is only needed twice during setup — the one-time `expect`/`su` bootstrap below installs the packages and configures passwordless `doas`, after which the rare root operation (e.g. loading the dtrace kernel module) goes through `doas`. - **Cloud-init means no console driving.** Unlike the DragonFly skill there is no QEMU `sendkey`/screendump/monitor dance — just boot and wait for ssh. But the **first** boot runs `nuageinit` (key install) and `growfs` (root-fs auto-resize), so ssh is not instant; poll for it (Step 4). - **The root fs auto-grows; there is no growfs command to run.** `qemu-img resize ... 60G` enlarges the disk and the image's first boot grows the root fs to fill it. You do not run `growfs` yourself — the wait-for-ssh loop is what accounts for it. - **FreeBSD's base system has no `bash`.** In-VM scripts must be POSIX `sh` (the two CI smoke scripts already are). Don't reach for bashisms in a `$SSH /bin/sh` heredoc. - **Detach long in-VM builds.** The libs build (LLVM) takes hours; run it `nohup … > /…/x.log 2>&1 &` and poll the log, so an ssh drop doesn't kill it. - **The one-time root bootstrap needs `expect`.** FreeBSD's `nuageinit` supports only `ssh_authorized_keys` + `chpasswd` (not cloud-init's `write_files`/`runcmd`), so the package install and `doas.conf` setup are driven through `su -m root` with `expect`, exactly as CI does it. After that, `doas` is on disk and persists across reboots. ## Step 0 — verify prerequisites (do NOT assume they're installed) This skill needs, on the host: - an existing ponyc checkout (you run the `rsync` from it) - hardware-accelerated virtualization for QEMU (KVM on Linux, HVF on macOS) - enough free disk for the VM's sparse qcow2 disk (provisioned at 60 GB nominal; it grows only as used — the libs build plus a debug build take the qcow2 to ~12 GB, and a release build on top adds more), ~1 GB for the compressed image, and network access to download.freebsd.org - `qemu-system-x86_64` and `qemu-img` - `cloud-localds` (from cloud-image-utils) or `genisoimage` to build the cloud-init seed image - `expect` (drives the one-time `su` root bootstrap) - `xz`, `rsync`, `git`, `curl`, `python3` - an OpenSSH client (`ssh`, `scp`, `ssh-keygen`) Note this list is shorter than the DragonFly skill's: FreeBSD needs no `bsdtar`/`bunzip2` (no ISO header mining) and no PPM-to-PNG converter (no console screendumps). Check that every required tool is present. Do not assume the environment has them, and do not install anything yourself — installing them is the user's call (it needs privilege, and the package names vary by OS). If anything is missing, **stop, tell the user which tools/capabilities are missing, and ask them to install them with their platform's package manager**, then re-run the check before proceeding. ```sh missing="" for t in qemu-system-x86_64 qemu-img expect xz rsync git ssh scp ssh-keygen curl python3; do command -v "$t" >/dev/null 2>&1 || missing="$missing $t" done command -v cloud-localds >/dev/null 2>&1 || command -v genisoimage >/dev/null 2>&1 \ || missing="$missing seed-builder(cloud-localds|genisoimage)" # Hardware acceleration is host-specific — check the right accelerator for this OS. case "$(uname -s)" in Linux) [ -w /dev/kvm ] || missing="$missing kvm(/dev/kvm-not-writable)" ;; Darwin) qemu-system-x86_64 -accel help 2>&1 | grep -q hvf || missing="$missing hvf-accelerator" ;; *) echo "NOTE: confirm this host has a QEMU hardware accelerator and adjust the boot accel= below" ;; esac [ -n "$missing" ] && echo "MISSING:$missing" || echo "all prerequisites present" ``` If the output is anything but `all prerequisites present`, name the missing tools/capabilities to the user and ask them to install them however their OS does — do NOT guess a distro package name. (`cloud-localds` ships in `cloud-image-utils` on Debian/Ubuntu; if it is unavailable, the cidata seed can also be built with `genisoimage -output seed.img -volid cidata -joliet -rock user-data meta-data`, which is what the OpenBSD CI path uses.) A missing hardware accelerator — no writable `/dev/kvm` on Linux, no HVF on macOS — is a host/BIOS/virtualization fix only the user can make; the VM is unusably slow without it, so don't fall back to TCG emulation. This flow is verified on Linux+KVM through the libs build and a debug build. The guest side is OS-agnostic, so macOS goes through the same steps with the HVF accelerator (`accel=kvm:hvf` in Step 3 selects it). ## Setup Pick a persistent VM directory and a FreeBSD version that **matches ponyc CI** (the tier-3 `freebsd` job in `.github/workflows/ponyc-tier3.yml` tests a single version — currently 15.1; the image URL is in `.ci-scripts/bsd/freebsd-provision.bash`). Use the same value in every block below. ### 1. Download the image (one time; keep the `.xz` to avoid re-downloading) The BASIC-CLOUDINIT image boots straight to a cloud-init-configured system — no installer, and it ships the full base system including `/usr/include`. ```sh VMDIR=~/vms/freebsd-15.1; FREEBSD_VERSION=15.1 mkdir -p "$VMDIR" && cd "$VMDIR" base="https://download.freebsd.org/releases/VM-IMAGES/${FREEBSD_VERSION}-RELEASE/amd64/Latest" curl -fL --retry 3 -o freebsd.qcow2.xz \ "$base/FreeBSD-${FREEBSD_VERSION}-RELEASE-amd64-BASIC-CLOUDINIT-ufs.qcow2.xz" xz -dk freebsd.qcow2.xz # -k keeps the .xz original qemu-img resize freebsd.qcow2 60G # first boot grows the root fs to fill this ``` ### 2. Key + cloud-init seed `nuageinit` reads a cidata seed image at boot: it installs your ssh public key for the `freebsd` user and sets root's password (so the Step 5 `su` bootstrap can reach root). ```sh VMDIR=~/vms/freebsd-15.1; cd "$VMDIR" test -f vm_key || ssh-keygen -t ed25519 -f vm_key -N "" -q cat > user-data <<USERDATA #cloud-config ssh_authorized_keys: - $(cat vm_key.pub) chpasswd: expire: false list: - root:ciroot USERDATA cat > meta-data <<'METADATA' instance-id: freebsd-dev local-hostname: freebsd-dev METADATA cloud-localds seed.img user-data meta-data ``` ### 3. Boot (daemonized, persistent) ```sh VMDIR=~/vms/freebsd-15.1; FREEBSD_VERSION=15.1; cd "$VMDIR" qemu-system-x86_64 \ -name freebsd-${FREEBSD_VERSION} \ -machine pc,accel=kvm:hvf -cpu host -smp 8 -m 12G \ -drive file=freebsd.qcow2,format=qcow2,if=virtio \ -drive file=seed.img,format=raw,if=virtio \ -netdev user,id=net0,hostfwd=tcp::2222-:22 \ -device virtio-net-pci,netdev=net0 \ -display none -pidfile freebsd.pid -daemonize ``` (`accel=kvm:hvf` picks KVM on Linux or HVF on macOS and errors if neither is available — it never silently falls back to slow TCG. `-smp`/`-m` are speed knobs; CI uses 4 CPUs / 12G. `hostfwd 2222->22` is the ssh port — to run two FreeBSD VMs at once (e.g. CI's 15.1 plus another version you want to compare against), give each its own `$VMDIR` and a distinct hostfwd port, and pass that port to every ssh/scp/rsync `-p`. The seed drive stays attached — `nuageinit` skips it on later boots once the instance-id is seen. This boot command is adapted from `.ci-scripts/bsd/freebsd-provision.bash`: it adds `-pidfile`/`-smp 8` and drops the CI host-bootstrap; CI has no rng or monitor device for FreeBSD and neither does this.) ### 4. Wait for ssh (first boot runs nuageinit + growfs) ```sh VMDIR=~/vms/freebsd-15.1; FREEBSD_VERSION=15.1; cd "$VMDIR" up="" for i in $(seq 1 150); do # ~5 min, matching CI's timeout if ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=ERROR -o ConnectTimeout=2 -i vm_key -p 2222 \ freebsd@localhost true 2>/dev/null; then up=1; echo "SSH up"; break; fi sleep 2 done [ -n "$up" ] || echo "SSH NEVER CAME UP — see recovery below." ``` If ssh never comes up, this VM has no VGA console to look at (it booted headless), so diagnose in this order: - **Is the process alive?** `pgrep -af "qemu-system-x86_64 -name freebsd-${FREEBSD_VERSION}"`. If it's gone, the boot died early (usually a bad/incomplete image from Step 1) — re-check the Step 1 download/resize and re-boot. - **Watch the boot.** The image logs to the serial console, so re-boot with a serial log added — `kill "$(cat freebsd.pid)"`, then re-run the Step 3 boot command with `-serial file:$VMDIR/console.log` appended — and read `console.log` to see how far it got. - **ssh actively *refuses* the key (vs. just timing out)?** That means `nuageinit` never installed it — the seed didn't apply. Confirm `seed.img` is attached as a drive in the boot command (Step 3) and was built in Step 2, then re-boot. ### 5. One-time root bootstrap (install deps + configure doas) via expect FreeBSD's `nuageinit` can't run `write_files`/`runcmd`, so — exactly as CI does — drive `su -m root` with `expect` to install the build dependencies and grant the `freebsd` user passwordless `doas`. This runs once; `doas.conf` persists on disk for the VM's life. (The expect block is adapted from the one embedded in `freebsd-provision.bash`.) ```sh VMDIR=~/vms/freebsd-15.1; cd "$VMDIR" expect <<'EXPECT' set timeout 600 spawn ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=ERROR -i vm_key -p 2222 -t freebsd@localhost su -m root expect { "Password:" { send "ciroot\r" } timeout { puts "Timeout waiting for password prompt"; exit 1 } } expect { "su: Sorry" { puts "su failed - wrong password or nuageinit didn't set it"; exit 1 } "#" {} timeout { puts "Timeout waiting for root shell"; exit 1 } } send "pkg install -y cmake gmake libunwind git python3 rsync doas\r" expect { "#" {} timeout { puts "Timeout during pkg install"; exit 1 } } send "echo 'permit nopass freebsd' > /usr/local/etc/doas.conf\r" expect { "#" {} timeout { puts "Timeout writing doas.conf"; exit 1 } } send "exit\r" expect eof EXPECT ``` If `expect` prints one of its `Timeout …` messages, it's safe to re-run this block — the `pkg install -y` and the clobbering `>` write to `doas.conf` are both idempotent. The one exception is `su failed`: that means the seed never set root's password, so root was never reachable. Re-running won't fix it — that's a boot/seed problem, so work through the Step 4 recovery (is the VM up? did `seed.img` apply?) first. ### 6. Rsync the ponyc checkout in (exclude the host `build/`) Run from your ponyc checkout. Exclude the top-level `build/` (host artifacts; the VM builds its own). Keep `.git` (CMake runs `git rev-parse`). The vendored LLVM submodule under `lib/llvm/src` IS transferred — the VM needs it for the libs build. ```sh VMDIR=~/vms/freebsd-15.1 rsync -az --exclude='/build' \ -e "ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=ERROR -i $VMDIR/vm_key -p 2222" \ "$(git rev-parse --show-toplevel)/" freebsd@localhost:/home/freebsd/ponyc/ ``` ## Using the VM The build uses FreeBSD's default clang — no `CC`/`CXX`/`LD_LIBRARY_PATH` exports (that is the DragonFly-only gcc13 dance). `cmake -P lib/build-libs.cmake` builds the vendored LLVM and is the multi-hour long pole — run it detached, then poll the log until it ends with `libs DONE rc=0` (a nonzero rc means it failed — read the log above the marker): ```sh VMDIR=~/vms/freebsd-15.1 SSH="ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=ERROR -i $VMDIR/vm_key -p 2222 freebsd@localhost" $SSH /bin/sh <<'EOF' set -e cat > /home/freebsd/run-libs.sh <<'S' #!/bin/sh cd /home/freebsd/ponyc && cmake -DJOBS=8 -P lib/build-libs.cmake echo "libs DONE rc=$?" S chmod +x /home/freebsd/run-libs.sh nohup /home/freebsd/run-libs.sh > /home/freebsd/libs-build.log 2>&1 & echo "libs building pid $!" EOF # poll until done: $SSH /bin/sh -c 'tail -3 /home/freebsd/libs-build.log' -> wait for "libs DONE rc=0" ``` Once `libs` is built (it persists in the VM), build and test with the same shell. Do not run this block until the poll shows `libs DONE rc=0` — `cmake --build` against a half-built `build/libs` fails: ```sh VMDIR=~/vms/freebsd-15.1 SSH="ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=ERROR -i $VMDIR/vm_key -p 2222 freebsd@localhost" $SSH /bin/sh <<'EOF' set -e cd /home/freebsd/ponyc cmake --preset debug cmake --build --preset debug ctest --preset debug -L ci-core EOF ``` To iterate on a fix: edit on the host, re-run Step 6's `rsync` (it's incremental — only changed files are sent), then re-run the build block above (long rebuilds should also be detached + polled). For exact CI parity, the tier-3 `freebsd` job (`.github/workflows/ponyc-tier3.yml`) runs more than this — a `--static` embedded-LLD link smoke, the self-hosted tool tests (`pony-doc-tests`/`pony-lint-tests`/`pony-lsp-tests`), a release build + the `ci-core` suite, and two extra smokes: