create-dragonfly-dev-env

name	create-dragonfly-dev-env
description	Load when you need to reproduce or validate a gcc-specific or DragonFly-specific ponyc issue on a local VM, or to stand up a local DragonFly BSD development VM that matches ponyc tier-3 CI. Covers the prerequisite check, the sudo-free QEMU/KVM setup, the reused console automation, the in-VM gcc13 build env, and the gotchas (csh, screendump paths, boot timing, detached builds, per-block shells) that make hand-rolling one error-prone.
disable-model-invocation	false

Create a DragonFly BSD dev/test VM for ponyc

Stand up a local DragonFly BSD VM that matches the ponyc tier-3 CI job, so you can build and test ponyc on DragonFly without GitHub Actions.

DragonFly only builds ponyc with gcc (the gcc13 package), so it is the authoritative local environment for gcc-specific ponyc issues — the ones clang-based CI never sees. The setup is fiddly and timing-sensitive; the steps below are the known-good sequence. Follow them in order.

Run the repo-relative commands (.ci-scripts/bsd/..., rsync of the source) from your ponyc checkout root. The VM itself lives in a separate directory ($VMDIR).

Each fenced sh block below is self-contained. It re-sets VMDIR/DFLY_VER (and, where needed, the SSH command) at its top, because an agent runs each block as a fresh shell — environment variables and the working directory do NOT carry from one block to the next. Do not "optimize" by setting them once and dropping them from later blocks: an empty $SSH would make a $SSH /bin/sh <<'EOF' ... EOF heredoc run on your host (silently, exit 0) — and Step 7 contains rm -rf /usr/local. Keep every block whole. Pick your VMDIR/DFLY_VER once and use the same values in every block.

Gotchas that will bite you (read first)

Each of these cost real debugging. Internalize them before you start — they recur at several steps:

• Per-block fresh shells. See above: re-establish VMDIR/DFLY_VER/SSH in every block; never let $SSH be empty in front of a /bin/sh heredoc.
• Root's login shell is csh, not sh. Any remote command with $(...), export, or a here-string MUST go through an explicit /bin/sh: ssh ... root@localhost /bin/sh <<'EOF' ... EOF. A bare ssh ... root@localhost 'cmd with $(...)' fails with Illegal variable name.
• Boot timing is handled by the script. dfly_configure_vm.py takes periodic VGA screendumps and waits for the screen to stabilize before attempting login, then retries the login + serial shell start until the serial port responds. If something goes wrong, check the diagnostic screendump the script saves, or use the manual screendump helper (Step 5) to see the VM state.
• A daemonized QEMU chdirs to /. The monitor screendump (and any relative path the daemon writes) needs an absolute path, or it fails Permission denied.
• No sudo needed (and often unavailable). CI loop-mounts the ISO with sudo to get /usr/include; locally, extract it with bsdtar instead — same result, no privilege.
• Detach long in-VM builds. The libs build (LLVM) takes hours; run it nohup … > /build/x.log 2>&1 & and poll the log, so an ssh drop doesn't kill it.
• Reuse the CI console script verbatim. .ci-scripts/bsd/dfly_configure_vm.py bootstraps a serial shell via QEMU sendkey, then runs setup commands through the serial console with prompt detection. Copy it, don't reimplement the bootstrap or setup.

Step 0 — verify prerequisites (do NOT assume they're installed)

This skill needs, on the host:

• an existing ponyc checkout (you run the repo-relative commands from it)
• hardware-accelerated virtualization for QEMU (KVM on Linux, HVF on macOS)
• enough free disk for the VM's sparse qcow2 disks (provisioned at 60 GB + 50 GB nominal; they grow only as used, and the libs build plus a build fill a large chunk of the data disk), ~2 GB for the image/ISO files, and network access to mirror-master.dragonflybsd.org
• qemu-system-x86_64 and qemu-img
• bsdtar (a libarchive tar that reads ISO9660; the default tar on macOS, FreeBSD, and DragonFly)
• bunzip2, rsync, git, curl, python3
• an OpenSSH client (ssh, scp, ssh-keygen)
• a PPM-to-PNG converter: any one of magick/convert, ffmpeg, or pnmtopng

Check that every required tool is present. Do not assume the environment has them, and do not install anything yourself — installing them is the user's call (it needs privilege, and the package names vary by OS, so naming a specific package would be wrong on half the platforms this runs on). If anything is missing, stop, tell the user which tools/capabilities are missing, and ask them to install them with their platform's package manager, then re-run the check before proceeding.

missing=""
for t in qemu-system-x86_64 qemu-img bsdtar bunzip2 rsync git ssh scp ssh-keygen curl python3; do
  command -v "$t" >/dev/null 2>&1 || missing="$missing $t"
done
command -v magick >/dev/null 2>&1 || command -v convert >/dev/null 2>&1 \
  || command -v ffmpeg >/dev/null 2>&1 || command -v pnmtopng >/dev/null 2>&1 \
  || missing="$missing ppm-converter(magick|ffmpeg|pnmtopng)"
# Hardware acceleration is host-specific — check the right accelerator for this OS.
case "$(uname -s)" in
  Linux)  [ -w /dev/kvm ] || missing="$missing kvm(/dev/kvm-not-writable)" ;;
  Darwin) qemu-system-x86_64 -accel help 2>&1 | grep -q hvf || missing="$missing hvf-accelerator" ;;
  *)      echo "NOTE: confirm this host has a QEMU hardware accelerator and adjust the boot accel= below" ;;
esac
[ -n "$missing" ] && echo "MISSING:$missing" || echo "all prerequisites present"

If the output is anything but all prerequisites present, name the missing tools/capabilities to the user and ask them to install them however their OS does — do NOT guess a distro package name. (bsdtar is the libarchive tar; on macOS/FreeBSD/ DragonFly the system tar is already bsdtar, elsewhere it comes from libarchive.) A missing hardware accelerator — no writable /dev/kvm on Linux, no HVF on macOS — is a host/BIOS/virtualization fix only the user can make; the VM is unusably slow without it, so don't fall back to TCG emulation.

This flow is verified on Linux+KVM. The guest side is OS-agnostic, so macOS goes through the same steps with the HVF accelerator (accel=kvm:hvf in Step 4 selects it).

Setup

Pick a persistent VM directory and a DragonFly version that matches ponyc CI (check the image URL in .ci-scripts/bsd/dragonfly-provision.bash; currently 6.4.2). Use the same two values in every block below.

1. Download the image + ISO (one time; keep the .bz2 to avoid re-downloading)

The raw .img boots; the .iso is only mined for system headers (the raw image ships without /usr/include).

VMDIR=~/vms/dragonfly-6.4.2; DFLY_VER=6.4.2
mkdir -p "$VMDIR" && cd "$VMDIR"
base="https://mirror-master.dragonflybsd.org/iso-images"
curl -fL --retry 3 -o dfly.img.bz2 "$base/dfly-x86_64-${DFLY_VER}_REL.img.bz2"
curl -fL --retry 3 -o dfly.iso.bz2 "$base/dfly-x86_64-${DFLY_VER}_REL.iso.bz2"

2. Build the disks

VMDIR=~/vms/dragonfly-6.4.2; cd "$VMDIR"
bunzip2 -k -f dfly.img.bz2 dfly.iso.bz2          # -k keeps the .bz2 originals
qemu-img convert -f raw -O qcow2 dfly.img dfly.qcow2
qemu-img resize dfly.qcow2 60G
qemu-img create -f qcow2 dfly-data.qcow2 50G     # build workspace; root part is ~1.8G
rm -f dfly.img                                    # keep dfly.qcow2 + the .bz2

3. Extract system headers from the ISO — no sudo (replaces CI's mount -o loop)

bsdtar reads ISO9660 directly. The tar must have include/ as its top entry so the in-VM extraction matches CI. (Both the extract and the repack use bsdtar, so no separate tar is needed on the host.)

VMDIR=~/vms/dragonfly-6.4.2; cd "$VMDIR"
rm -rf iso-stage && mkdir iso-stage
bsdtar -xf dfly.iso -C iso-stage usr/include
bsdtar -cf dfly-include.tar -C iso-stage/usr include
rm -rf iso-stage

4. Key + boot (daemonized, persistent)

VMDIR=~/vms/dragonfly-6.4.2; DFLY_VER=6.4.2; cd "$VMDIR"
test -f vm_key || ssh-keygen -t ed25519 -f vm_key -N "" -q
qemu-system-x86_64 \
  -name dragonfly-${DFLY_VER} \
  -machine pc,accel=kvm:hvf -cpu host -smp 8 -m 12G \
  -drive file=dfly.qcow2,format=qcow2,if=virtio \
  -drive file=dfly-data.qcow2,format=qcow2,if=virtio \
  -netdev user,id=net0,hostfwd=tcp::2222-:22 \
  -device virtio-net-pci,netdev=net0 \
  -object rng-random,id=rng0,filename=/dev/urandom \
  -device virtio-rng-pci,rng=rng0 \
  -monitor unix:dfly-monitor.sock,server,nowait \
  -serial unix:dfly-serial.sock,server,nowait \
  -display none -pidfile dfly.pid -daemonize

(accel=kvm:hvf picks KVM on Linux or HVF on macOS and errors if neither is available — it never silently falls back to slow TCG. -smp/-m are speed knobs; CI uses 4 CPUs / 12G. hostfwd 2222->22 is the ssh port.)

5. Run the console setup script

The CI console script takes periodic VGA screendumps to detect when boot finishes, then bootstraps a serial shell via sendkey with retries, and runs all setup commands through the serial console with prompt detection. It handles boot timing internally.

The script lives in the repo at .ci-scripts/bsd/dfly_configure_vm.py. Copy it into $VMDIR (it connects to dfly-monitor.sock and dfly-serial.sock in its working directory) and run it there. Set DFLY_ARTIFACTS_DIR to the VM directory so the script can write screendumps for boot detection and diagnostics. It logs in, brings up networking, configures sshd, and installs your key:

VMDIR=~/vms/dragonfly-6.4.2; cd "$VMDIR"
cp "$(git rev-parse --show-toplevel)/.ci-scripts/bsd/dfly_configure_vm.py" "$VMDIR/"
export PUB_KEY="$(cat vm_key.pub)"
export DFLY_ARTIFACTS_DIR="$VMDIR"
python3 dfly_configure_vm.py

A monitor_cmd.py helper is still useful for ad-hoc screendumps when debugging boot problems. Create one if needed:

VMDIR=~/vms/dragonfly-6.4.2; cd "$VMDIR"
cat > monitor_cmd.py <<'PY'
import socket, sys, time
s = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM); s.connect('dfly-monitor.sock')
time.sleep(0.4); s.recv(65536)
s.sendall((sys.argv[1] + '\n').encode()); time.sleep(0.5); s.settimeout(1.0)
try:
    while True:
        d = s.recv(65536)
        if not d: break
        sys.stdout.write(d.decode(errors='replace'))
except socket.timeout: pass
s.close()
PY
to_png() {  # convert PPM->PNG with whichever converter is installed
  if   command -v magick   >/dev/null 2>&1; then magick "$1" "$2"
  elif command -v convert  >/dev/null 2>&1; then convert "$1" "$2"
  elif command -v ffmpeg   >/dev/null 2>&1; then ffmpeg -y -loglevel error -i "$1" "$2"
  else pnmtopng "$1" > "$2"; fi
}
python3 monitor_cmd.py "screendump $VMDIR/console1.ppm"   # absolute path is required
to_png "$VMDIR/console1.ppm" "$VMDIR/console1.png"

6. Wait for ssh

If ssh never comes up within the budget, the console script likely ran before the VM was ready (Step 5) — re-screendump to check the console state, and re-run Step 5 if it's not logged in.

VMDIR=~/vms/dragonfly-6.4.2; cd "$VMDIR"
up=""
for i in $(seq 1 150); do   # ~5 min, matching CI's timeout
  if ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=ERROR -o ConnectTimeout=2 -i vm_key -p 2222 \
       root@localhost true 2>/dev/null; then up=1; echo "SSH up"; break; fi
  sleep 2
done
[ -n "$up" ] || echo "SSH NEVER CAME UP — check the console screendump (Step 5) and rerun if needed"

From here, always drive the VM through /bin/sh (root's shell is csh — see Gotchas), and define $SSH in the same block you use it (it does not carry between blocks):

VMDIR=~/vms/dragonfly-6.4.2
SSH="ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=ERROR -i $VMDIR/vm_key -p 2222 root@localhost"
$SSH /bin/sh <<'EOF'
uname -a
EOF

7. Format and mount the build disk

The root partition is only ~1.8G — not enough for the build or packages. Format the second disk and mount it at /build.

VMDIR=~/vms/dragonfly-6.4.2
SSH="ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=ERROR -i $VMDIR/vm_key -p 2222 root@localhost"
$SSH /bin/sh <<'EOF'
set -e
newfs /dev/vbd1
mkdir -p /build
mount /dev/vbd1 /build
EOF

8. Move /usr/local to the build disk, install headers, install deps

gcc13 alone is ~418M, so package installs must land on /build.

VMDIR=~/vms/dragonfly-6.4.2
SSH="ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=ERROR -i $VMDIR/vm_key -p 2222 root@localhost"
$SSH /bin/sh <<'EOF'
set -e
cpdup /usr/local /build/usr_local
rm -rf /usr/local
ln -s /build/usr_local /usr/local
EOF

scp -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=ERROR -i "$VMDIR/vm_key" -P 2222 "$VMDIR/dfly-include.tar" root@localhost:/build/
$SSH /bin/sh <<'EOF'
set -e
tar xf /build/dfly-include.tar -C /build
ln -s /build/include /usr/include
rm /build/dfly-include.tar
EOF

$SSH /bin/sh <<'EOF'
set -e
pkg install -y cmake gmake git python3 cxx_atomics rsync gcc13 ca_root_nss
pkg clean -ay
git config --global --add safe.directory /build/ponyc
EOF

9. Rsync the ponyc checkout in (exclude the host build/)

Run from your ponyc checkout. Exclude the top-level build/ (host artifacts; the VM builds its own). Keep .git (CMake runs git rev-parse). The vendored LLVM submodule under lib/llvm/src IS transferred — the VM needs it for the libs build.

VMDIR=~/vms/dragonfly-6.4.2
rsync -az --exclude='/build' \
  -e "ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=ERROR -i $VMDIR/vm_key -p 2222" \
  "$(git rev-parse --show-toplevel)/" root@localhost:/build/ponyc/

Using the VM

Always set the gcc13 build env first (matches the tier-3 dragonflybsd job). cmake -P lib/build-libs.cmake builds the vendored LLVM and is the multi-hour long pole — run it detached, then poll the log until it ends with libs DONE rc=0 (a nonzero rc means it failed — read the log above the marker):