spark-selfhosted-runner

name	spark-selfhosted-runner
description	Run BEFORE bringing a self-hosted GitHub Actions runner online on the DGX Spark, or when a workflow_dispatch GPU job is stuck "queued/waiting" against `[self-hosted, spark, aarch64]`, or when provisioning a model / SUT (e.g. Unlimited-OCR, SGLang) into a venv on the Spark for a `.github/workflows/*.yml` job. Fires on: "register the Spark runner", "runner offline", "self-hosted runner", a workflow step dying `python: command not found` / `exit code 127` at preflight, `pip`/`torch` "externally-managed-environment" (PEP 668), a backend that is NOT on PyPI, or an sglang/vllm wheel that silently clobbers a CUDA torch to `+cpu`. ALSO fires on SGLang refusing to serve a `trust_remote_code` HF model — `"<Arch> has no SGlang implementation and the Transformers implementation is not compatible with SGLang"` — and on the repeated one-dep-per-run "whack-a-mole" where each dispatch dies on the next missing custom-code import (matplotlib, easydict, trust_remote_code). Applies to BOTH the Spark lane and the paid RunPod lane (same SGLang provisioning traps). Use even if it looks like a one-line fix — each trap below cost a full queued-run cycle. Alias: /spark-selfhosted-runner. Augments spark-cluster-ops (cluster/GPU/worktree ops) and wisdom-gpu-prebaked (RunPod GPU deps).
metadata	{"layer":"A","origin":"U2 ocr-fabrication-bench runner bring-up, 2026-07-03"}

Spark self-hosted runner bring-up + SUT venv provisioning (this repo)

The Spark (spark-2f2d, GB10, aarch64) is the repo's free iteration lane: it runs every workflow_dispatch GPU job (ocr-fabrication-bench.yml, spark-gpu.yml, RLVR arms). A job sits queued forever if no runner with the exact labels [self-hosted, spark, aarch64] is online. Registering that runner and provisioning a real system-under-test onto it hits the traps below.

Install the runner as a service, not foreground

• ./config.sh --url https://github.com/tomyimkc/sophia-agi --token <FRESH> --labels spark,aarch64 --name spark-gb10 --unattended — the token is single-use (~1 h), maintainer-supplied; never scrape it.
• sudo ./svc.sh install <user> && sudo ./svc.sh start — systemd, not ./run.sh. Foreground dies on logout/reboot (the Mac-runner Q4 lesson). Verify systemctl is-enabled … = enabled.
• Check pickup with gh api repos/tomyimkc/sophia-agi/actions/runners — a busy=false, online runner whose labels are a superset of the workflow's runs-on is the only thing that clears a queued job.

The traps (each = one wasted queued-run cycle)

Symptom in the run log	Root cause	Fix (environment only — never edit the workflow/scorer/gate)
Preflight step: python: command not found, ##[error]…exit code 127 — but uname -m/nvidia-smi passed just above	Ubuntu ships python3 only; bare python is absent and workflows call python	sudo apt install python-is-python3. Idempotent; .path already has /usr/bin, so no restart needed.
pip install … → error: externally-managed-environment	System python is PEP 668 marked (/usr/lib/python3.12/EXTERNALLY-MANAGED)	Do NOT --break-system-packages on a shared box. Build a dedicated venv (e.g. /opt/sophia-ocr-venv, chown to the runner user) and wire it onto the runner.
pip install <backend> → No matching distribution found	The SUT backend is not on PyPI (source-only)	Clone the source repo (git clone --depth 1 …), record git rev-parse HEAD as the SUT version pin, install from source into the venv. The workflow's `pip install
After installing sglang/vllm: torch.cuda.is_available() flips to False, torch.__version__ shows …+cpu; and transformers jumped to 5.x	The backend wheel hard-pulls torch==X (cpu) + transformers>=5, uninstalling your cu128 torch. pip install torch==X --index-url …/cu128 then no-ops (the +cpu build already satisfies ==X)	Force it: pip install --index-url https://download.pytorch.org/whl/cu128 --force-reinstall --no-deps torch==X torchvision==Y torchaudio==X. Re-verify cuda True + get_device_name. Match the backend's pinned torch version, just the cu128 build (ABI match for its kernels).
Bench step: model weights re-download every run / HF_HOME unwritable	Workflow hardcodes HF_HOME=/home/runner/.cache/huggingface, a dir the runner user can't create	sudo mkdir -p /home/runner/.cache/huggingface && sudo chown -R <runner-user> /home/runner, then pre-warm snapshot_download(model, revision=<sha>) into it.
SGLang server loads weights fine, then the scheduler dies right after Load weight end with AssertionError: FlashAttention v3 Backend requires SM>=80 and SM<=90 (log also prints SMxxx (Blackwell) detected)	The SUT / infer.py pins --attention-backend fa3; FA v3 is SM 80–90 only, but GB10 is SM 120/121 (Blackwell) — SGLang itself refuses fa3 there and mandates flashinfer. The crash is after the ~6 GB weight load, so it looks late/mysterious	Select the backend by torch.cuda.get_device_capability: flashinfer on SM>90, fa3 on SM 80–90 (flashinfer already ships in the venv). Hardware-forced, not a tuned knob. If the launch command is frozen pre-registration, amend it via PR — never inline-edit (precedent: U2 run 28671947676 → annex A1.1 Blackwell override). Complements the [Spark aarch64 vLLM JIT chain] note (Blackwell needs special kernel/attention handling).
SGLang refuses to serve at all: ValueError: <Arch>ForCausalLM has no SGlang implementation and the Transformers implementation is not compatible with SGLang (e.g. UnlimitedOCRForCausalLM, U2 run 28683761217)	Stock PyPI sglang[all] does not register the model's architecture. A trust_remote_code SUT often ships its OWN SGLang build that registers the arch — bundled in the HF repo AND the GitHub tree (look for wheel/sglang-*.whl; the repo's infer.py/README installs exactly it, e.g. uv pip install wheel/sglang-…whl). PyPI sglang will never serve it.	Install the SUT's OWN bundled sglang wheel, not PyPI sglang. Fetch it deterministically at the pinned weights sha: huggingface_hub.hf_hub_download(model_id, "wheel/sglang-….whl", revision=<sha>) (install huggingface_hub first so the fetch can't itself whack-a-mole), then pip install <wheel>. This is STRICTLY MORE faithful to "as shipped" than PyPI — it is the SUT's own serving code — so it's a load-prerequisite, measurement-neutral, pre-run. Document it in the annex parallel to the trust_remote_code note; changes no gate value; canClaimAGI stays false.
Whack-a-mole: each dispatch dies on the NEXT missing import — ModuleNotFoundError/ImportError for the model's custom-code deps, one per run (trust_remote_code, then matplotlib, then easydict, then addict, …). 5+ wasted cycles incl. paid (28681097651/28681856749/28682776894/28684661135)	A trust_remote_code model's custom modeling .py files import third-party libs that transformers' check_imports validates at load — NOT in its requirements.txt. Fixing one surfaces the next. The subtle trap (run 28684661135): even after LISTING all imports, the pin set ASSUMED addict was transitively present via sglang[all] — it wasn't, and the run died after a 300 s health-wait.	Two rules, both required. (1) AST-extract the COMPLETE set up front, don't eyeball: for f in list_repo_files(repo, revision=sha) if f.endswith('.py'): ast.parse(hf_hub_download(...)) → collect every ast.Import/ast.ImportFrom top-level module. (2) NEVER assume any of them is transitively present — pin EVERY non-stdlib, non-stack leaf explicitly (addict/easydict/matplotlib/requests/tqdm for Unlimited-OCR). Then add a fatal import-preflight in the pod script that importlib.import_modules each pinned dep BEFORE sglang.launch_server and sys.exit(1)s naming any miss — converts a buried post-300 s traceback into a ~10 s named failure. Keep stack-provided imports (flash_attn/einops/numpy) as a NON-fatal observed check (they may be try/except-guarded).

OKF failure-mode map — the whole serving-provisioning pipeline (pre-mortem)

The whack-a-mole above is one symptom of a deeper habit: discovering failures one paid run at a time instead of enumerating the failure space up front. Before dispatching a paid serving run, walk EVERY stage below and confirm each guard. Order = pipeline order; a run can die at any stage, so a green earlier stage tells you nothing about a later one. O=observe in log, K=known root cause, F=fix/guard. (OKF = Observe → Known-cause → Fix; the point is to pre-write the K and F so the O is a 10-second named error, not a re-diagnosis.)

#	Stage	What can fail (O)	Cause (K)	Guard (F) — do this BEFORE the paid run
1	Runner/exec	job queued forever; python: not found; PEP 668	labels not superset; bare python; externally-managed	runner online+superset labels; python-is-python3; dedicated venv (rows above)
2	Repo/source clone	ModuleNotFoundError: tools; wrong/ës SHA	script run from tools/ not root; unpinned	repo-root sys.path.insert; clone at pinned SOURCE_COMMIT + record it
3	Serving engine	<Arch> has no SGlang implementation	PyPI sglang ≠ SUT's arch	install the SUT's OWN bundled wheel/sglang-*.whl at pinned sha (row above)
4	Wheel fetch	hf_hub_download fails / huggingface_hub missing	fetch dep not installed first	pip install huggingface_hub BEFORE the fetch; revision-pin the download
5	Custom-code deps	check_imports … requires X (matplotlib→easydict→addict)	model .py imports not in requirements.txt; assumed transitively present	AST-extract ALL imports; pin EVERY non-stack leaf; fatal import-preflight before serve (whack-a-mole row)
6	Load prerequisite	requires trust_remote_code=True	custom modeling code must execute	pass --trust-remote-code on the serve cmd (load-prereq, measurement-neutral)
7	Torch/CUDA ABI	torch…+cpu, cuda False; vLLM _C ABI crash	backend wheel clobbers cu-torch / shared-venv torch pin mismatch	isolated venv; --force-reinstall --no-deps cu-torch after backend; never mutate the shared .venv
8	Attention/kernel	FA v3 requires SM 80–90; missing sm_121 cubin	frozen fa3 on Blackwell; prebuilt kernel gap	select backend by device capability (Spark) OR run the paid A100/H100 lane where fa3 is native
8b	JIT kernel build	server Killed at CUDA-graph capture; FileNotFoundError: 'ninja' (or nvcc/c++) deep in jit_kernel/rope.py→tvm_ffi build_ninja (U2 run 28685494510, x86 A100)	bundled sglang JIT-compiles its fused RoPE kernel at first forward; the pip ninja is in the venv bin/ (NOT on the server subprocess PATH); nvcc/g++ come from the CUDA image	before launch: export CUDA_HOME=${CUDA_HOME:-/usr/local/cuda}; export PATH="$VENV/bin:$CUDA_HOME/bin:$PATH"; verify ninja/nvcc/c++ resolve (fatal-fast, named) — apt install ninja-build build-essential if absent. Environment provisioning, NOT an invocation change. Bites BOTH aarch64 Spark and x86 RunPod (sglang JITs on any arch).
9	Weights/HF cache	re-download each run; HF_HOME unwritable; rate-limit	hardcoded cache dir; no HF_TOKEN	writable+chowned HF_HOME; pre-warm snapshot_download(revision=sha); set HF_TOKEN if throttled
10	Server health	Connection refused at /health → SKIP after 300 s	any stage 3–9 failed at launch, surfaced late	the stage-5 preflight + reading the server-log tail (the launcher copies it back) tells you the REAL cause, not "unreachable"
11	Scorer/adapter	0 regions scored; adapter import error; wrong invocation	adapter shapes the number (escalate, don't author)	adapter is pre-registered/maintainer-owned; a missing adapter is a recorded SKIP, never a guess
12	Cost/teardown	leaked pod; 401 can't reap	no triple-guard teardown; expired key	launcher finally + in-pod watchdog + if: always() reap; on 401 have the human confirm the dashboard

How to use it: for a trust_remote_code-SGLang SUT, stages 3–6 are the ones that cost paid runs; resolve ALL of them from the source at the pinned sha in one PR (bundled wheel + complete pinned dep set + trust_remote_code + fatal preflight) rather than one dispatch per stage. The server-log tail (stage 10) is the single most useful diagnostic — always read it before concluding "unreachable."

Wire the venv onto the runner (so job steps use it)

• Prepend venv/bin to <runner>/.path (back it up first), then restart the service. The .path file overrides job-step PATH; the runsvc process's own env is separate and will still show the old PATH — that is expected, not a failure.
• Verify without dispatching (hold the lane): env -i PATH="$(cat .path)" bash -c 'command -v python; python -c "import torch;print(torch.cuda.is_available())"'.

Record a provisioning manifest (pin discipline)

Beside the venv write PROVISIONING-MANIFEST.json + a pip freeze lock: venv path, exact pins (torch+cuda build, transformers, backend), SUT source commit sha, HF model resolved sha, weights-prewarmed flag. Flag any pin the chosen path forced against a standing guideline (e.g. the SGLang path requires transformers>=5, which contradicts the transformers<5 wisdom-gpu note — record the operative pin and make the advisor ratify it; do not silently pick).

Do not (the escalation boundary)

• Do not --break-system-packages on the shared box's system python.
• Do not author a SUT adapter (the module the scorer imports to drive the model) as "just provisioning". Choosing its invocation path / prompt / region-crop / output→abstention parsing shapes the measured number — that is a pre-registered, maintainer-owned decision. Provision the environment; escalate the adapter. (Precedent: U2, 2026-07-03.)
• Do not re-dispatch against a missing adapter or half-provisioned env — the honest state is a recorded SKIP; a re-run does not improve it.

Trap: run_os_redteam_ondevice.py direct invocations (2026-07-05, cost 2 reruns)

1. Provision BEFORE the driver. The sandbox suite reads SOPHIA_FS_ROOT/SOPHIA_FS_VAR_ROOT but the driver does NOT provision them — pointing the env at an empty temp dir fails the sandbox red-team 4/6 (looks like a seal regression; is not). Run tools/provision_sophia_fs_layout.py --root ... --var-root ... (+ --check) first, and pin --bpf-out to your SOPHIA_SECCOMP_BPF path (the driver's default writes /opt/sophia and overrides your env for the child).
2. The driver's roofline stage inherits the driver interpreter. Run the driver with /usr/bin/python3 (seccomp+pytest) and the roofline stage silently writes NOTHING (torch absent → exit 0). Run tools/spark_roofline_report.py --run separately with the torch venv python, exactly like os-redteam-spark.yml's ROOFLINE_PY split.
3. governed_benchmark_evidence.py wants {sandbox, airgap, cgroup} top-level keys, not the driver report's suites: {"redteam-sandbox": ...} shape — key-map verbatim into a summary JSON (no numbers invented) or the receipt synthesizer fail-closes to pending.