ci

name	ci
description	Local and cloud CI for Pulp — validate branches, create PRs, merge on green. Handles "push to main", "ship this", "run CI", "check PR", and "list PRs".
requires	{"scripts":["tools/local-ci/local_ci.py"],"tools":["gh"]}

CI Skill

Validate branches and ship code safely. This skill handles all CI workflows for Pulp across local machines and VMs.

Runner timing metrics

When asked whether Pulp's local runners are fast, stuck, regressing, or worth monitoring, query Shipyard's metrics surface before guessing. Pulp does not mirror these records into pulp CLI or pulp-mcp; Shipyard is the metrics store and tartci is an optional VM runtime emitter.

This metrics surface requires a Shipyard build that includes the shipyard metrics subcommand. Pulp's current pinned source-checkout Shipyard in tools/shipyard.toml is v0.68.0, which does not include it yet. If a checkout only has the pinned binary, use a newer Shipyard binary or source checkout for the optional metrics workflow, or skip metrics and inspect live jobs directly.

Use these commands as the normal agent loop:

shipyard metrics import github --repo danielraffel/pulp --limit 50 --json
tartci runtime export --repo danielraffel/pulp --since-days 14 \
  | shipyard metrics import tartci --json
shipyard metrics summary --project pulp --json
shipyard metrics watch --project pulp --since 14d --json
shipyard metrics advise --project pulp --json

Read watch as a triage signal, not as a hard failure. insufficient_samples means the lane does not have enough history yet; drift, failure-rate, or slowest findings are the useful prompts to inspect runner logs, VM boot behavior, cache state, or GitHub queue time. If tartci is not installed for a repo, skip the tartci runtime export import and still use the GitHub import plus Shipyard summary/watch commands.

If a PR's required macos check has been queued >30 min or the repo's PRs are all in mergeable_state=blocked with no movement, jump to "Self-hosted runner ops" near the end of this file. One-shot recovery is shipyard rescue <PR> (Shipyard v0.53.0+). Continuous prevention is shipyard runner watch --kill-hung-workers (v0.54.0+). Keep Shipyard itself current with shipyard update (v0.55.0+; Pulp currently pins v0.56.2+). All three replace the legacy planning/scripts/runner-watchdog.sh --fix workflow, which is now an anti-pattern (cancels queued runs but registers failure on required checks).

Test lanes — what gates the required macos check

Full model: docs/guides/test-lanes.md. Operationally, when a PR's required macos check goes red on a test unrelated to the diff, check the label:

• slow and validation-labeled tests are excluded from the required gate (.shipyard/config.toml test = "ctest ... --label-exclude \"validation|slow\"", matching build.yml's PR ctest and cross-platform-check.yml). The required gate also runs --repeat until-pass:2, so a single timing-flake self-heals.
• validation is example-only — the pluginval-* / auval-* / clap-dlopen-* validators under examples/. They do NOT gate core PRs; they gate on the example-validation lane (.github/workflows/examples-validation.yml), which blocks PRs touching examples/** and internally skips otherwise (required-safe: always reports).
• The required macos gate does NOT compile the examples. build.yml (and cross-platform-check.yml, the windows gates) configure with PULP_BUILD_EXAMPLES=OFF. Only release-cli.yml compiles the examples — at release time. Do not assume a green required gate means the examples build.

So a red required gate should NOT be an example pluginval/auval timeout — if it is, the exclusion regressed. The example-validation lane is not yet in required_status_checks; promote it once it is green on a real examples/** run.

Gotcha: a broken example goes green on main and breaks the RELEASE build

Because the required gate builds PULP_BUILD_EXAMPLES=OFF and only release-cli builds examples, an example that fails to compile can merge green and then fail every release build until it is fixed — silently making the SDK unpublishable. The nastiest form is a compiler disagreement: an out-of-declaration-order designated initializer (.kind before .range) is a hard error under GCC but only a warning under Clang. It compiled clean on every macOS/Clang check, then killed both Linux (GCC) legs of the release (#6082, the .kind/.range swap; it stranded ~10 tags).

Guard (build(examples) PR): examples/CMakeLists.txt promotes -Wreorder-init-list to an error under Clang (GCC already errors by default), so the reorder now fails wherever examples compile — release-cli AND the example-validation lane — on every compiler. examples-validation.yml also triggers on core/state/include/** + core/format/include/**, because a field reorder in a header the examples brace-initialize breaks their inits without touching examples/**. Proven by the cmake-examples-reorder-init-guard ctest. If you add a NEW example struct pattern that a compiler tolerates but the release compiler rejects, extend that guard rather than discovering it at release time.

A test that "fails" on the required gate may only have run out of clock

Before debugging what a failing gate test does, check whether it failed on elapsed time rather than behavior. A subprocess-spawning test that reddens macos on PRs that cannot possibly have caused it — a docs-only or CI-only PR — is the tell. pulp ship sign discovers desktop bundles via env and config identities read as a signing/keychain break for exactly this reason; it was a 10s cap on a case that spawns codesign three times, runs ~2s unloaded, and loses its margin under a -j8 run over 13k tests. The symptom is a bare REQUIRE_FALSE(x.timed_out) → !true, which names neither the subprocess nor the duration — so it reads like a logic failure.

The layering rule that prevents this class:

• ctest --timeout is the binding guard. build.yml runs ctest … --repeat until-pass:2 -j8 --timeout 120: a per-test hang guard plus one automatic retry. An in-test subprocess cap must stay comfortably looser than it, so a slow machine fails at the outer layer — which reports the test name and elapsed time — instead of at the inner one, which reports neither. An inner cap tighter than the outer guard is strictly harmful: it fires first and diagnoses worse.
• These caps are hang guards, not performance budgets, so the costs are asymmetric. Too generous only delays a genuinely wedged child (already bounded at 120s). Too tight buys recurring false reds on unrelated PRs. Err generous.
• Don't make them adaptive. Self-calibrating or load-scaled timeouts make failures unreproducible and stretch to accommodate real perf regressions. A fixed generous default plus an env override is the design.

Root cause of the drift: the CLI shellout suites each hand-roll their own helper and hardcode a timeout, so there is no shared default to inherit and a codesign-heavy suite could sit at 10s while its siblings used 30-60s. When adding a shellout test, reuse the shared helper rather than picking a number.

Gate: framework-neutrality (tools/scripts/framework_neutrality_check.py)

Hard-fails a PR when Pulp's own source names another UI framework — in a comment, a doc-comment, an identifier, or a test name — or adopts one of its class names into a Pulp API. Runs in gates.sh (gate 14), in CI, and as a ctest.

Two halves, and the second is the one that catches the deeper problem:

• prose — a foreign framework named in Pulp source.
• names — a foreign framework's class name adopted into Pulp's API. A comment is a liability; an adopted type name is the liability shipped in the public headers, where every downstream user inherits it.

It doubles as a design check: if a doc-comment needs a foreign framework's vocabulary to make sense, the API is a compatibility shim wearing a feature's clothes — redesign it rather than renaming it.

Three carve-outs, each of which would be a bug to "clean up":

• core/format/host_quirks/** and the tests that pin its LessonOnly rows. The prior-art citation IS the audit trail the Reference-Lineage trailer policy requires; removing it destroys the provenance it exists to record.
• test/test_cli_import*.cpp — those files carry the importer's own denylist literals. Strip the framework names and the check that rejects vendored foreign code silently passes everything.
• VST3 interface names Pulp implements (IPlugView, IPluginFactory). Word boundaries keep them out of the net — they are Steinberg's names, spoken by necessity.

Translation tables (SomeFramework::Widget → pulp::view::Widget) are data and belong in the importer that owns them, never in core/.

If you hit it: rewrite the comment to say what the code does, in units a reader who has never seen that framework can act on. Rename an adopted type and ship a [[deprecated]] alias so downstream keeps compiling — lines containing [[deprecated are skipped, since naming a foreign name in order to point away from it is the mechanism that makes a rename shippable.

--selftest proves the gate can fail (8 cases). A gate that cannot fail is not a gate.

Pre-flight: plugin ↔ CLI skew check

Before shelling out to pulp (or shipyard pr, which ultimately invokes pulp), source the shared skew-check helper so a user on an outdated CLI sees a one-line hint rather than running into obscure flag-missing errors mid-ship:

source "$(git rev-parse --show-toplevel)/tools/scripts/cli_version_check.sh"
pulp_cli_version_check

This is advisory only — the helper never blocks. See the upgrade skill for the full banner contract and override knobs (PULP_SKEW_CHECK_DISABLE, PULP_SKEW_CHECK_CACHE). Release-discovery Slice 6 (#551).

Diagnosing a slow / stuck PR — investigate before assuming runner saturation

When a PR sits without merging, don't ASSUME "the macOS CI pool is saturated" from a long-queued run — investigate. Saturation IS possible (a genuine burst, or a wedged/dead runner — see the pulp-runner-ops skill), so it's worth checking; just don't conclude it without evidence. In the 2026-06-18 case the cause turned out to be non-hardware (a misdiagnosis worth not repeating). Check in this order:

1. Did the required checks even register? A PR opened by the Shipyard GitHub App does NOT auto-trigger pull_request workflows, so the required macos and Enforce version & skill sync checks never appear on the PR head SHA until you dispatch them by hand:

   ghapp workflow run build.yml --ref <branch>             # posts the required `macos` check
   ghapp workflow run version-skill-check.yml --ref <branch>  # posts `Enforce version & skill sync`
   

   This is the most common reason a Shipyard PR "sits." (shipyard pr dispatches build.yml itself but you may still need version-skill-check.yml.) After a new push the head SHA changes — re-dispatch on the new SHA.
2. Is it a version-bump race? The other concurrent agent re-bumping main's CMakeLists.txt VERSION makes the PR DIRTY (conflict on the VERSION line). Merge origin/main in, re-resolve the VERSION to one above main, push, re-dispatch the checks. (Longer-term fix for this whole class: planning/2026-07-07-parallel-merge-land-coordination.md. In progress: the version-at-land.yml bot — currently DRY-RUN — will make a single writer assign the version from Version-Bump: intent trailers post-merge, removing the per-PR race. Until it flips to --apply, keep hand-bumping as today.) 2b. Did a gate reject the push? Besides skill-sync / version-bump, the pre-push + CI planning-gitlink gate (tools/scripts/planning_gitlink_guard.py) fails if the PR moved the planning submodule pointer without a Planning-Bump: trailer — usually an accidental bump from a git reset --hard
   • git add -A. Drop it with git restore --staged --worktree planning && git submodule update planning, or add Planning-Bump: reason="..." for a deliberate re-pin. 2b1. Config→doc drift? The config-doc gate (tools/scripts/config_doc_check.py, map in tools/scripts/config_doc_map.json) fails if a mapped CI/release config surface (.shipyard/config.toml, the shipyard pin/installer, the build/auto-release/version-skill-check/coverage workflows) changed without its guide doc (versioning.md / local-ci.md / release-watchdog.md). Editing a workflow under .github/workflows/ therefore usually needs a matching guide edit; a genuinely doc-irrelevant change bypasses with a Config-Doc: skip reason="..." trailer on any commit in the range. 2b1a. Reaping a release tracker: SHA-keyed ones cannot be judged from the title. The watchdog opens two shapes of tracking issue, and they are reaped differently. A version-keyed tracker ("release: stuck vX.Y.Z") names its version in the title, so the reaper closes it as soon as that tag exists. The stranded fix/feat detector instead opens one tracker per tip SHA and the title carries only the short SHA — a tag existing says nothing about whether that commit is in it. Reaping it requires parsing the issue BODY for the full tip SHA and the uncovered surfaces, then closing only once a later tag for every uncovered surface contains the commit (git tag --contains <sha>) — i.e. consumers can actually reach the change. Closing on "a tag appeared" would mark a still-unreleased change as shipped.

2b2. Hotspot growth? The hotspot-size gate is now net-delta vs merge-base: it fails only if THIS PR grows a frozen hotspot past its reference size (main growing the same file is NOT your fault and passes). If you must grow one, make the change net-neutral (extract to a sibling file) or add Hotspot-Grow: <path> reason="..." — do NOT raise max_loc in hotspot_size_guard.json (that counter no longer gates growth, and raising it races with other PRs). 2b3. Hotspot shrink? The ceiling is a one-way ratchet, and this is the opposite instruction to 2b2. If your PR makes a frozen hotspot smaller, the gate fails with missing hotspot ceiling reduction(s) until you lower max_loc to the new LOC. Lowering is safe (it cannot race a concurrent PR into a false pass); raising is not. So: never raise it, always lower it when you shrink. A PR that both grows one hotspot and shrinks another needs a Hotspot-Grow: trailer and a max_loc reduction. 2c. Is a RED check even your fault? Before investigating a failing check, run python3 tools/scripts/pr_check_triage.py <PR#> — it labels each red check REQUIRED vs advisory and PRE-EXISTING (also red / not run on main — not your change) vs REGRESSED (green on main, red here). Advisory + pre-existing red (e.g. a known-broken sanitizer lane on main) does NOT block the merge and is not yours to fix; only a REQUIRED + REGRESSED row needs action. This alone avoids chasing main-side breakage. 3. Only THEN consider capacity — and verify, don't assume. The required macos gate runs on the local self-hosted Mac Studios (pulp-studio-01/02/03,

• the M5 overflow), which are usually idle. Confirm with:

ghapp api repos/danielraffel/pulp/actions/runners \
  | python3 -c "import sys,json;[print(r['name'],r['status'],'busy='+str(r['busy'])) for r in json.load(sys.stdin)['runners']]"

If the Studios show busy=False, the pool is NOT saturated — say so. What DOES queue independently is the GitHub-hosted advisory lanes (Linux, Windows, sanitizers, coverage, android) on GitHub's shared pool; those are advisory, not the required gate, so a long queue there does not block merge. 4. Is shipyard run/ship failing at stage=configure with D external/skia-build/build? That's the Skia symlink loop (a tracked machine-specific absolute symlink autofetch deletes at configure → tree-drift), not capacity or a code failure — the local mac validation dies before posting its macos status, so the PR stays BLOCKED with the required check absent. Fixed at the repo level by PR #5588 (untracked + .gitignored), but a pre-#5588 checkout keeps the stale looped symlink until it pulls main. Recovery

• full mechanics: the skia-gpu-build skill's Gotchas section (ln -sfn ~/.cache/pulp/skia-build/build <primary>/external/skia-build/build).

If you don't use Shipyard + the self-hosted Mac pool: steps 1 and 3 are specific to that setup (App-dispatched workflows; local runners) — skip them. The tool-agnostic rule still holds: distinguish the required checks from advisory lanes, and verify a runner is actually busy before blaming capacity.

Caveat to step 3 — idle Studios do NOT prove the required gate is unblocked. The required macos check is gated by a routing preamble (classify → resolve-provider) AND is itself an alias job that reports the leg's outcome — all three default to runs-on: ubuntu-latest. So when GitHub's shared ubuntu pool saturates (repo-wide actions/runs?status=queued in the hundreds during a PR burst), the preamble can't get a slot, the macOS leg is never dispatched, and the required gate sits pending with the Studios idle. Tell this apart from a real failure and from a wedged runner: queued preamble + online/busy=false Studios + high repo-wide queued count = GitHub-hosted starvation, not a code or hardware problem. Immediate unblock is admin-merge after local validation (ghapp pr merge <n> --merge --admin, --merge not --squash to preserve the bump marker); merging also drains that PR's ~11-workflow fan-out from the queue. The structural fix is the PULP_PREAMBLE_RUNS_ON_JSON repo var: set it to a self-hosted Linux selector (e.g. ["self-hosted","Linux","ARM64", "pulp-build-linux"]) to move those three jobs off the saturated pool — but only once that pool is confirmed always-on, or the gate just starves elsewhere. It defaults to ubuntu-latest (no-op) until set. A tartci launchd detector watches for this triad; full design in planning/2026-07-06-ci-queue-saturation-watchdog.md.

A dead lane is only visible as queue age — never as a missing runner

.github/workflows/runner-health-check.yml sweeps every 30 min from ubuntu-latest (off-fleet on purpose — a guard on the fleet dies with the fleet) and opens/updates one tracking issue when a lane stops serving work. Read its issue before hand-diagnosing a stuck PR: it names the labels the stalled jobs asked for, which is the "which lane is sick" answer step 3 above otherwise costs you a fleet probe to get.

Do not "improve" this into a runner-label check. The macOS lanes are JIT/ephemeral: a runner registers with GitHub only while it serves a job. gh api .../actions/runners returning zero runners for label pulp-studio-01 is therefore BOTH the healthy-idle state AND the dead-lane state — the two are indistinguishable from GitHub's side. This is a trap that has already been walked into: a label-satisfiability probe was recommended, built on, and used to declare a perfectly healthy lane dead. An empty runner list at 3am is not evidence of anything. Queue age is the only observable that separates alive from dead, and it is cause-agnostic, so it catches failure modes nobody enumerated.

The alarm needs two conditions, and the second one is the important one. An alarm requires age >= 45 min AND no sign of life on the lane (nothing comparable in_progress, nothing comparable started since the job queued). Age alone is not enough: the measured healthy baseline is median 5 min / oldest 31 min / 3 runs past 30 min, so a "queued > 30 min" rule fires every busy afternoon. It also mis-reads one runner grinding a 90-min job with a queue behind it as death. A busy runner is a live runner. If you retune the thresholds, do it in tools/scripts/queue_age_watchdog.py — test_queue_age_watchdog.py pins the measured baseline as a must-stay-quiet case and will fail a tuning that re-introduces afternoon false alarms. Rationale + operator surface: docs/guides/local-ci.md (the config-doc gate maps the workflow and the script to that guide).

Gotcha: a lane pointed at a label NO runner carries is silent — and looks exactly like saturation

The trap behind step 3. Before concluding "the pool is saturated", check that the lane can be served at all. GitHub does not validate runs-on: a job asking for a label no runner carries is not rejected, it is queued — forever. No error, no annotation, no failed check. The only symptom is jobs piling up while the pool looks busy, which is indistinguishable from a genuine burst. So "18 runs queued + every runner busy" is not evidence of saturation; it is equally consistent with a lane routed into a black hole.

Found live 2026-07-16: PULP_OVERFLOW_BUILD_MACOS_RUNS_ON_JSON targeted ["self-hosted","macOS","ARM64","pulp-build","pulp-build-vm"], and zero runners carried pulp-build-vm — the intended Tart-VM topology had drifted to bare-metal (pulp-build-studio) and the variable was never reconciled. The relief valve had been routing into nothing for an unknown period, so the queue it existed to drain simply grew behind it. A relief valve routed into a black hole is worse than none: it reports healthy and relieves nothing.

Check it directly — a runner must carry every label in the array (subset containment, not "any label overlaps"):

# Authoritative: reconciles every lane against the live fleet.
python3 tools/scripts/runner_topology_check.py --mode=report

Watch for three traps when reading this by hand:

• Zero runners ≠ broken. Tart runners register JIT/ephemeral and exist only while a job runs. An idle ephemeral lane has no registered runner and is perfectly healthy — the release lanes look "dead" between releases. Judge those on service history (did a job recently run with that exact label set?), not on the registry.
• Offline ≠ absent. A registered-but-offline runner may just be asleep (m1 is intermittent). A label nothing owns is always a black hole. Different failures; don't conflate them.
• The fleet mutates while you look. Runner count changed between two API calls during this investigation. Re-read before concluding.

Related instance of the same class: build.yml's busy probe needs Administration: Read to call actions/runners; the default GITHUB_TOKEN lacks it, so the probe 403s and falls back to BUSY=0 — silently disabling overflow. Whenever routing "does nothing", suspect a silent read failure before suspecting load.

The standing guard is runner-topology-check.yml (hourly, opens a tracking issue) plus the runner-topology-selftest ctest. Lane→label intent lives in tools/scripts/runner_topology.json — edit a routing variable and its lane together, or the drift check fails. Full rationale: docs/guides/local-ci.md → "Routing contract (checked)".

Host-vitals preflight — back off before a saturating CI host reboots

The self-hosted Mac Studio that runs the required macos gate ALSO hosts the interactive agent session and a heavy MCP stack (RepoPrompt, Figma, chrome-devtools, several pulp-mcp). When RAM fills, macOS jetsam starts killing processes, the window server crashes, and the host reboots uncleanly — taking any in-flight required-gate CI job down with it and reddening the leg for reasons that have nothing to do with the code (the 2026-07-01 incident). This whole class of failure is predictable ~20 minutes out from cheap metrics, so the agent, the CI pool, and Shipyard should all read one shared signal and back off.

tools/scripts/host_vitals.sh is that signal. It reports green / warn / critical (exit 0 / 10 / 20) keyed on memory pressure first (kern.memorystatus_vm_pressure_level + fresh JetsamEvent-* reports), with load only ever corroborating a warn — a healthy parallel build (load 1–2× cores, normal pressure) never trips it. Use it before heavy work:

tools/scripts/host_vitals.sh            # one-line summary; exit code = level
tools/scripts/host_vitals.sh --json     # machine-readable

• Agent admission control: before launching a local build or a heavy MCP call on a CI-host session, check host_vitals.sh. If critical, don't pile on — ship via shipyard pr / GitHub-native auto-merge (survives a restart) instead of a foreground shipyard/ci watch, and shed idle load (close RepoPrompt/Figma /idle MCP) before building. gates.sh prints this banner advisorily on every pre-push.
• A "hung"/"stuck" git push is almost always the pre-push diff-cover BUILD, not the network. When the diff touches a coverage surface (core/, tools/cli/, tools/scripts/), .githooks/pre-push runs a full local configure + compile before the transfer. It prints a loud DIFF-COVERAGE BUILD running banner and a …still running (Ns elapsed) heartbeat every 30s so this is unmistakable — if you see those, it is a BUILD (minutes), NOT a transport hang. Tell: git fetch/ls-remote succeed while the push "hangs", so SSH/HTTPS are fine. Do NOT diagnose the network, and do NOT background-and-kill the push at a short timeout (that hides the banner/heartbeat). When the work is already validated (e.g. via shipyard pr), push with PULP_SKIP_DIFF_COVER=1 git push and it completes in seconds. (Learned 2026-07-07 after ~an hour lost mis-blaming MTU/SSH for a diff-cover build during a deps-bump push.)
• Continuous sensor: tools/scripts/install_host_vitals_sensor.sh installs a per-user launchd agent (com.pulp.host-vitals, 60 s) that publishes the latest reading to ~/.local/state/pulp/host_vitals.json and a rotating host_vitals.log. It is observation-only — it never stops a runner or kills a process — so it is safe to run on the required-gate host. Installed on the m3/m5 /m1 pool. install_host_vitals_sensor.sh --status shows the launchd + latest reading; --uninstall removes the agent.
• A whole-pool-fails-at-once red leg is infra, not code (per macos-required-leg-timeout-saturation): if windows + both macos legs fail together and the diff can't explain it, correlate against host reboot / jetsam time and re-run the required leg rather than debugging the change. Active back-off in the CI pool (tartci auto-yield) and Shipyard (host-health dispatch gate + infra-vs-code classification) consume this same host_vitals state.

GitHub workflow gotchas

• A cache-save step gated on github.event_name != 'pull_request' && github.ref == 'refs/heads/main' is dead code unless the workflow has a push: trigger. GitHub's cloud cache scopes a PR-written entry to that PR's own ref, so PR runs can never warm each other and only a default-branch non-PR run publishes a shared entry. build.yml had that gate with no push trigger, so both Save … steps were unreachable and their Restore … partners were a permanent miss. Two traps when fixing this:

  1. The base must be event-aware, or the fix no-ops. classify diffs <base>...HEAD. On a push to main the checked-out HEAD is the new main tip, so origin/main resolves to HEAD and the diff is always empty. The classifier is fail-closed (empty ⇒ build), so this reads as "working" while never once distinguishing a docs merge from a core merge. Use github.event.before on push, github.event.pull_request.base.sha on PR — tools/scripts/resolve_classify_base.py owns that mapping, including the all-zero-sha fallback that event.before carries when a push creates a ref.
  2. Never let the new trigger schedule work on the self-hosted Macs. They serve the one required check and keep ccache + FetchContent on local disk, so a macOS leg on a push burns required-gate capacity to upload nothing. resolve-provider omits the macOS leg for push events; the aliases and windows-*-gate jobs skip too (the macos alias polls up to 60 min for a leg that isn't there). Scope saves to runner.environment == 'github-hosted' && runner.os != 'macOS' — the restore side's runner.os != 'macOS' || … disjunction is wrong here, it re-admits self-hosted non-macOS runners.

  Push runs are also exempt from cancel-in-progress (they share the refs/heads/main group; cancelling one discards the cache it exists to publish). Covered by tools/scripts/test_resolve_classify_base.py.

• Every on: schedule workflow must carry the fork guard. When someone forks the repo, GitHub copies all workflows and runs the scheduled ones on the fork's default branch — then emails the fork owner whenever one fails, which our monitors reliably do on a fork (they probe this repo's state or use secrets the fork lacks). So each entry job (a job with no needs: — dependents cascade-skip) of a scheduled workflow gets:

  jobs:
    check:
      if: github.event_name != 'schedule' || github.repository == 'danielraffel/pulp'
  

  compose it with an existing condition as if: (github.event_name != 'schedule' || github.repository == 'danielraffel/pulp') && (<existing>). It only suppresses the schedule event on forks — push / pull_request / workflow_dispatch are untouched, so PRs to this repo (which run in this repo's context) and manual dispatches behave exactly as before. A workflow that should run on forks' schedules opts out with a top-of-file # fork-guard-exempt: <reason> comment. This is enforced: tools/scripts/scheduled_workflow_fork_guard_check.py runs in gates.sh, the pre-push hook, and workflow-lint.yml, so a new scheduled workflow missing the guard fails the PR. Add the guard when you add the workflow.

• Codecov "total lines/files dropped" is usually upload starvation, not config drift. Two distinct guard layers exist and they catch different things: test_codecov_components.py / test_codecov_config.py (PR-gate) guard the codecov.yml mapping (a subsystem matching no component → invisible); coverage-upload-watchdog.yml (hourly, main) guards the outcome — "main had a successful Coverage run in the last N hours." The watchdog exists because the dashboard degrades silently when fresh complete uploads stop arriving, from causes the config gate can't see: coverage runs cancelled by stale-run-reaper.yml when a CI dispatch throttle makes them queue past the cutoff (the 2026-06-28 incident — 8 consecutive main Coverage runs cancelled), an llvm-cov -object-set regression (a libpulp-*.a drops out → a whole subsystem vanishes while lines-valid stays > 0, which verify_cobertura_xml.py's lines-valid > 0 check cannot catch), or Codecov's after_n_builds waiting forever on a missing per-OS leg. When triaging a coverage-surface complaint: first check gh run list --workflow coverage.yml --branch main for recent successful runs (cancelled ≠ uploaded); only then suspect codecov.yml. Note the config gate is advisory (not in branch protection), so a fleet-auto-merged PR can land config drift despite a red gate — the watchdog is the main-branch backstop.

• web-plugins.yml is the headless-browser web lane (advisory). It builds Pulp's WAMv2 (Emscripten) and WebCLAP (wasi-sdk) web plugin formats on a Linux GitHub-hosted runner and runs every web validation, including the browser fixtures in headless Chrome (browser-actions/setup-chrome + playwright-core, which drives the system Chrome — no browser download). It is deliberately NOT on the self-hosted VMs: a headless-Linux browser lane needs emsdk + wasi-sdk + Chrome, all of which install cleanly on GitHub-hosted, so no golden VM or QEMU work is warranted. The WAM build vendors choc by cloning the pinned fork (PulpWam.cmake needs PULP_WAM_CHOC_INCLUDE; the WebCLAP build FetchContents choc/clap itself). The browser drivers are examples/web-demos/*/{browser-test,browser-host}/validate.mjs; run them locally with a system Chrome/Canary via node validate.mjs --screenshot out.png (set CHROME_PATH or pass --browser). The WAM node validations also cover the in-worklet rack path (wam_rack_runner.mjs against the PulpPluckGainRack target — the only committed consumer that compiles wam_chain_entry.cpp/WamChainBridge), the wire protocol + version-skew bridge contract (core/format/src/wasm/wam-runtime.test.mjs, pure JS), and hostile-state rejection (overflow/CRC/truncation) folded into wam_feature_runner.mjs — so a regression in the state codec or chain runtime fails here rather than only in a browser.

• The GPU-audio proof inside web-plugins.yml needs a real GPU, so it is a macOS job. The Linux leg has no WebGPU adapter at all (measured: --use-webgpu-adapter=swiftshader and forceFallbackAdapter:true both return null), so validate-gpu.mjs there skips with a named reason and exits 0 — and a non-gating probe step prints what requestAdapter() actually returns, so "does Linux CI have a software adapter yet?" stays a measured question rather than an assumption. The real gate runs on macOS with PULP_REQUIRE_WEBGPU=1, which turns "no adapter" from a skip into a failure. If you see the GPU-audio proof "passing" on Linux, it did not run. Two traps when editing that job: pulp_add_wclap(Foo) declares the CMake target as Foo-wclap (the bare name is a hard "No rule to make target"), and the emsdk llvm-nm must be on PATH or verify_wasm_skia_slice.py — which exits 77 to mean CTest-SKIP — kills the step under set -e instead of skipping.

• wclap-cloudflare.yml is WebCLAP's canonical-host lane (Cloudflare Pages). Separate from web-plugins.yml: it builds the threaded WebCLAP PulpGain module (pinned wasi-sdk), assembles a self-contained deploy dir, proves it headlessly under the real _headers, then owner-gated-deploys. Two pages are assembled + proven: the single-plugin isolation page (assemble.mjs → validate-deploy.mjs) and the shared-player WebCLAP demo (assemble-player.mjs → player/ → validate-player.mjs) — the latter mounts the SAME @danielraffel/web-player shell the WAM demos use, driven by the WebCLAP adapter (a worklet-resident CLAP host), and asserts crossOriginIsolated, real-time render (worklet quanta ≈ sampleRate/128 per wall-second), an audible param change that updates the shared widget, and a clap.state round-trip. The lane also runs the @danielraffel/web-player adapter unit tests (packages/pulp-web-player — npm test), which include an ABI-parity check that the worklet's inlined CLAP struct offsets match wclap-abi.mjs. Hard-won gotchas encoded there: posting a WebAssembly.Module INTO an AudioWorklet is silently dropped in Chrome (transfer the raw bytes and compile in-worklet), and transferring an ArrayBuffer OUT of an AudioWorklet is unreliable (the receiver gets a detached buffer) — clone small payloads (state, sysex) instead, and never transfer a caller-owned buffer (it detaches theirs). It also needs TWO EXTERNAL CHECKOUTS, and forgetting them is a silent build with a loud, misleading failure. The gallery's 23 plugins do not live in pulp — they are in the public danielraffel/pulp-example-plugins and danielraffel/pulp-classic-effects repos (the examples-out-of-core split), and wclap-build/CMakeLists.txt declares each gallery target ONLY IF its plugin header is present, skipping it quietly otherwise (deliberately — that is what keeps the in-repo build green without the externals). So a job that checks out pulp alone builds exactly PulpGain, skips all 23 without a word, and then dies hundreds of lines later in assemble-gallery.mjs with "missing wasm for example-plugins/mono-synth" — which reads like an assembler bug and is not one. The two roots default to SIBLINGS of the pulp checkout, which actions/checkout cannot produce (a path: cannot escape the workspace), so clone them under _ext/ and pass -DPULP_EXAMPLE_PLUGINS_DIR / -DPULP_CLASSIC_EFFECTS_DIR. The lane now asserts a couple of gallery wasms exist right after the build, so the cause is reported where it happens rather than downstream.

• screenshot-sync is a three-layer gate that mirrors skill-sync. A repo opts in by committing a .pulp/screenshots.toml manifest (presence == opt-in); tools/scripts/screenshot_sync_check.py then diffs the manifest's [trigger].paths and fails when a triggered target's committed PNG/OG image wasn't refreshed. Wired identically to skill-sync: PostToolUse hint in hooks/scripts/cli-plugin-sync.sh, pre-push report in .githooks/pre-push, and authoritative CI step in version-skill-check.yml. Bypass a single commit with a Screenshot-Sync: skip target=<id|all> reason="..." trailer. Pulp core itself is NOT opted in (no manifest), so the gate no-ops here; it exists for downstream plugin repos (GPU NAM, example plugins, Bendr) and the WCLAP/WAM OG images.

• Android native .cxx caches must be dependency-aware. The Android workflow builds through Gradle's external native build, and android/app/.cxx can hold FetchContent checkouts under _deps. Do not cache .cxx under a Gradle-only key or a broad Gradle restore key: after a native dependency bump, CMake's UPDATE_DISCONNECTED path can see the stale local checkout and fail because the new git ref is not present. Keep the Gradle cache (~/.gradle) separate from the native CMake cache, key .cxx on the native dependency inputs (CMakeLists.txt, tools/cmake/PulpAndroid.cmake, tools/cmake/PulpDependencies.cmake, tools/deps/manifest.json, plus Android Gradle files), and do not give .cxx a restore key that ignores those inputs.

• version-at-land.yml + version_at_land.py are the single-writer, post-merge half of the version-bump intent-trailer model — and the workflow ships in DRY-RUN. They exist to kill the version-bump merge treadmill (PRs editing CMakeLists VERSION / plugin.json / marketplace.json re-conflict every time main advances, and N parallel PRs endlessly re-bump the same shared counter). The endgame: a PR declares Version-Bump: <surface>=<level> and touches NO version files, and this bot assigns the exact number AFTER merge from main's current version — so no two PRs ever contend for the same number. Today (dry-run): computes and logs what it WOULD assign; writes and pushes nothing, so it is safe to land while PRs still hand-bump. The --push path (apply_and_push) is built and unit-tested but the workflow does not call it yet.

  • Intent is read --no-merges-scoped. version_at_land.intent_trailers reads Version-Bump: trailers only from the range's NON-merge commits (git_range_trailers(..., no_merges=True)). A "Merge origin/main into " re-sync commit can carry a stale intent trailer that was never meant to declare this range's release; honoring it would silently escalate the version. A PR's real intent lives on its own commits (or the squash commit, single-parent), so this keeps every genuine declaration while dropping re-sync noise. Do NOT change git_range_trailers' default (merge-walking) — the bypass-trailer gates depend on it; use the opt-in no_merges= flag.
  • The --push transaction is race-safe by construction: it recomputes from a fresh origin/main each attempt, pushes with no --force (git's default non-fast-forward rejection IS the --ff-only guarantee), and on rejection re-syncs to the new tip — where the drain range now starts after the winner's Version-Bump-Applied marker and collapses to empty, so the loser no-ops instead of double-bumping. This is why the older intent-bump-on-merge.yml was DELETED: it did a bare git push origin HEAD:main with no --ff-only + retry, so a second merge during its ~30s window silently discarded the bump (a SILENT RELEASE LOSS). Never reintroduce a force/unguarded push on this path.
  • Before the --push flip (a separate, reviewed change — see docs/guides/version-at-land-cutover.md): verify the release bot can push a commit to protected main (it already pushes tags from auto-release.yml; a commit needs the bot on the branch-protection bypass list, and the bot commit must be SSH-signed via configure_release_bot_ssh_signing.sh), that the Version-Bump: trailer survives squash-merge into main's message, and that the one-time in-flight straggler rule (PRs already carrying chore: bump versions commits) is applied. The chore: bump versions commit the bot pushes triggers auto-release.yml exactly like a PR-side bump.

• test/CMakeLists.txt is now an include hub, not a registration manifest. Add new add_test, add_executable, and pulp_add_test_suite blocks to the matching test/cmake/*_tests.cmake manifest instead of rebuilding the old monolith. If no existing manifest owns the subsystem, create a focused new test/cmake/<area>_tests.cmake file and include it from test/CMakeLists.txt in dependency order. Focused owner hubs may include smaller manifests for their own subsystem, but do not hide sibling manifests inside an unrelated owner just to keep the top-level file short. Keep the top-level hotspot_size_guard.json ceiling at the exact tiny hub LOC; raising it for ordinary test additions is a regression. Before burning a PR CI job on a manifest-only cleanup, run the local preflight: python3 tools/scripts/docs_noise_lint.py --mode=report --base origin/main, python3 tools/scripts/hotspot_size_guard.py --base origin/main --config tools/scripts/hotspot_size_guard.json --mode=report --require-ceiling-reduction, and a clean CMake configure/target-list compare against origin/main.

• Source hotspots (e.g. core/view/src/design_cpp_codegen.cpp) are frozen too — bump the ceiling for a coherent feature, split when it's accretion. hotspot_size_guard.json also freezes large source files. When a single, cohesive feature legitimately grows one (e.g. teaching the C++ codegen to emit faithful_svg as a DesignFrameView), raise that file's max_loc in the same PR — splitting a tightly-coupled emitter mid-feature would hurt readability more than the extra LOC. Reserve the split for genuine grab-bag growth. When a PR shrinks a tracked hotspot, lower that file's max_loc in hotspot_size_guard.json to the new exact LOC in the same PR; the --require-ceiling-reduction gate compares the merge-base blob against HEAD for branch-touched tracked hotspots, so leaving headroom after a shrink fails the pre-push/CI guard. Exception — a hotspot already ABOVE its ceiling. check_hotspots only fails the PR that itself grew a file, so main can drift past a max_loc while every individual PR stays net-neutral (e.g. window_host_mac.mm at 2780 LOC against a 2770 ceiling). Shrinking such a file cannot lower the ceiling — setting max_loc to the new LOC would raise it — so the guard does not ask for a reduction there, and prints still at/over ceiling; no reduction to make. The ratchet is demanded only when the shrink lands strictly under the existing ceiling. If you see the guard demand a max_loc larger than the one in the config, that is the bug this rule fixed, not a config you should edit. Editing hotspot_size_guard.json itself trips the ci skill-sync gate; the ceiling bump is normally part of a non-ci feature PR, so either fold this note's rationale into that PR or skip the ci gate with a Skill-Update: skip skill=ci reason="ceiling bump only" trailer on the tip commit (note: a later chore: bump versions commit from shipyard pr displaces the tip, so updating this SKILL is the more robust path).

• Inspector hotspots are frozen too. hotspot_size_guard.json watches newly added inspect/** files and freezes the current inspector overlay, window, domain handler, and tweak-store hotspots. When an inspector extraction shrinks a tracked file, lower its max_loc to the exact new line count in the same change. When a new overlay companion file triggers the large-file warning, split it before it becomes another hotspot.

• Reskinnability ratchet (token-coverage-ratchet ctest). Driven by tools/scripts/token_coverage_check.py: fails if a core/view/src paint file gains a NEW colour literal that is not a resolve_color(...) fallback. Mark a deliberate material-effect literal // token-lint:allow; after intentionally lowering a file's count, re-freeze with --update-baseline.

• SignalGraph single-backend governance (single_backend_guard.py + processing_model_terms_lint.py). Whole-tree structural guards (not diff-scoped) that keep the DSP runtime convergence from regrowing a second surface: exactly one type (GraphRuntimeExecutor) may define process_routed / process_parallel; pulp create offers no graph-plugin authoring scaffold; the public generated-DSP ABI entry symbols stay the sanctioned pair (pulp_native_core_entry_v1, pulp_node_v1_entry); and the differential parity test stays registered as a built target. They run in gates.sh, the pre-push hook, and the Versioning & Skill-Sync workflow (hard-fail). To sanction a genuinely-new routing engine, authoring template, or generated-DSP ABI, widen the allowlist in single_backend_guard.py in the same PR — that diff is the architecture-review record. Both guards carry a --selftest run in the workflow's fixture-test step. Hardening notes (from the series adversarial review): a non-sanctioned Rival::process_routed is flagged even when defined inside graph_runtime_executor.{hpp,cpp} (no TU-filename escape), and the parity-registration check requires the test source to appear on a real SOURCES/add_executable/target_sources/pulp_add_test_suite line — a bare mention in a dead variable no longer counts.

• Conflict-marker guard (conflict_marker_check.py). Whole-tree guard (not diff-scoped): no tracked file may carry a git conflict marker. Born from the incident where a squash-merge's stale side collided with an already-advanced project() VERSION line and wrote <<<<<<< / ======= / >>>>>>> straight into CMakeLists.txt (fixed in #5477), breaking every build until a human noticed. Keyed on the start/base/end markers (<<<<<<< / ||||||| / >>>>>>> at column 0, followed by whitespace/EOL) — verified zero-false-positive across the whole tracked tree, external/ included; the ======= separator is reported only inside a real conflict block, so Markdown headings and ASCII banners stay clean. Runs in three layers: gates.sh + the pre-push hook (local), the Versioning & Skill-Sync workflow's Conflict-marker guard step (which scans the pull_request MERGE ref, so a marker born from the merge itself is caught, not only one on the PR head), and conflict-marker-guard.yml — a push:main backstop that reddens the branch and opens a tracking issue if a marker reaches main by any path (the squash case, where GitHub had no clean mergeable ref for the PR job to inspect). Carries a --selftest in the fixture-test step. A vendored fixture that legitimately ships markers is a reviewable ALLOWLIST edit in the script. If the push:main guard reddens main, run python3 tools/scripts/conflict_marker_check.py for the file:line list, resolve, and push — the tracker auto-closes on the next clean push. Exit codes are meaningful: 0 clean, 1 markers found (the backstop opens the tracker), 2+ internal error (the backstop fails the run without opening a wrong "markers committed" issue) — the selftest locks this contract. Documented scope limitations (deliberate, to keep zero false positives): only default seven-char markers (a non-default .gitattributes conflict-marker-size is missed), submodule contents are out of scope (the superproject sees a gitlink), and NUL-bearing/UTF-16 text is skipped as binary. The upstream fix for the whole class — the merge tool refusing to commit a conflicted result — is tracked in Shipyard #372; this guard is the consumer-side backstop.

• Thread-safe-assertions guard (thread_assert_check.py). Catch2 3.7.1's assertion macros are NOT thread-safe (thread-safe assertions are opt-in only from Catch2 3.9.0), so a REQUIRE/CHECK/FAIL/SUCCEED inside a std::thread / std::jthread / std::async lambda in test/ is undefined behavior: it corrupts Catch's per-run assertion counters + section state. Bare metal tolerated it, but the move of the required macOS gate onto Tart VMs made the different scheduler timing trip it — a HotSwapSlot hammer test "failed" in 0.00s though the code under test was race-free by design. The correct pattern: record the violation into an atomic/guarded value in the worker, join(), then assert on the test thread (REQUIRE_FALSE(bad.load())). The lint is a lexical scan (best-effort: it does not follow calls into helpers) with a same-length string/comment-blanking pass that also skips C++ digit separators (10'000); suppress a verified-safe line with a trailing // thread-assert:allow. Runs as the thread-safe-assertions ctest case and in gates.sh. When graduating any required lane to VMs, expect this class of latent UB to surface — fix at the source, don't suppress.

• Release builds must pass -DPULP_BUILD_EXAMPLES=OFF. The pulp-design-tool example hard-fails CMake configure when PULP_HAS_SKIA is FALSE (belt-and-suspenders, code 78). sign-and-release.yml builds on a GitHub-hosted macOS runner with no Skia, so configuring the full tree (examples included) aborts the entire run → no GitHub Release publishes even when release-cli.yml (the CLI build) is green. This silently broke Release publication (the release watchdog flagged the missing Releases). The release ships the CLI + plugins (build/VST3,/CLAP,/AU), never the example apps, so the release legs build with -DPULP_BUILD_EXAMPLES=OFF (matching build.yml). If you add a new release/packaging workflow, configure with examples OFF (or a populated SKIA_DIR), or the design-tool Skia gate will block it.

• Release/SDK builds must pass -DPULP_ENABLE_AUDIO_PROBES=OFF. The standalone audio probe is a dev/example inspection surface and defaults ON for local development, but shipped CLI, standalone, and SDK artifacts must not export it. Keep release-cli.yml, release-path-pr-gate.yml, release-dry-run.yml, sign-and-release.yml, release-cli-local.sh, and checkout-backed SDK configure paths aligned when touching release CMake flags.

• The install-consumer smoke must compile against the installed prefix, not the source tree. When an exported SDK target gains or exposes public headers, add representative include/pulp/... existence checks and include those headers from the generated consumer probe. Link the exported targets that own the APIs as well. This catches a target such as Pulp::host being exported with a library while its public headers are accidentally omitted from the install manifest. Keep tools/validation/sdk-smoke in sync so the same proof is runnable locally without GitHub Actions.

• sign-and-release.yml does NOT wait on the release any more — do not add the poll back. It used to poll gh release view "$TAG" until release-cli created the release, so it could attach appcast.xml. That poll ran on the macOS notarize job, which holds the single release VM — the same VM release-cli's darwin-arm64 leg needs to produce the release being waited for. It was a circular wait, and every tag where signing won the race deadlocked until the poll timed out.

  release-cli.yml now writes appcast.xml itself (the Sparkle feed is a pure function of the tag name, so nothing about it needed macOS signing) and is the sole owner of publication end to end. sign-and-release.yml holds contents: read and cannot write a release at all: it may fail, hang, or be cancelled without affecting whether the SDK ships.

  Widening a poll timeout was the old mitigation and it is exactly backwards — a longer wait squats the scarce VM for longer. If you find yourself raising a timeout on the release VM, stop: move the wait off the VM instead.

• Hooks inherit GIT_DIR — tests that shell out to git can corrupt the live worktree. Git exports GIT_DIR/GIT_WORK_TREE into hook environments, and a set GIT_DIR overrides git -C <dir> discovery. So when the pre-push hook (or shipyard) runs the full ctest and a test does git -C <tempdir> init/commit/checkout, those commands silently hit THIS worktree's repo instead — stray initial commits, throwaway branches, and a core.bare=true flip in the shared config (which makes the main checkout look "not a work tree"). .githooks/pre-push and tools/scripts/local_diff_cover.sh now unset GIT_DIR GIT_WORK_TREE … before running gates/ctest, and the git-shelling tests scrub the same vars in-process (see the regression test in test_mcp_server.cpp). If you add a test that shells out to git on a temp repo, clear the inherited git env first (or run from a context with no GIT_DIR), and never assume -C alone isolates it. Recovery if a worktree was hit: git config core.bare false, reset the branch off the stray initial commit, delete the throwaway branch.

• The required macos alias in .github/workflows/build.yml mirrors the macOS matrix leg by polling the Actions jobs API. Keep that poll retry-safe: API failures or malformed JSON must log and continue the loop, not trip set -euo pipefail before the macOS leg has a chance to report.

• .github/workflows/release-dry-run.yml (P9-2, #2576) exercises the release build → package_cli.py → pulp ship appcast chain on a synthetic version (0.0.0-dryrun) WITHOUT publishing — no GitHub release, no signing/notarize, no appcast upload; artifacts are throwaway. It's additive (does not touch release-cli.yml / sign-and-release.yml) and runs weekly + on demand, so a build/packaging/appcast-generation regression surfaces BEFORE a real tag. Keep it credential-free (notarize/sign stay in the real path) so it can run on a schedule without secrets.

• Release-bot source refs must be SSH-signed. auto-release.yml creates signed annotated v* tags with git tag -s, and the bot commit workflow (post-tag-sync.yml, and version-at-land.yml once flipped to --push) configures the same SSH signing helper before committing. The required Actions secret is RELEASE_BOT_SSH_SIGNING_KEY; it is a file-backed OpenSSH private key backed up outside the repo. The workflow uses [e-mail] because GitHub verifies SSH signatures against the account that owns the uploaded signing key. If this secret is absent, the signing setup must fail closed rather than create unsigned release tags or bot commits.

• Release-runner Xcode must be pinned (C++20 parity). sign-and-release.yml runs on GitHub-hosted macos-14, whose DEFAULT Xcode is 15.4 — its Apple clang lacks C++20 P0960 (parenthesized aggregate init, Type p(arg) for a ctor-less aggregate). The self-hosted PR macos lane uses a much newer clang that accepts it, so a CLI/import TU compiled on every PR but FAILED only in the release build — silently breaking GitHub Releases v0.372–v0.391 (tags kept being cut; only the release-cadence watchdog noticed). The job now selects the newest installed Xcode 16.x via xcode-select (shell, no third-party action so an actions-allowlist can't hold the release hostage), restoring C++20 parity with the PR lane. When a release/packaging workflow builds C++ on a GitHub- hosted macOS runner, pin a modern Xcode — the default lags and silently diverges from the PR toolchain. (Code-side defense: always brace aggregate init Type p{arg} in CLI/import code — see the import-design skill.)

• The release build is NOT a test gate. sign-and-release.yml no longer re-runs the unit suite (ctest). By the time a commit is tagged it has already passed the FULL suite on the PR/merge gate (self-hosted lane, real GPU/display/iOS-SDK). Re-running on the HEADLESS GitHub-hosted release runner is redundant and yields false failures from environment-only tests (Skia-raster screenshot → empty, cmake-require-gpu → timeout, cmake-ios-hostapp-links) that pass on real hardware — that blocked Releases AFTER the Xcode-pin let the build through. Principle: tests gate at PR on representative hardware; the release builds + signs + notarizes + packages the validated commit (the Build step is the release-config compile smoke; validate.yml gates format validators). The replacement gate is a built-ARTIFACT smoke (the "Smoke built plugins" step): it nm-reads each built build/CLAP/*.clap and FAILS only if a Mach-O was produced without its clap_entry C-ABI export (a real linkage regression), warning-and-passing when no artifact is found (a path/setup miss must never re-block the release). Static symbol read = NO execution, so it's headless-safe — never use dlopen+init (loads GPU/Skia libs the headless runner lacks) or re-run the hardware-dependent ctest suite.

• Sandbox E2E macOS has a long cold C++ CLI build. .github/workflows/sandbox-e2e.yml builds the pulp-cli target before running the Python sandbox harness. On GitHub-hosted macos-latest, a cold C++ CLI build can exceed 30 minutes and be reported as cancelled with The operation was canceled plus orphaned clang processes, before pytest starts. Treat that as a job timeout/build-cost issue, not a sandbox assertion failure. The workflow uses timeout-minutes: 60 so the cold macOS build has room to finish while still bounding genuinely wedged runs.

• .github/workflows/header-self-contained.yml (pulp #2576) is a BLOCKING gate for the "compiles on Apple Clang, breaks on Linux" transitive-include class (e.g. uint32_t without #include <cstdint> — broke the v0.197.4 release). It compiles each PR-changed public header standalone with Linux Clang via tools/scripts/check_headers_selfcontained.py. Unlike the advisory IWYU gate it only fails on a header that genuinely won't compile alone (no "unused include" false positives), so it is safe to block on. Headers whose module isn't in the GPU-off compile DB are skipped, not failed.

• Windows FetchContent subbuilds need a short base dir. GitHub-hosted Windows runners can still route MSBuild metadata through the legacy 260-character path limit. The wgpu prebuilt-populate subbuild has deeply nested stamp paths, so build-windows/_deps/... can exceed MAX_PATH during configure even before compilation. build.yml passes -DFETCHCONTENT_BASE_DIR="$PWD/fc" only on Windows to keep dependency subbuilds short while preserving the normal build-windows artifact/test directory.

• .github/workflows/watchdog-reaper.yml (pulp #2576) sweeps ALL open release watchdog trackers daily and closes any whose version is released or superseded — the existing watchdogs only auto-close inside a recent window, so historical per-version trackers orphaned (334 had accumulated). It only matches the exact auto-generated tracker titles and only closes objectively-resolved ones. SHA-keyed release: stuck trackers carry no version in their title; the reaper skips them without failing the sweep. Close those only after verifying the affected SDK/plugin surface shipped the stranded commit.

• Keep watchdog/issue-maintenance workflows on REST gh api calls. Avoid gh issue list / gh pr * helpers in those paths because they can use the shared GraphQL quota; a watchdog must not fail while reporting that the watched workflow already recovered.

• Pulp's docs site is deployed by .github/workflows/docs-deploy.yml. If GitHub Pages is set to legacy main/docs builds, the generated Pages checkout tries to clone the private planning submodule with the default token and fails before docs deploys. Fix that at the Pages configuration layer (build_type=workflow), not by weakening normal submodule checkout.

• .github/workflows/sanitizers.yml runs broad ASan/UBSan matrices, but its exclude regex may carry narrow sanitizer-lane skips for non-memory-safety failures that remain covered by regular Build/Test. Keep those skips named exactly to the failing CTest cases and leave comments explaining the alternate coverage path; do not use sanitizer excludes to hide new targeted coverage tests.

• Flaky-lane retry (de-flaking). The ASan/UBSan/TSan lanes (sanitizers.yml) and the coverage lanes (scripts/run_coverage.sh) run ctest with --repeat until-pass:2. Timing-sensitive tests intermittently fail under sanitizer slowdown (a different test each run — RenderLoop coalescing, ImageView fill, etc.), and a single failure aborts ctest and cascades into the diff-coverage gate (partial profile → 0% on the diff), leaving every PR UNSTABLE even when the required gates pass. until-pass:2 retries a failed test once so a flake passes on retry while a genuine failure still fails both attempts (no masking). This complements the per-lane --exclude-regex flake lists by catching not-yet-listed flakes generically — prefer it over growing the exclude list for transient timing flakes.

web-plugins.yml — pin the wasm toolchain, and fetch Skia from the manifest

The WAM / WebCLAP / browser-UI lane (WAMv2 + WebCLAP (Linux, headless Chrome)) has two rules that exist because breaking them produces failures on PRs that touched nothing:

• emsdk is PINNED, never latest. emsdk floats, and has already shipped breaking WebGPU API changes and a change to how SINGLE_FILE embeds the wasm. Each landed here as a mystery red X on an unrelated PR. The lane pins the version it is verified against (6.0.2 today); 4.0.10 is the floor for the emdawnwebgpu port, so a bump stays at or above it. Bump it deliberately, in its own PR, and update determinism.web_toolchain in tools/deps/manifest.json in the same change.
• The Skia wasm slice is fetched from tools/deps/manifest.json, not a hardcoded URL. The lane reads the URL + sha256 out of the manifest, verifies the checksum, and keys the actions/cache entry on hashFiles('tools/deps/manifest.json') — so the CI lane and the dependency audit can never disagree about which binary is in use, and a pin bump self-invalidates the cache. Do not paste a release URL into the workflow. The lane then runs tools/scripts/verify_wasm_skia_slice.py to assert the slice really is Ganesh/WebGL2 (no Dawn, no Graphite) — the invariant FindSkia.cmake's Emscripten arm is built on. See the skia-gpu-build skill.

Path-filter hygiene: this workflow is paths:-filtered, so a new web-facing directory (a demo, a packages/pulp-web-player/** change, a new PulpWeb*.cmake module) is not covered until you add its glob. A web demo that silently stops being built is the failure mode.

NEVER set run-name: on release-cli.yml (it stops all releases)

GitHub returns a workflow's run-name as workflow_run.name in the REST API — it REPLACES the workflow name, it does not sit alongside it. The self-hosted tartci supervisor that provisions release-cli's macOS VMs picks up its work with:

select(.name == "Release CLI")

So the moment a run-name is set, every release run becomes invisible to the supervisor. Its log reads queued=0 running_macos_vms=0/2 while release runs sit with their required darwin-arm64 leg queued forever. No macOS VM is booted, no runner appears, and no release can ever build — silently, for every future tag.

This shipped once (a run-name was added so release-reconcile.yml could attribute its repair runs, since a workflow_dispatch run's head_branch is the ref it was dispatched FROM, not the tag it builds). The tag now travels in a job name (resolve-macos-runner), which the API exposes as a separate field and the supervisor does not key on. tools/scripts/test_release_workflow_test_step.py asserts run-name never returns.

The general rule: a workflow's public identity (name, and therefore run-name) is an interface that self-hosted infrastructure keys on. Renaming a run is not cosmetic. Before changing it, grep the tartci supervisor config (TARTCI_RUNNER_WORKFLOW_NAME) for anything matching on it.

Keep ONE -O0 lane: it sees UB that -O3 provably hides

.shipyard/config.toml's macOS validation lane builds Debug (-O0). This contradicts CLAUDE.md ("Release is the default") and reads like config drift — a config audit will want to flip it. Don't. It is the only lane in CI that can see a whole class of undefined behaviour.

2026-07-12 (#6081): a macro-gated inline function template in a header (snap_to_zero()), plus a test TU that redefined that macro before including it, is an ODR violation — two TUs emit the same mangled symbol with different bodies.

-O3   each TU inlines its own copy -> each behaves per its own macro
      the A/B test appears to work. RELEASE IS GREEN. Invisible by construction.
-O0   nothing inlines -> both emit a weak symbol, the linker keeps ONE arbitrarily
      the "disabled" reference silently ran the enabled code. DEBUG IS RED.

The red test was the mild outcome: had the linker kept the other definition, the assertions would have PASSED while exercising a no-op — a null test asserting nothing, green forever.

The fix shape is not "delete the redefine" — it is give the variant its own binary, compiled consistently end to end, linking no default-built TU (see test/denormal_null_refgen.cpp). Guarded by tools/scripts/test_odr_macro_gated_headers.py, which only flags macros that gate an inline/template function body (a macro guarding an #include or a platform block cannot cause ODR).

A perf gate failing on that lane is a MIS-CALIBRATED GATE, not a reason to flip the lane. test_yoga_layout_bench's timing threshold was sized at ~11x a Release baseline; in Debug the same pass is ~11.6x slower, eating the entire margin, so it sat permanently at the edge and load merely tipped it. It is now #ifdef NDEBUG-gated (the GitHub macOS lane is Release, so real coverage is preserved), while the structural assertions still run in every build.

A false red is worse than no gate: it trains everyone to wave away red as "probably the box" — which is exactly how a real bug gets dismissed.

Never let a flaky advisory leg decide a run's conclusion

nightly-intel concluded cancelled on every scheduled run for a long time, which reads as "this workflow produces no Intel coverage". That reading is wrong, and the trap is worth internalising:

Universal + lipo + dual-arch auval (macos-15) : SUCCESS    <- every night
Native Intel build + test (macos-15-intel)    : cancelled  <- its 120m job TIMEOUT
Intel nightly watchdog                        : SUCCESS

universal-crosscheck — the arm64+Rosetta lipo + dual-arch auval signal that release-cli.yml relies on after the per-tag universal gate was removed — succeeds every night. The Intel signal was there the whole time, buried under a run-level conclusion poisoned by a different leg.

native-intel on macos-15-intel had never once completed: that image CPU-pegs, so the job hit its 120-minute limit every run. GitHub reports a job timeout as cancelled, and a cancelled job cancels the whole RUN. So a leg producing zero signal was deciding the conclusion of the leg producing the real one — while burning two hours of a scarce Intel runner nightly.

Fix pattern: bound the work in the STEP, not with the job timeout.

if timeout 75m cmake --build "$BUILD_DIR" -- -k 0 2>&1 | tee build.log; then
  echo "status=pass" >> "$GITHUB_OUTPUT"
elif [ "${PIPESTATUS[0]}" = "124" ]; then     # WE killed it, not GitHub
  echo "::warning::runner pegged — INFRA, not a product failure"
  echo "status=infra-timeout" >> "$GITHUB_OUTPUT"
fi

The job then finishes normally with a loud, explicit infra-skip, and the run can reach a conclusive success/failure. Do not reach for job-level continue-on-error here: GitHub documents it for a job that fails, and a timeout is reported as cancelled — whether it covers that is a semantics gamble. Bounding the step is correct by construction.

A silent cancel is indistinguishable from "this workflow does nothing" — which is exactly how working coverage got written off as absent.

Current Build-and-Test routing

As of the 2026-05-20 classify gate, build.yml runs a cheap classify job before allocating the native matrix. PRs that touch only skip-safe docs/planning paths report the native aliases green without running macOS/Linux/Windows builds; PRs that touch C++/Swift/CMake or workflow inputs must run the native matrix. A code PR whose native build is skipped is a CI bug.

workflow_dispatch defaults runner_provider to github-hosted, not Namespace; do not dispatch with runner_provider=namespace. Linux/Windows use GitHub-hosted runners by default. macOS routes through the self-hosted pulp-build pool (pulp-m1-01, pulp-m1-02) via PULP_LOCAL_MACOS_RUNS_ON_JSON; repository variables control any overflow.

The authoritative Windows x64 functional matrix is pinned to windows-2022 (Visual Studio 2022) by .github/workflows/build.yml. Shipyard's current ship path does not send a windows_runner_selector_json, so the workflow fallback is the operative selector. .shipyard/ci-profiles/normal-local-fast.toml mirrors that PR policy with its PR-only github.windows-x64-runtime target for profile inspection; the current profile planner is read-only and does not override dispatch. Its shared coverage/scheduled github.windows-x64 target remains windows-latest. Keep those mirrors truthful, but never rely on them to route a run. The standalone MSVC release-path, MIDI 2, and BLE compile gates remain on windows-latest, as do release builds, so the newest hosted compiler and SDK are still exercised without allowing an in-place runner-image migration to change the CRT/toolchain beneath the complete runtime suite. tools/scripts/test_windows_runner_policy.py reads every operative surface (build, release, coverage, nightly, release resolver, and Shipyard profile) independently and runs in the PR workflow-lint lane. Update that one cross-surface invariant whenever the split changes; a profile-only or workflow-only assertion is not enough because the two can self-agree while a different production lane drifts.

Do not push empty commits just to churn queued macOS checks. Cancel superseded SHAs, rebase or push only when a PR needs current main, and wait unless a check has actually failed.

Gotcha: .shipyard.local can silently route the macOS lane off the gate

.shipyard/config.toml declares [targets.mac] backend = "local", and Shipyard merges the gitignored .shipyard.local/config.toml on TOP of it. A [targets.mac] block there overrides the backend — and the failure does not look like a misconfiguration, it looks like a hang: shipyard status reports mac: cloud, shipyard watches a redundant GitHub-hosted run and times out at 3600s, and the required macos check (posted ONLY by the local runner) never appears. That is why the block in Pulp's own .shipyard.local/config.toml is commented out and annotated DISABLED 2026-07-09.

A missing .shipyard.local/ is the CORRECT state, not a gap — the repo config's local mac target stands on its own, which is how the Mac Studio runs. Do not "fix" a fresh worktree by copying a config in; that is what causes the reroute. In particular never cp -R over .shipyard.local/ — config.toml is gitignored but config.toml.example next to it is TRACKED, so a recursive copy clobbers a tracked file.

tools/scripts/gates.sh runs shipyard_local_check.py as an advisory that reports an active non-local mac override before you push. It is read-only by design: it never copies or repairs, because the repair instinct is the hazard.

coverage.yml's macOS leg resolves its runs-on via resolve_runs_on.py --deny-labels pulp-build,pulp-build-vm: a coverage selector (repo var PULP_COVERAGE_MACOS_RUNS_ON_JSON or a dispatch input) that names the shared gate pool fails the resolver fast rather than letting a long advisory coverage run contend with the required macos check. The dedicated coverage lane uses pulp-coverage-vm-macos; a bare GitHub-hosted label (macos-15) is never denied. (Push-triggered coverage on a busy main is designed to be superseded while queued — the supersession-immune scheduled run, cron 17 */8 * * *, is the one that produces the green full-matrix upload that clears the coverage-stale watchdog.)

The os-windows coverage leg is best-effort. The instrumented MSVC build + ~9k instrumented tests + llvm-cov over 1000+ objects exceeds the 150-min job cap on GitHub-hosted windows-latest (it is ~1h on Linux/macOS), and the staleness watchdog keys off a successful run, not per-OS Codecov flags — so a red Windows leg would otherwise keep a healthy full run red forever. The subtle trap (verified by canary): job-level continue-on-error does NOT neutralize a timeout-minutes cancellation — a cancelled job still makes the run conclude cancelled. It DOES neutralize a normal job failure. So the coverage suite step self-terminates at an internal budget (135 min) below the job cap, turning the would-be cancellation into a normal non-zero exit that the job-level continue-on-error: matrix.os=='windows' then absorbs → the run concludes success. Don't "simplify" this to bare continue-on-error; it will silently stop closing the watchdog. And the watchdog that enforces the budget must separate its steps with ;, NOT && — if the kill is &&-gated behind a : > marker write (which can fail on a Windows RUNNER_TEMP backslash path), a failed marker write short-circuits the chain and the suite is never killed, so the job hits the 150-min cap and is cancelled anyway. The kill is mandatory; the marker is best-effort (cleanup of any partial Cobertura also triggers on a 143/137 signal-kill exit, not just the marker). Real os-windows correctness bugs are still worth fixing (the ARG_MAX response-file + vanished--object existence-filter + mass-drop guard in run_coverage.sh were real); only the runtime/timeout is accepted as best-effort.

release-cli.yml needs Rust; the VM golden does not have it

The macOS build gate and release-cli.yml's darwin leg can share the Tart VM golden (pulp-build-runner), but that golden is provisioned only for the C++ "Build and Test" lane — Xcode, homebrew cmake/ninja/node, baked Skia, and no Rust toolchain (manifests/pulp.macos.toml [brew] has no rust/rustup). build.yml stays green on it because it never builds the Rust build/pulp CLI (experimental/pulp-rs, pulled into the CMake graph only when PULP_ENABLE_GPU && top-level — which release-cli's GPU-enabled configure trips). So release-cli is the first VM workload that needs cargo, and on a fresh golden its "Ensure a working Rust toolchain on PATH" step exhausts every probe (PATH cargo, ~/.cargo/bin, ~/.rustup/toolchains/*) and fails the darwin release leg — while the bare-metal studios stay green because they have rustup from operator setup. That step now bootstraps rustup as a last resort (curl … | sh -s -- -y --profile minimal --default-toolchain stable --no-modify-path, then front-loads rustfmt+clippy per rust-toolchain.toml), so the lane is self-sufficient on any runner; the pin is stable, so no reproducibility drift vs the studios. Before flipping PULP_RELEASE_MACOS_RUNS_ON_JSON to a VM label, either that fallback must be in place or the golden must bake rustup — otherwise the darwin publish leg fails and the whole release never publishes (the release job needs the full build-cli matrix). Steady-state fix is to bake rustup into the golden (tartci manifests/pulp.macos.toml); the workflow fallback stays as the portability belt for any un-baked runner.

release-cli.yml's Intel (darwin-x64) leg is CROSS-COMPILED, not native

release-cli.yml's macOS matrix ships TWO slices: darwin-arm64 (routed through resolve-macos-runner) and darwin-x64, which cross-compiles on an Apple-Silicon runner via the macos-15-xcompile sentinel — routed by PULP_INTEL_RELEASE_MACOS_RUNS_ON_JSON (default ["macos-15"]), deliberately NOT through resolve-macos-runner. It builds with -DCMAKE_OSX_ARCHITECTURES=x86_64 plus -DPULP_RUST_CLI_TARGET=x86_64-apple-darwin, and smoke-tests the thin binary under Rosetta. It is a REQUIRED leg, the same reliability class as darwin-arm64, so Intel ships in every release.

The native macos-15-intel image is not used: it CPU-pegs, queues for hours, and never reliably shipped an artifact. Do not "restore" a native Intel leg, and do not route Intel to the self-hosted studios (the no Intel on the studios discipline in docs/guides/intel-support.md).

A release starves behind advisory lanes — check contention before blaming a build

Three separate capacity walls have each, on their own, stopped a tagged SDK from publishing while every platform binary built green. When a tag is stuck, work down this list before touching build code:

1. The macOS release VM is a capacity-1 pool. It takes a 6-core lease and the host budget fits exactly one, so release-cli's darwin-arm64 leg cannot run alongside anything else that wants that VM. Anything that waits on another workflow while holding it deadlocks the release outright.
2. release-path-pr-gate.yml routes its macOS leg at that same release VM (via PULP_RELEASE_MACOS_RUNS_ON_JSON). So every PR competes with actual releases for the one VM. A release job queued 90 minutes will keep losing to PR jobs queued 2 minutes — meanwhile the Studios sit idle. The tell is a darwin-arm64 leg stuck queued for hours with no failure.
3. GitHub-hosted macOS concurrency is effectively ~1 job at a time. The release's darwin-x64 and universal-arch-gate legs both land on hosted macos-15, where they queue behind advisory lanes — sanitizers (×4 per PR), coverage, sandbox-e2e, Android, Intel-portability, consumer smoke. None of those are required checks; the release is. It loses to all of them.

Diagnose, don't guess. tartci observe macos on the VM host shows what the runner is actually running (it prints the live Running job: line), and

ghapp api repos/danielraffel/pulp/actions/runners     # busy/idle per runner

distinguishes "saturated" from "wedged" from "starved by a lower-priority lane." An offline-but-busy=True runner is a dead VM still holding its slot.

Advisory cross-lane workflow: macos-cross-advisory.yml

.github/workflows/macos-cross-advisory.yml is a path-scoped advisory