ci

name	ci
description	Local and cloud CI for Pulp — validate branches, create PRs, merge on green. Handles "push to main", "ship this", "run CI", "check PR", and "list PRs".
requires	{"scripts":["tools/local-ci/local_ci.py"],"tools":["gh"]}

CI Skill

Validate branches and ship code safely. This skill handles all CI workflows for Pulp across local machines and VMs.

Runner timing metrics

When asked whether Pulp's local runners are fast, stuck, regressing, or worth monitoring, query Shipyard's metrics surface before guessing. Pulp does not mirror these records into pulp CLI or pulp-mcp; Shipyard is the metrics store and tartci is an optional VM runtime emitter.

This metrics surface requires a Shipyard build that includes the shipyard metrics subcommand. Pulp's current pinned source-checkout Shipyard in tools/shipyard.toml is v0.68.0, which does not include it yet. If a checkout only has the pinned binary, use a newer Shipyard binary or source checkout for the optional metrics workflow, or skip metrics and inspect live jobs directly.

Use these commands as the normal agent loop:

shipyard metrics import github --repo danielraffel/pulp --limit 50 --json
tartci runtime export --repo danielraffel/pulp --since-days 14 \
  | shipyard metrics import tartci --json
shipyard metrics summary --project pulp --json
shipyard metrics watch --project pulp --since 14d --json
shipyard metrics advise --project pulp --json

Read watch as a triage signal, not as a hard failure. insufficient_samples means the lane does not have enough history yet; drift, failure-rate, or slowest findings are the useful prompts to inspect runner logs, VM boot behavior, cache state, or GitHub queue time. If tartci is not installed for a repo, skip the tartci runtime export import and still use the GitHub import plus Shipyard summary/watch commands.

If a PR's required macos check has been queued >30 min or the repo's PRs are all in mergeable_state=blocked with no movement, jump to "Self-hosted runner ops" near the end of this file. One-shot recovery is shipyard rescue <PR> (Shipyard v0.53.0+). Continuous prevention is shipyard runner watch --kill-hung-workers (v0.54.0+). Keep Shipyard itself current with shipyard update (v0.55.0+; Pulp currently pins v0.56.2+). All three replace the legacy planning/scripts/runner-watchdog.sh --fix workflow, which is now an anti-pattern (cancels queued runs but registers failure on required checks).

Pre-flight: plugin ↔ CLI skew check

Before shelling out to pulp (or shipyard pr, which ultimately invokes pulp), source the shared skew-check helper so a user on an outdated CLI sees a one-line hint rather than running into obscure flag-missing errors mid-ship:

source "$(git rev-parse --show-toplevel)/tools/scripts/cli_version_check.sh"
pulp_cli_version_check

This is advisory only — the helper never blocks. See the upgrade skill for the full banner contract and override knobs (PULP_SKEW_CHECK_DISABLE, PULP_SKEW_CHECK_CACHE). Release-discovery Slice 6 (#551).

Diagnosing a slow / stuck PR — investigate before assuming runner saturation

When a PR sits without merging, don't ASSUME "the macOS CI pool is saturated" from a long-queued run — investigate. Saturation IS possible (a genuine burst, or a wedged/dead runner — see the pulp-runner-ops skill), so it's worth checking; just don't conclude it without evidence. In the 2026-06-18 case the cause turned out to be non-hardware (a misdiagnosis worth not repeating). Check in this order:

1. Did the required checks even register? A PR opened by the Shipyard GitHub App does NOT auto-trigger pull_request workflows, so the required macos and Enforce version & skill sync checks never appear on the PR head SHA until you dispatch them by hand:

   ghapp workflow run build.yml --ref <branch>             # posts the required `macos` check
   ghapp workflow run version-skill-check.yml --ref <branch>  # posts `Enforce version & skill sync`
   

   This is the most common reason a Shipyard PR "sits." (shipyard pr dispatches build.yml itself but you may still need version-skill-check.yml.) After a new push the head SHA changes — re-dispatch on the new SHA.
2. Is it a version-bump race? The other concurrent agent re-bumping main's CMakeLists.txt VERSION makes the PR DIRTY (conflict on the VERSION line). Merge origin/main in, re-resolve the VERSION to one above main, push, re-dispatch the checks. (Longer-term fix for this whole class: planning/2026-07-07-parallel-merge-land-coordination.md. In progress: the version-at-land.yml bot — currently DRY-RUN — will make a single writer assign the version from Version-Bump: intent trailers post-merge, removing the per-PR race. Until it flips to --apply, keep hand-bumping as today.) 2b. Did a gate reject the push? Besides skill-sync / version-bump, the pre-push + CI planning-gitlink gate (tools/scripts/planning_gitlink_guard.py) fails if the PR moved the planning submodule pointer without a Planning-Bump: trailer — usually an accidental bump from a git reset --hard
   • git add -A. Drop it with git restore --staged --worktree planning && git submodule update planning, or add Planning-Bump: reason="..." for a deliberate re-pin. 2b1. Config→doc drift? The config-doc gate (tools/scripts/config_doc_check.py, map in tools/scripts/config_doc_map.json) fails if a mapped CI/release config surface (.shipyard/config.toml, the shipyard pin/installer, the build/auto-release/version-skill-check/coverage workflows) changed without its guide doc (versioning.md / local-ci.md / release-watchdog.md). Editing a workflow under .github/workflows/ therefore usually needs a matching guide edit; a genuinely doc-irrelevant change bypasses with a Config-Doc: skip reason="..." trailer on any commit in the range. 2b2. Hotspot growth? The hotspot-size gate is now net-delta vs merge-base: it fails only if THIS PR grows a frozen hotspot past its reference size (main growing the same file is NOT your fault and passes). If you must grow one, make the change net-neutral (extract to a sibling file) or add Hotspot-Grow: <path> reason="..." — do NOT raise max_loc in hotspot_size_guard.json (that counter no longer gates growth, and raising it races with other PRs). 2b3. Hotspot shrink? The ceiling is a one-way ratchet, and this is the opposite instruction to 2b2. If your PR makes a frozen hotspot smaller, the gate fails with missing hotspot ceiling reduction(s) until you lower max_loc to the new LOC. Lowering is safe (it cannot race a concurrent PR into a false pass); raising is not. So: never raise it, always lower it when you shrink. A PR that both grows one hotspot and shrinks another needs a Hotspot-Grow: trailer and a max_loc reduction. 2c. Is a RED check even your fault? Before investigating a failing check, run python3 tools/scripts/pr_check_triage.py <PR#> — it labels each red check REQUIRED vs advisory and PRE-EXISTING (also red / not run on main — not your change) vs REGRESSED (green on main, red here). Advisory + pre-existing red (e.g. a known-broken sanitizer lane on main) does NOT block the merge and is not yours to fix; only a REQUIRED + REGRESSED row needs action. This alone avoids chasing main-side breakage.
3. Only THEN consider capacity — and verify, don't assume. The required macos gate runs on the local self-hosted Mac Studios (pulp-studio-01/02/03,
   • the M5 overflow), which are usually idle. Confirm with:

   ghapp api repos/danielraffel/pulp/actions/runners \
     | python3 -c "import sys,json;[print(r['name'],r['status'],'busy='+str(r['busy'])) for r in json.load(sys.stdin)['runners']]"
   

   If the Studios show busy=False, the pool is NOT saturated — say so. What DOES queue independently is the GitHub-hosted advisory lanes (Linux, Windows, sanitizers, coverage, android) on GitHub's shared pool; those are advisory, not the required gate, so a long queue there does not block merge.
4. Is shipyard run/ship failing at stage=configure with D external/skia-build/build? That's the Skia symlink loop (a tracked machine-specific absolute symlink autofetch deletes at configure → tree-drift), not capacity or a code failure — the local mac validation dies before posting its macos status, so the PR stays BLOCKED with the required check absent. Fixed at the repo level by PR #5588 (untracked + .gitignored), but a pre-#5588 checkout keeps the stale looped symlink until it pulls main. Recovery
   • full mechanics: the skia-gpu-build skill's Gotchas section (ln -sfn ~/.cache/pulp/skia-build/build <primary>/external/skia-build/build).

If you don't use Shipyard + the self-hosted Mac pool: steps 1 and 3 are specific to that setup (App-dispatched workflows; local runners) — skip them. The tool-agnostic rule still holds: distinguish the required checks from advisory lanes, and verify a runner is actually busy before blaming capacity.

Caveat to step 3 — idle Studios do NOT prove the required gate is unblocked. The required macos check is gated by a routing preamble (classify → resolve-provider) AND is itself an alias job that reports the leg's outcome — all three default to runs-on: ubuntu-latest. So when GitHub's shared ubuntu pool saturates (repo-wide actions/runs?status=queued in the hundreds during a PR burst), the preamble can't get a slot, the macOS leg is never dispatched, and the required gate sits pending with the Studios idle. Tell this apart from a real failure and from a wedged runner: queued preamble + online/busy=false Studios + high repo-wide queued count = GitHub-hosted starvation, not a code or hardware problem. Immediate unblock is admin-merge after local validation (ghapp pr merge <n> --merge --admin, --merge not --squash to preserve the bump marker); merging also drains that PR's ~11-workflow fan-out from the queue. The structural fix is the PULP_PREAMBLE_RUNS_ON_JSON repo var: set it to a self-hosted Linux selector (e.g. ["self-hosted","Linux","ARM64", "pulp-build-linux"]) to move those three jobs off the saturated pool — but only once that pool is confirmed always-on, or the gate just starves elsewhere. It defaults to ubuntu-latest (no-op) until set. A tartci launchd detector watches for this triad; full design in planning/2026-07-06-ci-queue-saturation-watchdog.md.

Host-vitals preflight — back off before a saturating CI host reboots

The self-hosted Mac Studio that runs the required macos gate ALSO hosts the interactive agent session and a heavy MCP stack (RepoPrompt, Figma, chrome-devtools, several pulp-mcp). When RAM fills, macOS jetsam starts killing processes, the window server crashes, and the host reboots uncleanly — taking any in-flight required-gate CI job down with it and reddening the leg for reasons that have nothing to do with the code (the 2026-07-01 incident). This whole class of failure is predictable ~20 minutes out from cheap metrics, so the agent, the CI pool, and Shipyard should all read one shared signal and back off.

tools/scripts/host_vitals.sh is that signal. It reports green / warn / critical (exit 0 / 10 / 20) keyed on memory pressure first (kern.memorystatus_vm_pressure_level + fresh JetsamEvent-* reports), with load only ever corroborating a warn — a healthy parallel build (load 1–2× cores, normal pressure) never trips it. Use it before heavy work:

tools/scripts/host_vitals.sh            # one-line summary; exit code = level
tools/scripts/host_vitals.sh --json     # machine-readable

• Agent admission control: before launching a local build or a heavy MCP call on a CI-host session, check host_vitals.sh. If critical, don't pile on — ship via shipyard pr / GitHub-native auto-merge (survives a restart) instead of a foreground shipyard/ci watch, and shed idle load (close RepoPrompt/Figma /idle MCP) before building. gates.sh prints this banner advisorily on every pre-push.
• A "hung"/"stuck" git push is almost always the pre-push diff-cover BUILD, not the network. When the diff touches a coverage surface (core/, tools/cli/, tools/scripts/), .githooks/pre-push runs a full local configure + compile before the transfer. It prints a loud DIFF-COVERAGE BUILD running banner and a …still running (Ns elapsed) heartbeat every 30s so this is unmistakable — if you see those, it is a BUILD (minutes), NOT a transport hang. Tell: git fetch/ls-remote succeed while the push "hangs", so SSH/HTTPS are fine. Do NOT diagnose the network, and do NOT background-and-kill the push at a short timeout (that hides the banner/heartbeat). When the work is already validated (e.g. via shipyard pr), push with PULP_SKIP_DIFF_COVER=1 git push and it completes in seconds. (Learned 2026-07-07 after ~an hour lost mis-blaming MTU/SSH for a diff-cover build during a deps-bump push.)
• Continuous sensor: tools/scripts/install_host_vitals_sensor.sh installs a per-user launchd agent (com.pulp.host-vitals, 60 s) that publishes the latest reading to ~/.local/state/pulp/host_vitals.json and a rotating host_vitals.log. It is observation-only — it never stops a runner or kills a process — so it is safe to run on the required-gate host. Installed on the m3/m5 /m1 pool. install_host_vitals_sensor.sh --status shows the launchd + latest reading; --uninstall removes the agent.
• A whole-pool-fails-at-once red leg is infra, not code (per macos-required-leg-timeout-saturation): if windows + both macos legs fail together and the diff can't explain it, correlate against host reboot / jetsam time and re-run the required leg rather than debugging the change. Active back-off in the CI pool (tartci auto-yield) and Shipyard (host-health dispatch gate + infra-vs-code classification) consume this same host_vitals state.

GitHub workflow gotchas

• Every on: schedule workflow must carry the fork guard. When someone forks the repo, GitHub copies all workflows and runs the scheduled ones on the fork's default branch — then emails the fork owner whenever one fails, which our monitors reliably do on a fork (they probe this repo's state or use secrets the fork lacks). So each entry job (a job with no needs: — dependents cascade-skip) of a scheduled workflow gets:

  jobs:
    check:
      if: github.event_name != 'schedule' || github.repository == 'danielraffel/pulp'
  

  compose it with an existing condition as if: (github.event_name != 'schedule' || github.repository == 'danielraffel/pulp') && (<existing>). It only suppresses the schedule event on forks — push / pull_request / workflow_dispatch are untouched, so PRs to this repo (which run in this repo's context) and manual dispatches behave exactly as before. A workflow that should run on forks' schedules opts out with a top-of-file # fork-guard-exempt: <reason> comment. This is enforced: tools/scripts/scheduled_workflow_fork_guard_check.py runs in gates.sh, the pre-push hook, and workflow-lint.yml, so a new scheduled workflow missing the guard fails the PR. Add the guard when you add the workflow.

• Codecov "total lines/files dropped" is usually upload starvation, not config drift. Two distinct guard layers exist and they catch different things: test_codecov_components.py / test_codecov_config.py (PR-gate) guard the codecov.yml mapping (a subsystem matching no component → invisible); coverage-upload-watchdog.yml (hourly, main) guards the outcome — "main had a successful Coverage run in the last N hours." The watchdog exists because the dashboard degrades silently when fresh complete uploads stop arriving, from causes the config gate can't see: coverage runs cancelled by stale-run-reaper.yml when a CI dispatch throttle makes them queue past the cutoff (the 2026-06-28 incident — 8 consecutive main Coverage runs cancelled), an llvm-cov -object-set regression (a libpulp-*.a drops out → a whole subsystem vanishes while lines-valid stays > 0, which verify_cobertura_xml.py's lines-valid > 0 check cannot catch), or Codecov's after_n_builds waiting forever on a missing per-OS leg. When triaging a coverage-surface complaint: first check gh run list --workflow coverage.yml --branch main for recent successful runs (cancelled ≠ uploaded); only then suspect codecov.yml. Note the config gate is advisory (not in branch protection), so a fleet-auto-merged PR can land config drift despite a red gate — the watchdog is the main-branch backstop.

• web-plugins.yml is the headless-browser web lane (advisory). It builds Pulp's WAMv2 (Emscripten) and WebCLAP (wasi-sdk) web plugin formats on a Linux GitHub-hosted runner and runs every web validation, including the browser fixtures in headless Chrome (browser-actions/setup-chrome + playwright-core, which drives the system Chrome — no browser download). It is deliberately NOT on the self-hosted VMs: a headless-Linux browser lane needs emsdk + wasi-sdk + Chrome, all of which install cleanly on GitHub-hosted, so no golden VM or QEMU work is warranted. The WAM build vendors choc by cloning the pinned fork (PulpWam.cmake needs PULP_WAM_CHOC_INCLUDE; the WebCLAP build FetchContents choc/clap itself). The browser drivers are examples/web-demos/*/{browser-test,browser-host}/validate.mjs; run them locally with a system Chrome/Canary via node validate.mjs --screenshot out.png (set CHROME_PATH or pass --browser). The WAM node validations also cover the in-worklet rack path (wam_rack_runner.mjs against the PulpPluckGainRack target — the only committed consumer that compiles wam_chain_entry.cpp/WamChainBridge), the wire protocol + version-skew bridge contract (core/format/src/wasm/wam-runtime.test.mjs, pure JS), and hostile-state rejection (overflow/CRC/truncation) folded into wam_feature_runner.mjs — so a regression in the state codec or chain runtime fails here rather than only in a browser.

• wclap-cloudflare.yml is WebCLAP's canonical-host lane (Cloudflare Pages). Separate from web-plugins.yml: it builds the threaded WebCLAP PulpGain module (pinned wasi-sdk), assembles a self-contained deploy dir, proves it headlessly under the real _headers, then owner-gated-deploys. Two pages are assembled + proven: the single-plugin isolation page (assemble.mjs → validate-deploy.mjs) and the shared-player WebCLAP demo (assemble-player.mjs → player/ → validate-player.mjs) — the latter mounts the SAME @danielraffel/web-player shell the WAM demos use, driven by the WebCLAP adapter (a worklet-resident CLAP host), and asserts crossOriginIsolated, real-time render (worklet quanta ≈ sampleRate/128 per wall-second), an audible param change that updates the shared widget, and a clap.state round-trip. The lane also runs the @danielraffel/web-player adapter unit tests (packages/pulp-web-player — npm test), which include an ABI-parity check that the worklet's inlined CLAP struct offsets match wclap-abi.mjs. Hard-won gotchas encoded there: posting a WebAssembly.Module INTO an AudioWorklet is silently dropped in Chrome (transfer the raw bytes and compile in-worklet), and transferring an ArrayBuffer OUT of an AudioWorklet is unreliable (the receiver gets a detached buffer) — clone small payloads (state, sysex) instead, and never transfer a caller-owned buffer (it detaches theirs).

• screenshot-sync is a three-layer gate that mirrors skill-sync. A repo opts in by committing a .pulp/screenshots.toml manifest (presence == opt-in); tools/scripts/screenshot_sync_check.py then diffs the manifest's [trigger].paths and fails when a triggered target's committed PNG/OG image wasn't refreshed. Wired identically to skill-sync: PostToolUse hint in hooks/scripts/cli-plugin-sync.sh, pre-push report in .githooks/pre-push, and authoritative CI step in version-skill-check.yml. Bypass a single commit with a Screenshot-Sync: skip target=<id|all> reason="..." trailer. Pulp core itself is NOT opted in (no manifest), so the gate no-ops here; it exists for downstream plugin repos (GPU NAM, example plugins, Bendr) and the WCLAP/WAM OG images.

• Android native .cxx caches must be dependency-aware. The Android workflow builds through Gradle's external native build, and android/app/.cxx can hold FetchContent checkouts under _deps. Do not cache .cxx under a Gradle-only key or a broad Gradle restore key: after a native dependency bump, CMake's UPDATE_DISCONNECTED path can see the stale local checkout and fail because the new git ref is not present. Keep the Gradle cache (~/.gradle) separate from the native CMake cache, key .cxx on the native dependency inputs (CMakeLists.txt, tools/cmake/PulpAndroid.cmake, tools/cmake/PulpDependencies.cmake, tools/deps/manifest.json, plus Android Gradle files), and do not give .cxx a restore key that ignores those inputs.

• intent-bump-on-merge.yml is the merge-time half of the version-bump intent-trailer model — and it ships DORMANT. It exists to kill the version-bump merge treadmill (PRs editing CMakeLists VERSION / plugin.json / marketplace.json re-conflict every time main advances). The endgame: a PR declares Version-Bump: <surface>=<level> and touches NO version files, and this workflow assigns the exact number after merge from main's current version via tools/scripts/apply_intent_bump.py. Phase 1 (current): no-op. Nothing emits intent trailers yet (Shipyard still file- bumps on the PR side; version-skill-check.yml still runs WITHOUT --accept-intent-trailers), so every run finds no trailer and exits clean. Two things must be verified before the phase-2 flip (a separate, reviewed change): (1) RELEASE_BOT_TOKEN can push a commit to protected main (it already pushes tags from auto-release.yml; a commit needs the bot on the branch-protection bypass list), and (2) the Version-Bump: trailer survives squash-merge into main's commit message. The workflow has a recursion guard (skips its own chore: bump versions commit) and a concurrency group so near-simultaneous merges bump the version line one at a time. The chore: bump versions commit it pushes triggers auto-release.yml exactly like a PR-side bump.

• test/CMakeLists.txt is now an include hub, not a registration manifest. Add new add_test, add_executable, and pulp_add_test_suite blocks to the matching test/cmake/*_tests.cmake manifest instead of rebuilding the old monolith. If no existing manifest owns the subsystem, create a focused new test/cmake/<area>_tests.cmake file and include it from test/CMakeLists.txt in dependency order. Focused owner hubs may include smaller manifests for their own subsystem, but do not hide sibling manifests inside an unrelated owner just to keep the top-level file short. Keep the top-level hotspot_size_guard.json ceiling at the exact tiny hub LOC; raising it for ordinary test additions is a regression. Before burning a PR CI job on a manifest-only cleanup, run the local preflight: python3 tools/scripts/docs_noise_lint.py --mode=report --base origin/main, python3 tools/scripts/hotspot_size_guard.py --base origin/main --config tools/scripts/hotspot_size_guard.json --mode=report --require-ceiling-reduction, and a clean CMake configure/target-list compare against origin/main.

• Source hotspots (e.g. core/view/src/design_cpp_codegen.cpp) are frozen too — bump the ceiling for a coherent feature, split when it's accretion. hotspot_size_guard.json also freezes large source files. When a single, cohesive feature legitimately grows one (e.g. teaching the C++ codegen to emit faithful_svg as a DesignFrameView), raise that file's max_loc in the same PR — splitting a tightly-coupled emitter mid-feature would hurt readability more than the extra LOC. Reserve the split for genuine grab-bag growth. When a PR shrinks a tracked hotspot, lower that file's max_loc in hotspot_size_guard.json to the new exact LOC in the same PR; the --require-ceiling-reduction gate compares the merge-base blob against HEAD for branch-touched tracked hotspots, so leaving headroom after a shrink fails the pre-push/CI guard. Exception — a hotspot already ABOVE its ceiling. check_hotspots only fails the PR that itself grew a file, so main can drift past a max_loc while every individual PR stays net-neutral (e.g. window_host_mac.mm at 2780 LOC against a 2770 ceiling). Shrinking such a file cannot lower the ceiling — setting max_loc to the new LOC would raise it — so the guard does not ask for a reduction there, and prints still at/over ceiling; no reduction to make. The ratchet is demanded only when the shrink lands strictly under the existing ceiling. If you see the guard demand a max_loc larger than the one in the config, that is the bug this rule fixed, not a config you should edit. Editing hotspot_size_guard.json itself trips the ci skill-sync gate; the ceiling bump is normally part of a non-ci feature PR, so either fold this note's rationale into that PR or skip the ci gate with a Skill-Update: skip skill=ci reason="ceiling bump only" trailer on the tip commit (note: a later chore: bump versions commit from shipyard pr displaces the tip, so updating this SKILL is the more robust path).

• Inspector hotspots are frozen too. hotspot_size_guard.json watches newly added inspect/** files and freezes the current inspector overlay, window, domain handler, and tweak-store hotspots. When an inspector extraction shrinks a tracked file, lower its max_loc to the exact new line count in the same change. When a new overlay companion file triggers the large-file warning, split it before it becomes another hotspot.

• Reskinnability ratchet (token-coverage-ratchet ctest). Driven by tools/scripts/token_coverage_check.py: fails if a core/view/src paint file gains a NEW colour literal that is not a resolve_color(...) fallback. Mark a deliberate material-effect literal // token-lint:allow; after intentionally lowering a file's count, re-freeze with --update-baseline.

• SignalGraph single-backend governance (single_backend_guard.py + processing_model_terms_lint.py). Whole-tree structural guards (not diff-scoped) that keep the DSP runtime convergence from regrowing a second surface: exactly one type (GraphRuntimeExecutor) may define process_routed / process_parallel; pulp create offers no graph-plugin authoring scaffold; the public generated-DSP ABI entry symbols stay the sanctioned pair (pulp_native_core_entry_v1, pulp_node_v1_entry); and the differential parity test stays registered as a built target. They run in gates.sh, the pre-push hook, and the Versioning & Skill-Sync workflow (hard-fail). To sanction a genuinely-new routing engine, authoring template, or generated-DSP ABI, widen the allowlist in single_backend_guard.py in the same PR — that diff is the architecture-review record. Both guards carry a --selftest run in the workflow's fixture-test step. Hardening notes (from the series adversarial review): a non-sanctioned Rival::process_routed is flagged even when defined inside graph_runtime_executor.{hpp,cpp} (no TU-filename escape), and the parity-registration check requires the test source to appear on a real SOURCES/add_executable/target_sources/pulp_add_test_suite line — a bare mention in a dead variable no longer counts.

• Conflict-marker guard (conflict_marker_check.py). Whole-tree guard (not diff-scoped): no tracked file may carry a git conflict marker. Born from the incident where a squash-merge's stale side collided with an already-advanced project() VERSION line and wrote <<<<<<< / ======= / >>>>>>> straight into CMakeLists.txt (fixed in #5477), breaking every build until a human noticed. Keyed on the start/base/end markers (<<<<<<< / ||||||| / >>>>>>> at column 0, followed by whitespace/EOL) — verified zero-false-positive across the whole tracked tree, external/ included; the ======= separator is reported only inside a real conflict block, so Markdown headings and ASCII banners stay clean. Runs in three layers: gates.sh + the pre-push hook (local), the Versioning & Skill-Sync workflow's Conflict-marker guard step (which scans the pull_request MERGE ref, so a marker born from the merge itself is caught, not only one on the PR head), and conflict-marker-guard.yml — a push:main backstop that reddens the branch and opens a tracking issue if a marker reaches main by any path (the squash case, where GitHub had no clean mergeable ref for the PR job to inspect). Carries a --selftest in the fixture-test step. A vendored fixture that legitimately ships markers is a reviewable ALLOWLIST edit in the script. If the push:main guard reddens main, run python3 tools/scripts/conflict_marker_check.py for the file:line list, resolve, and push — the tracker auto-closes on the next clean push. Exit codes are meaningful: 0 clean, 1 markers found (the backstop opens the tracker), 2+ internal error (the backstop fails the run without opening a wrong "markers committed" issue) — the selftest locks this contract. Documented scope limitations (deliberate, to keep zero false positives): only default seven-char markers (a non-default .gitattributes conflict-marker-size is missed), submodule contents are out of scope (the superproject sees a gitlink), and NUL-bearing/UTF-16 text is skipped as binary. The upstream fix for the whole class — the merge tool refusing to commit a conflicted result — is tracked in Shipyard #372; this guard is the consumer-side backstop.

• Thread-safe-assertions guard (thread_assert_check.py). Catch2 3.7.1's assertion macros are NOT thread-safe (thread-safe assertions are opt-in only from Catch2 3.9.0), so a REQUIRE/CHECK/FAIL/SUCCEED inside a std::thread / std::jthread / std::async lambda in test/ is undefined behavior: it corrupts Catch's per-run assertion counters + section state. Bare metal tolerated it, but the move of the required macOS gate onto Tart VMs made the different scheduler timing trip it — a HotSwapSlot hammer test "failed" in 0.00s though the code under test was race-free by design. The correct pattern: record the violation into an atomic/guarded value in the worker, join(), then assert on the test thread (REQUIRE_FALSE(bad.load())). The lint is a lexical scan (best-effort: it does not follow calls into helpers) with a same-length string/comment-blanking pass that also skips C++ digit separators (10'000); suppress a verified-safe line with a trailing // thread-assert:allow. Runs as the thread-safe-assertions ctest case and in gates.sh. When graduating any required lane to VMs, expect this class of latent UB to surface — fix at the source, don't suppress.

• Release builds must pass -DPULP_BUILD_EXAMPLES=OFF. The pulp-design-tool example hard-fails CMake configure when PULP_HAS_SKIA is FALSE (belt-and-suspenders, code 78). sign-and-release.yml builds on a GitHub-hosted macOS runner with no Skia, so configuring the full tree (examples included) aborts the entire run → no GitHub Release publishes even when release-cli.yml (the CLI build) is green. This silently broke Release publication (the release watchdog flagged the missing Releases). The release ships the CLI + plugins (build/VST3,/CLAP,/AU), never the example apps, so the release legs build with -DPULP_BUILD_EXAMPLES=OFF (matching build.yml). If you add a new release/packaging workflow, configure with examples OFF (or a populated SKIA_DIR), or the design-tool Skia gate will block it.

• Release/SDK builds must pass -DPULP_ENABLE_AUDIO_PROBES=OFF. The standalone audio probe is a dev/example inspection surface and defaults ON for local development, but shipped CLI, standalone, and SDK artifacts must not export it. Keep release-cli.yml, release-path-pr-gate.yml, release-dry-run.yml, sign-and-release.yml, release-cli-local.sh, and checkout-backed SDK configure paths aligned when touching release CMake flags.

• sign-and-release.yml waits on release-cli.yml to create the release — the wait must cover release-cli's FULL runtime. Since the release split (release-cli is the sole creator of the GitHub release; sign-and-release only attaches appcast.xml on top), sign-and-release polls gh release view "$TAG" until the release exists. release-cli creates it only after its whole chain (build-cli → smoke-cli → universal-arch-gate), which includes a self-hosted darwin-arm64 leg that can queue behind a busy runner — routinely far longer than 10 minutes. A too-short poll loses that race on every tag and leaves the published release without its Sparkle feed (the watchdog flags it as a broken release). Keep the wait window generous (currently 150 * 20s = 50 min); if release-cli grows a slower leg, widen it rather than letting the appcast attach silently time out. The SDK/CLI binaries are unaffected either way — they are release-cli's output; only the appcast rides on sign-and-release.

• Hooks inherit GIT_DIR — tests that shell out to git can corrupt the live worktree. Git exports GIT_DIR/GIT_WORK_TREE into hook environments, and a set GIT_DIR overrides git -C <dir> discovery. So when the pre-push hook (or shipyard) runs the full ctest and a test does git -C <tempdir> init/commit/checkout, those commands silently hit THIS worktree's repo instead — stray initial commits, throwaway branches, and a core.bare=true flip in the shared config (which makes the main checkout look "not a work tree"). .githooks/pre-push and tools/scripts/local_diff_cover.sh now unset GIT_DIR GIT_WORK_TREE … before running gates/ctest, and the git-shelling tests scrub the same vars in-process (see the regression test in test_mcp_server.cpp). If you add a test that shells out to git on a temp repo, clear the inherited git env first (or run from a context with no GIT_DIR), and never assume -C alone isolates it. Recovery if a worktree was hit: git config core.bare false, reset the branch off the stray initial commit, delete the throwaway branch.

• The required macos alias in .github/workflows/build.yml mirrors the macOS matrix leg by polling the Actions jobs API. Keep that poll retry-safe: API failures or malformed JSON must log and continue the loop, not trip set -euo pipefail before the macOS leg has a chance to report.

• .github/workflows/release-dry-run.yml (P9-2, #2576) exercises the release build → package_cli.py → pulp ship appcast chain on a synthetic version (0.0.0-dryrun) WITHOUT publishing — no GitHub release, no signing/notarize, no appcast upload; artifacts are throwaway. It's additive (does not touch release-cli.yml / sign-and-release.yml) and runs weekly + on demand, so a build/packaging/appcast-generation regression surfaces BEFORE a real tag. Keep it credential-free (notarize/sign stay in the real path) so it can run on a schedule without secrets.

• Release-bot source refs must be SSH-signed. auto-release.yml creates signed annotated v* tags with git tag -s, and the bot commit workflows (intent-bump-on-merge.yml, post-tag-sync.yml) configure the same SSH signing helper before committing. The required Actions secret is RELEASE_BOT_SSH_SIGNING_KEY; it is a file-backed OpenSSH private key backed up outside the repo. The workflow uses [e-mail] because GitHub verifies SSH signatures against the account that owns the uploaded signing key. If this secret is absent, the signing setup must fail closed rather than create unsigned release tags or bot commits.

• Release-runner Xcode must be pinned (C++20 parity). sign-and-release.yml runs on GitHub-hosted macos-14, whose DEFAULT Xcode is 15.4 — its Apple clang lacks C++20 P0960 (parenthesized aggregate init, Type p(arg) for a ctor-less aggregate). The self-hosted PR macos lane uses a much newer clang that accepts it, so a CLI/import TU compiled on every PR but FAILED only in the release build — silently breaking GitHub Releases v0.372–v0.391 (tags kept being cut; only the release-cadence watchdog noticed). The job now selects the newest installed Xcode 16.x via xcode-select (shell, no third-party action so an actions-allowlist can't hold the release hostage), restoring C++20 parity with the PR lane. When a release/packaging workflow builds C++ on a GitHub- hosted macOS runner, pin a modern Xcode — the default lags and silently diverges from the PR toolchain. (Code-side defense: always brace aggregate init Type p{arg} in CLI/import code — see the import-design skill.)

• The release build is NOT a test gate. sign-and-release.yml no longer re-runs the unit suite (ctest). By the time a commit is tagged it has already passed the FULL suite on the PR/merge gate (self-hosted lane, real GPU/display/iOS-SDK). Re-running on the HEADLESS GitHub-hosted release runner is redundant and yields false failures from environment-only tests (Skia-raster screenshot → empty, cmake-require-gpu → timeout, cmake-ios-hostapp-links) that pass on real hardware — that blocked Releases AFTER the Xcode-pin let the build through. Principle: tests gate at PR on representative hardware; the release builds + signs + notarizes + packages the validated commit (the Build step is the release-config compile smoke; validate.yml gates format validators). The replacement gate is a built-ARTIFACT smoke (the "Smoke built plugins" step): it nm-reads each built build/CLAP/*.clap and FAILS only if a Mach-O was produced without its clap_entry C-ABI export (a real linkage regression), warning-and-passing when no artifact is found (a path/setup miss must never re-block the release). Static symbol read = NO execution, so it's headless-safe — never use dlopen+init (loads GPU/Skia libs the headless runner lacks) or re-run the hardware-dependent ctest suite.

• Sandbox E2E macOS has a long cold C++ CLI build. .github/workflows/sandbox-e2e.yml builds the pulp-cli target before running the Python sandbox harness. On GitHub-hosted macos-latest, a cold C++ CLI build can exceed 30 minutes and be reported as cancelled with The operation was canceled plus orphaned clang processes, before pytest starts. Treat that as a job timeout/build-cost issue, not a sandbox assertion failure. The workflow uses timeout-minutes: 60 so the cold macOS build has room to finish while still bounding genuinely wedged runs.

• .github/workflows/header-self-contained.yml (pulp #2576) is a BLOCKING gate for the "compiles on Apple Clang, breaks on Linux" transitive-include class (e.g. uint32_t without #include <cstdint> — broke the v0.197.4 release). It compiles each PR-changed public header standalone with Linux Clang via tools/scripts/check_headers_selfcontained.py. Unlike the advisory IWYU gate it only fails on a header that genuinely won't compile alone (no "unused include" false positives), so it is safe to block on. Headers whose module isn't in the GPU-off compile DB are skipped, not failed.

• Windows FetchContent subbuilds need a short base dir. GitHub-hosted Windows runners can still route MSBuild metadata through the legacy 260-character path limit. The wgpu prebuilt-populate subbuild has deeply nested stamp paths, so build-windows/_deps/... can exceed MAX_PATH during configure even before compilation. build.yml passes -DFETCHCONTENT_BASE_DIR="$PWD/fc" only on Windows to keep dependency subbuilds short while preserving the normal build-windows artifact/test directory.

• .github/workflows/watchdog-reaper.yml (pulp #2576) sweeps ALL open release watchdog trackers daily and closes any whose version is released or superseded — the existing watchdogs only auto-close inside a recent window, so historical per-version trackers orphaned (334 had accumulated). It only matches the exact auto-generated tracker titles and only closes objectively-resolved ones. SHA-keyed release: stuck trackers carry no version in their title; the reaper skips them without failing the sweep. Close those only after verifying the affected SDK/plugin surface shipped the stranded commit.

• Keep watchdog/issue-maintenance workflows on REST gh api calls. Avoid gh issue list / gh pr * helpers in those paths because they can use the shared GraphQL quota; a watchdog must not fail while reporting that the watched workflow already recovered.

• Pulp's docs site is deployed by .github/workflows/docs-deploy.yml. If GitHub Pages is set to legacy main/docs builds, the generated Pages checkout tries to clone the private planning submodule with the default token and fails before docs deploys. Fix that at the Pages configuration layer (build_type=workflow), not by weakening normal submodule checkout.

• .github/workflows/sanitizers.yml runs broad ASan/UBSan matrices, but its exclude regex may carry narrow sanitizer-lane skips for non-memory-safety failures that remain covered by regular Build/Test. Keep those skips named exactly to the failing CTest cases and leave comments explaining the alternate coverage path; do not use sanitizer excludes to hide new targeted coverage tests.

• Flaky-lane retry (de-flaking). The ASan/UBSan/TSan lanes (sanitizers.yml) and the coverage lanes (scripts/run_coverage.sh) run ctest with --repeat until-pass:2. Timing-sensitive tests intermittently fail under sanitizer slowdown (a different test each run — RenderLoop coalescing, ImageView fill, etc.), and a single failure aborts ctest and cascades into the diff-coverage gate (partial profile → 0% on the diff), leaving every PR UNSTABLE even when the required gates pass. until-pass:2 retries a failed test once so a flake passes on retry while a genuine failure still fails both attempts (no masking). This complements the per-lane --exclude-regex flake lists by catching not-yet-listed flakes generically — prefer it over growing the exclude list for transient timing flakes.

Current Build-and-Test routing

As of the 2026-05-20 classify gate, build.yml runs a cheap classify job before allocating the native matrix. PRs that touch only skip-safe docs/planning paths report the native aliases green without running macOS/Linux/Windows builds; PRs that touch C++/Swift/CMake or workflow inputs must run the native matrix. A code PR whose native build is skipped is a CI bug.

workflow_dispatch defaults runner_provider to github-hosted, not Namespace; do not dispatch with runner_provider=namespace. Linux/Windows use GitHub-hosted runners by default. macOS routes through the self-hosted pulp-build pool (pulp-m1-01, pulp-m1-02) via PULP_LOCAL_MACOS_RUNS_ON_JSON; repository variables control any overflow.

Do not push empty commits just to churn queued macOS checks. Cancel superseded SHAs, rebase or push only when a PR needs current main, and wait unless a check has actually failed.

coverage.yml's macOS leg resolves its runs-on via resolve_runs_on.py --deny-labels pulp-build,pulp-build-vm: a coverage selector (repo var PULP_COVERAGE_MACOS_RUNS_ON_JSON or a dispatch input) that names the shared gate pool fails the resolver fast rather than letting a long advisory coverage run contend with the required macos check. The dedicated coverage lane uses pulp-coverage-vm-macos; a bare GitHub-hosted label (macos-15) is never denied. (Push-triggered coverage on a busy main is designed to be superseded while queued — the supersession-immune scheduled run, cron 17 */8 * * *, is the one that produces the green full-matrix upload that clears the coverage-stale watchdog.)

The os-windows coverage leg is best-effort. The instrumented MSVC build + ~9k instrumented tests + llvm-cov over 1000+ objects exceeds the 150-min job cap on GitHub-hosted windows-latest (it is ~1h on Linux/macOS), and the staleness watchdog keys off a successful run, not per-OS Codecov flags — so a red Windows leg would otherwise keep a healthy full run red forever. The subtle trap (verified by canary): job-level continue-on-error does NOT neutralize a timeout-minutes cancellation — a cancelled job still makes the run conclude cancelled. It DOES neutralize a normal job failure. So the coverage suite step self-terminates at an internal budget (135 min) below the job cap, turning the would-be cancellation into a normal non-zero exit that the job-level continue-on-error: matrix.os=='windows' then absorbs → the run concludes success. Don't "simplify" this to bare continue-on-error; it will silently stop closing the watchdog. And the watchdog that enforces the budget must separate its steps with ;, NOT && — if the kill is &&-gated behind a : > marker write (which can fail on a Windows RUNNER_TEMP backslash path), a failed marker write short-circuits the chain and the suite is never killed, so the job hits the 150-min cap and is cancelled anyway. The kill is mandatory; the marker is best-effort (cleanup of any partial Cobertura also triggers on a 143/137 signal-kill exit, not just the marker). Real os-windows correctness bugs are still worth fixing (the ARG_MAX response-file + vanished--object existence-filter + mass-drop guard in run_coverage.sh were real); only the runtime/timeout is accepted as best-effort.

release-cli.yml needs Rust; the VM golden does not have it

The macOS build gate and release-cli.yml's darwin leg can share the Tart VM golden (pulp-build-runner), but that golden is provisioned only for the C++ "Build and Test" lane — Xcode, homebrew cmake/ninja/node, baked Skia, and no Rust toolchain (manifests/pulp.macos.toml [brew] has no rust/rustup). build.yml stays green on it because it never builds the Rust build/pulp CLI (experimental/pulp-rs, pulled into the CMake graph only when PULP_ENABLE_GPU && top-level — which release-cli's GPU-enabled configure trips). So release-cli is the first VM workload that needs cargo, and on a fresh golden its "Ensure a working Rust toolchain on PATH" step exhausts every probe (PATH cargo, ~/.cargo/bin, ~/.rustup/toolchains/*) and fails the darwin release leg — while the bare-metal studios stay green because they have rustup from operator setup. That step now bootstraps rustup as a last resort (curl … | sh -s -- -y --profile minimal --default-toolchain stable --no-modify-path, then front-loads rustfmt+clippy per rust-toolchain.toml), so the lane is self-sufficient on any runner; the pin is stable, so no reproducibility drift vs the studios. Before flipping PULP_RELEASE_MACOS_RUNS_ON_JSON to a VM label, either that fallback must be in place or the golden must bake rustup — otherwise the darwin publish leg fails and the whole release never publishes (the release job needs the full build-cli matrix). Steady-state fix is to bake rustup into the golden (tartci manifests/pulp.macos.toml); the workflow fallback stays as the portability belt for any un-baked runner.

release-cli.yml has a native Intel (darwin-x64) leg

release-cli.yml's macOS matrix ships TWO slices: darwin-arm64 (routed through resolve-macos-runner) and darwin-x64 on GitHub-hosted macos-15-intel. The Intel leg deliberately does not go through the resolver — the runs-on ternary only routes matrix.os == 'macos-15', so macos-15-intel runs on that fixed hosted label (no self-hosted/Namespace/VM routing, and Rust is preinstalled so the toolchain-bootstrap above is a no-op there). It's a native build (no cross-compile, no Rosetta) with a CMAKE_OSX_DEPLOYMENT_TARGET=13.0 floor and an arch-asserting smoke. macos-15-intel is the pinned stable hosted x86_64 macOS image (supported ~through Aug 2027; macos-13 retired Dec 2025; macos-26-intel also exists but is newer/less-baked) — when it EOLs, the successor is cross-compiling on the arm64 pool (planning/2026-07-10-intel-mac-cli-support.md). Do NOT route this leg to the self-hosted studios (the no Intel on the studios discipline in docs/guides/intel-support.md).

Advisory cross-lane workflow: macos-cross-advisory.yml

.github/workflows/macos-cross-advisory.yml is a path-scoped advisory job for the Linux-hosted macOS arm64 cross lane (Phase 5 scaffolding, see planning/2026-05-24-linux-hosted-macos-arm64-cross-lane.md). It runs on ubuntu-latest, does not bootstrap osxcross, and does not download a macOS SDK — it only confirms the Pulp-side cross scaffolding (tools/cmake/toolchains/macos-arm64-osxcross.cmake, tools/scripts/verify_macos_cross_artifacts.py, the PULP_RUST_CLI_TARGET / PULP_MACOS_CROSS_ALLOW_MISSING_ICON_TOOLS / OTOOL / INSTALL_NAME_TOOL hooks) stays wired and that the verifier unit tests still pass. It is non-gating by design; do not promote it to a required check until a self-hosted Linux runner with pinned osxcross

• private SDK is provisioned and the matching full-build job lands.

Advisory compile-gate: windows-midi2-gate

build.yml's windows-midi2-gate job (continue-on-error: true, NOT a required check, NOT part of the build matrix) compile-verifies Pulp's opt-in WinRT MIDI 2.0 backend (core/midi/platform/win/winrt_midi_device.cpp, gated by PULP_HAS_WINRT_MIDI). That backend consumes the Microsoft.Windows.Devices.Midi2 C++/WinRT projection, which ships out-of-band with the Windows MIDI Services SDK — a GitHub-only NuGet, NOT in the base Windows SDK, so no other lane can compile it. The job provisions it through Microsoft's official vcpkg port microsoft-windows-devices-midi2 (it downloads the SDK NuGet, runs cppwinrt to generate the projection headers, and exports the Microsoft::Windows::Devices::Midi2 CMake target). Pins + rationale live in tools/ci/midi2/ (vcpkg.json + README.md). The default Windows build never sets PULP_HAS_WINRT_MIDI, so this is purely additive. Watch points: the port requires Windows SDK >= 10.0.26100.0 (windows-latest is right at that floor), and the drafted backend's API surface may still drift from the real winrt::Microsoft::Windows::Devices::Midi2 namespace.

Advisory compile-gate: windows-ble-gate

build.yml's windows-ble-gate job (continue-on-error: true, NOT a required check, NOT part of the build matrix) compile-verifies Pulp's WinRT Bluetooth-LE scan backend (core/midi/platform/win/ble_midi_win.cpp). Unlike windows-midi2-gate, the BLE GATT / advertisement APIs (Windows.Devices.Bluetooth*) ship in the BASE Windows SDK cppwinrt projection, so this gate needs NO vcpkg / out-of-band NuGet provisioning — the default Windows build already compiles the backend (it links WindowsApp + runtimeobject). The job is a fast, isolated cmake --build … --target pulp-midi so a blind (macOS-authored) Windows TU gets a compile signal without waiting on the full matrix. Watch point: the backend's winrt::Windows::Devices::Bluetooth::Advertisement API surface is written without a local Windows compiler, so signature drift surfaces here first.

Advisory build-gate: tracing-build.yml

.github/workflows/tracing-build.yml (advisory, ubuntu-latest, NOT a required check) is the only lane that builds the opt-in Perfetto tracing configuration (-DPULP_TRACING=ON). Every other lane builds the default OFF config, so a break in the ON path — the Perfetto amalgamation fetch/compile in tools/cmake/PulpTracing.cmake or the trace macros lighting up in <pulp/runtime/trace.hpp> — would otherwise land unnoticed. It configures PULP_TRACING=ON -DPULP_ENABLE_GPU=OFF (Release), builds pulp-test-tracing, pulp-test-tracing-session, pulp-test-offline-tracing, and pulp-test-ship-tracing-guard, and runs them. GPU/Skia stay OFF because the tracing tests only need pulp::runtime + pulp::audio, keeping the lane fast and hostable on a stock GitHub runner. Watch point: test_tracing.cpp's "tracing is off by default" case asserts kTracingEnabled == false and is designed to fail under ON, so the lane excludes exactly that case (Catch2 ~"tracing is off by default"); the other suites are config-agnostic and must fully pass under ON. runs-on is a hard-coded ubuntu-latest — never route it to a self-hosted label or add it to branch protection.

Governance is declared, and mirrors LIVE state — not aspiration

.shipyard/config.toml declares Pulp's branch-protection posture so shipyard governance {diff,apply} can reconcile it, and it is checked in alongside .github/rulesets/main-protection.json. Two hard rules:

• The declared required checks must match the LIVE GitHub ruleset, not what we wish we enforced. Live main requires exactly two contexts: macos and Enforce version & skill sync. Both the [governance] required_status_checks list in .shipyard/config.toml AND the required_status_checks array in main-protection.json are pinned to that two-context set. [branch_protection."main"] require_strict_status = true mirrors the ruleset's strict_required_status_checks_policy. Before editing either, run shipyard governance diff — a clean run prints OK main: no changes; any other output means the checked-in intent has drifted from live and you must reconcile (don't just re-declare).
• Never promote linux/windows into the required set. They validate as advisory GitHub-hosted lanes and deliberately do NOT gate merge. Making an advisory lane blocking craters throughput — a saturated or flaky advisory lane would wedge every merge — without adding signal. This is why the ruleset was trimmed from four contexts to two: so nobody can "fix drift" by pushing a config that flips the advisory lanes blocking. test_ruleset_drift_config.py asserts the two-context required set; keep it and the two config surfaces in lockstep.

PR Review Thread Hygiene

Before opening a follow-up PR or declaring a phase complete, sweep review threads for the PRs touched by that phase:

gh api graphql -f query='
query($owner:String!, $repo:String!, $number:Int!) {
  repository(owner:$owner, name:$repo) {
    pullRequest(number:$number) {
      reviewThreads(first:100) {
        nodes {
          isResolved
          isOutdated
          comments(first:20) {
            nodes { url body author { login } }
          }
        }
      }
    }
  }
}' -F owner=danielraffel -F repo=pulp -F number=<PR>

For every unresolved thread, either fix it in the follow-up branch or verify that current main already fixed it. Leave a reply on the original thread with the fixing commit/PR and the validation that proved it, so future sweeps can distinguish addressed-but-unresolved GitHub state from actual pending work.

Shipping a PR: route through shipyard pr

When the user says any of: "push to main", "ship this", "ship it", "we're done", "merge this", "push it", "run CI", "push a PR" — run shipyard pr (not gh pr create + shipyard ship separately).

shipyard pr is the single orchestrator (Shipyard v0.19.1+; currently pinned in tools/shipyard.toml). It:

1. Calls tools/scripts/skill_sync_check.py (resolved via Shipyard's [validation] path-discovery, explicit in .shipyard/config.toml) and hard-fails if a mapped skill path was touched without a SKILL.md update or a Skill-Update: trailer.
2. Calls tools/scripts/version_bump_check.py --mode=apply to bump SDK, Claude plugin, and marketplace versions consistently, honoring any Version-Bump: trailers. This applies patch/minor/major bumps — including patch, which every fix: PR gets (fixed in pulp #3626; before that, patch bumps were silently skipped and fix:/feat: PRs stranded at the gate, forcing a manual chore: bump versions commit — no longer needed).
3. Runs the no-build source-contract registry gate: tools/import-validation/check-source-contracts.py --strict plus tools/import-validation/test_source_contracts.py. This mirrors the GitHub Versioning & Skill-Sync workflow and the local pre-push hook.
4. Commits the bump (if any) as chore: bump <surfaces>.
5. Pushes the branch, creates the PR, and records Shipyard tracking state.
6. Runs cross-platform validate + merge on green.
7. Auto-release workflow (.github/workflows/auto-release.yml) tags and publishes binaries on merge. The full pipeline (tag → 5-platform build → sign + notarize → 11-asset publish) is documented end-to-end in docs/guides/release-pipeline.md. Keep that doc in sync when you touch any release workflow. Auto-supersede behavior: when a new SDK tag is created, the workflow cancels any in-flight release-cli.yml / sign-and-release.yml runs for strictly-older SemVer tags and deletes their draft releases. This prevents the 2026-05-15 v0.101.x saga from recurring (5 bumps each queueing a full macos-15 darwin-arm64 build, latest waiting hours behind already-obsolete builds). Opt out via vars.PULP_RELEASE_MODE=land-all (repo-wide) or Release-Supersede: skip reason="..." trailer on the merging commit (per-release). See #2076 + the release-draft-stuck-check.yml newest-SemVer skip for the watchdog side of the cleanup.

Never run gh pr create + shipyard ship separately for a normal ship cycle. Never invoke the two version/skill scripts by hand — shipyard pr wires them together with the right flags.

After opening/merging a material PR, sweep its review comments. merge on green fires before the automated reviewers (Codex, and cubic on Shipyard) finish, so a PR can land with unaddressed P1s. For any logic-bearing or destructive-path PR, follow the pr-review-sweep skill: read pulls/<n>/comments (via ghapp), verify each finding against the code, and ship a follow-up with a test for anything confirmed. Docs-only PRs can skip it.

Direct gh pr create is an explicit emergency/manual bypass only. If the user asks for that path, state the tracking gap up front: the PR may not appear in Shipyard-managed state or the macOS GUI until it is reconciled or re-shipped through Shipyard.

pulp pr is a Pulp-side wrapper that delegates to shipyard pr; both are valid, agents should prefer shipyard pr for directness. Humans can opt out of Shipyard for their own checkout with pulp config set pr.workflow github or manual, but agents should not choose those workflows unless the user explicitly asks for a manual/emergency bypass. pulp status reports the effective workflow and whether its required local tool is installed.

Gotcha: shipyard-merged PRs don't reliably auto-close linked issues

Always verify an issue actually closed after a shipyard merge; close it with gh issue close if it didn't. Don't trust closing-keyword auto-close to do it for you here.

What was actually observed (#3299, twice): both Closes #3299 (no colon, PR #3420) and Closes: #3299 (with colon, PR #3413) were correctly recognized as closing keywords — GitHub registered the closing reference in both cases (gh api graphql … closingIssuesReferences returned [{number:3299}] for each). Yet neither shipyard-local[bot] merge auto-closed the issue. So:

• The colon is a red herring for auto-close. Closes #N and Closes: #N both register a closing reference; GitHub accepts either after the keyword. (An earlier version of this note wrongly blamed the colon — corrected here.)
• The real failure is that the app-performed merge didn't trigger GitHub's issue-closing automation. Verify with gh issue view <N> --json state after merge and gh issue close <N> -r completed if still open.

Separately — and this is a real colon rule, but for a different system — git trailer bypasses (Skill-Update:, Version-Bump:, Release-Supersede:) are parsed by git interpret-trailers --parse, which requires the trailer paragraph to be all Key: value lines. A colon-less Closes #N dropped into that paragraph breaks the parse and silently voids every bypass trailer in it. So keep issue references like Closes #N in the body prose, above an all-colon trailer block (Skill-Update: …, Co-Authored-By: …); inside the trailer block, only Refs: #N (colon) is safe. Dry-run git log -1 --format='%(trailers)' to confirm the bypass trailers still parse — that part is unchanged and true.

Backward compatibility: raw shipyard ship / shipyard run still work for diagnostics, experimental branches, existing Shipyard-managed PRs, or when shipyard pr itself is being debugged. Do not use them as the primary ship path.

A ship must survive the session dying — arm GitHub auto-merge as a backstop

shipyard pr performs merge-on-green inside the CLI/worker process. If that process dies before the merge — the cmux app relaunching under resource starvation, or this Claude session running out of quota — the validated PR is stranded unmerged and its ship-state record orphans. This is a recurring real failure (521 orphaned records were reaped across the CI Macs on 2026-06-30). The merge must not depend on any interactive session staying alive.

Standing policy — after creating/validating a pulp PR, arm GitHub-native auto-merge as a server-side backstop:

ghapp pr merge <PR> --auto --merge      # merge commit, NOT --squash

GitHub then merges the moment required checks (macos + Enforce version & skill sync) go green, regardless of whether shipyard pr, cmux, or this session survive. Use --merge (merge commit), never --squash: a squash folds the chore: bump versions commit into the PR-title commit, which trips the auto-release watchdog into a false "merged without bump." It is safe alongside shipyard pr — whichever merges first wins; the other no-ops on already-merged. (The Shipyard repo has no branch protection, so GitHub auto-merge is unavailable there; the host-side queue janitor below covers it.)

Host-side backstop (both repos + orphan reaping): a launchd queue-tick on each CI Mac (tartci scripts/shipyard_queue_tick.sh) periodically drives in-flight ship-state to completion via shipyard's own fail-closed auto-merge and reaps records whose PR GitHub reports merged/closed — independent of any session. Reap a stale local pile by hand with shipyard ship-state list → shipyard ship-state discard <pr> for each merged/closed PR (never discard an OPEN one). Full design: pulp planning/2026-06-30-ship-queue-resilience-design.md.

Stale-SHA merge race — DO NOT push onto a PR that's being shipped

The failure mode (observed 2026-05-29): Shipyard's can_merge() validates the exact merge-candidate SHA and then merges that SHA. If a developer/agent pushes new commits to the same branch while a shipyard ship is mid-flight, Shipyard merges the already-validated older SHA and the new commits are stranded on the branch — the PR squash-merges without them. In the observed case a PR merged at its pre-fix SHA, dropping a whole review-fix push; the fixes had to be re-landed via a fresh fast-follow PR.

expected_head_oid alone would NOT have caught it — the validated SHA was GitHub's PR head at the merge instant; the new push only advanced the branch ref moments later. The only true prevention is the merger re-checking the branch tip immediately before merging and aborting if it advanced past the validated SHA — that lives in Shipyard (tracked upstream as Shipyard issue 321). .

Operational rules (enforce these; they are the practical fix):

1. Never push commits onto a PR that has an in-flight shipyard ship or armed auto-merge. Check shipyard ship-state list (shows PR/url/sha) and GitHub's auto-merge state first. If a ship is running, let it finish.
2. Land post-review fixes as a fresh PR off main, not as a push onto the already-green PR you just reviewed. (Review comes after green; the green PR is exactly when auto-merge fires.)
3. After any ship, verify the merge actually carried your latest commits. Don't trust "merged" — confirm the merge SHA's tree contains your changes. git fetch origin main then check a file you changed is present on origin/main (e.g. git show origin/main:<path> | grep <marker>). If you cross-check via gh api repos/<o>/<r>/pulls/<n> --jq .merge_commit_sha, note that the SHA alone does NOT prove your push made it — that field returns the PR's squash/merge commit on the base branch and exists for any merged PR, including the stale-SHA case. The SHA is only useful if you then inspect its tree/diff (git show <sha>:<path> | grep <marker> or git show <sha> --stat | grep <expected-file>). If your commits are missing, re-land them as a fresh PR immediately.
4. A degraded/rate-limited gh pr view ... headRefOid read can return a stale SHA — corroborate branch state with git ls-remote / a real git fetch before concluding the branch moved or was reset.

Gotcha: a long-lived integration branch drifts from main's whole-tree gates

The hotspot-size guard (tools/scripts/hotspot_size_guard.json frozen ceilings) and the codecov-config gates (codecov.yml flags/components mirroring the live core/* tree) are whole-tree, not diff-scoped — they assert a property of the entire checkout, so they fail on any push to a branch whose tree violates them, regardless of what that push changed. A long-lived develop/* integration branch accumulates this kind of drift relative to main:

• main may lower a frozen hotspot ceiling (e.g. after a comment-hygiene cleanup) while the integration branch has legitimately grown the same file (new test targets), so the merged tree exceeds the ceiling.
• The integration branch may add a whole subsystem (core/<new>/) and its upload flag without registering the matching codecov.yml component, so the flag/component-alignment and "every first-party file maps to a component" tests fail.

Either way the integration branch cannot pass its own pre-push/CI gates, and the failure surfaces on the first unrelated PR that tries to land on it (looks like the PR's fault; it is not). Fix by reconciling the gate config to the branch's real state in the landing PR: raise the hotspot ceiling to the file's current LOC, and add the missing codecov flag+component (paths: core/<new>/**). Run tools/scripts/gates.sh <integration-branch> locally first — the fast gates are sub-second and catch all of this before a multi-minute ship cycle. Reconciling config you didn't author trips skill-sync/version-bump (config maps to the ci skill; examples/config-only diffs still need a Version-Bump: skip trailer under a feat:/fix: title) — expect to add those trailers too.

Shipyard pin and behaviour notes

Pin bumps must go through shipyard pin bump --to vX.Y.Z, not a hand edit. Shipyard v0.50.0+ is Rust-backed and macOS ships as an Apple-Silicon-only signed/notarized .dmg, so the version and asset metadata must move together.

• Pinned at v0.68.0+: a killed shipyard pr/ship worker no longer wedges the PR. Before v0.68.0, a worker that died (crash, kill, launching a second shipyard pr for the same PR) left its job stuck running in the durable queue, and every later same-PR ship was refused with SamePrShipRunning — recoverable only by hand-editing queue.json. At this pin the queue auto-reaps a dead-worker job (heartbeat stale >180s) at ship-submit time and on each drain pass. So if a same-PR ship is refused as "already running" and nothing is actually live, just retry after ~180s. Still: never run two shipyard pr for the same PR concurrently.

• Installing/upgrading Shipyard on macOS uses the GitHub API, which is rate-limited at 60/hr unauthenticated. A "No binary found for shipyard-macos-arm64" error from install.sh / shipyard update is almost always that rate limit, not a missing asset — re-run with GITHUB_TOKEN set (./tools/install-shipyard.sh runs in an authenticated context).

• Release SDKs are expected to include desktop WebView symbols (pulp #695). .github/workflows/release-cli.yml now configures the release build with -DPULP_BUILD_WEBVIEW=ON, installs Linux's libgtk-3-dev + libwebkit2gtk-4.1-dev, and verifies the staged pulp-view-core archive still contains WebViewPanel and make_webview_embedded_resource_fetcher. If you touch the release workflow or tools/scripts/release-cli-local.sh, preserve that contract or WebView-using downstream SDK consumers will link-fail.

• Release bodies are composed in release-cli.yml. The release job runs tools/scripts/compose_release_notes.py for grouped Highlights and calls GitHub's generated-notes API for the native "What's Changed" / "Full Changelog" block before the Install section. CHANGELOG.md remains Shipyard-owned via shipyard changelog regenerate; do not route the Release body back through the deleted in-tree changelog generator.

• Phase 8 CLI flip ships two CLI binaries. Release CLI jobs must preserve Rust pulp as the user-facing binary and C++ pulp-cpp as the fallthrough delegate in the same archive. Smoke both names: pulp version --json for the Rust path, and at least one C++-owned command through PULP_RS_CPP_BINARY=/path/to/pulp-cpp pulp ... or a direct pulp-cpp ... invocation. Do not resurrect pulp-rs as the shipped binary name.

• macOS binary is signed + notarized (Shipyard v0.29.0). On macOS 26.3+ XProtect skips the deep scan for notarized binaries, cutting shipyard pr cold-start ~4-5x (from ~5-6s to ~1-1.5s). No pulp-side action; transparent.

• Format baseline diff is a plugin-only gate. Preserve -DPULP_ENABLE_GPU=OFF in .github/workflows/format-baseline-diff.yml: the self-hosted macOS runner may not have the pinned Skia archive, and this workflow only needs the PulpEffect AU/VST3/CLAP bundles, not GPU examples such as pulp-design-tool.

• Build-and-Test workflow_dispatch is Shipyard PR validation. Preserve -DPULP_ENABLE_GPU=OFF -DPULP_BUILD_EXAMPLES=OFF on the workflow_dispatch configure path in .github/workflows/build.yml: the local self-hosted macOS runner may not have the pinned Skia archive, and no-GPU dispatches must not link example bundles that require the GPU plugin view host. Pull-request validation also disables example bundles, while nightly / release workflows own full example/product and GPU coverage. When adding optional shell arguments in build.yml (for example macOS-only -G Ninja), use bash arrays and expand them as "${args[@]}"; scalar $args trips actionlint/shellcheck word-splitting checks.

• Heartbeat line during long validation (Shipyard v0.29.0). A 20-minute lane now prints periodic progress instead of leaving a silent terminal. Helpful when watching shipyard ship interactively.

• Backend errors are surfaced under the summary table (Shipyard v0.28.0). A bare ubuntu error ssh 12s row used to give zero diagnostic signal — the captured stderr tail (bundle upload failure, remote cmake apply failure, SSH transport error, etc.) now prints below the table. Closes pulp #665's diagnosis-blind-spot complaint.

• Worktree-local .shipyard.local/ falls back to the primary checkout (Shipyard v0.27.2). Pulp uses worktrees heavily for parallel agent work; without this every new worktree had to manually cp .shipyard.local/config.toml from the primary repo before shipyard pr could see the SSH host config. Now it inherits automatically.

• Ship preflight runs BEFORE gh pr create (Shipyard v0.27.1). Earlier the PR was opened first and ship aborted on unreachable SSH backend, leaving stranded PRs with no validation (the Apr 22 pattern that left several pulp PRs mid-flight). Now an unreachable target fails fast and the PR is never opened.

• Daemon tunnel supervisor (Shipyard v0.27.0). Tailscale Funnel transients no longer kill the daemon — the supervisor restarts the funnel on backoff. Periodic reconcile loop runs independently of per-PR polls. Both apply to the macOS GUI's webhook delivery.

• Long-running daemons keep accepting fresh subscribers (Shipyard v0.26.0). The subscribe-replay path now uses blocking put() instead of put_nowait(), so once the replay ring grows past 64 events the daemon no longer silently stops handing new IPC clients their initial snapshot. This was the root cause of the macOS GUI falling back to "polling" mode and showing no active PRs after enough ship-state churn.

• Daemon/CLI drift is now diagnosable over IPC (Shipyard v0.26.0). IPC protocol is bumped to 2, daemon hello/status frames advertise the daemon's own shipyard_version, and shipyard doctor flags a daemon-vs-CLI version mismatch explicitly instead of leaving stale subscribers to fail mysteriously.

• Auto-PR titles and bodies use the feature commit (Shipyard v0.24.0 / Shipyard #151). The orchestrator walks past the mechanical chore: bump versions commit when composing the title/body, and scrubs the Automated by shipyard pr. tool-branding text. Pulp PR #624 was the canonical repro before the fix. Previously the auto-PR pointed at the bump commit — generic and uninformative. Shipped PRs now read as first-party.

• Version-Bump: trailers are authoritative, not ceiling-raising (Shipyard v0.25.0 / Shipyard #152). An author-declared Version-Bump: <surface>=patch reason="..." is no longer silently raised to minor when the conventional-commit heuristic on other subjects classifies the diff as minor. This matches the pulp-side behaviour in tools/scripts/version_bump_check.py at this pin.

• shipyard ship-state list is served from the daemon via IPC when shipyard daemon is running (Shipyard v0.25.0 / Shipyard #154). The PyInstaller cold-start (~5-6s) is bypassed. Callers that tight-loop over ship-state — the macOS GUI polls every 7s; pulp pr preflight calls it indirectly — see a meaningful CPU saving. Nothing to do at the pulp side; it's transparent.

SSH preflight (v0.20.0+ / Shipyard #106)

Exit codes are distinct:

Exit	Meaning
0	Success
1	Validation failed
2	Configuration error
3	Backend unreachable (new; surfaces within 10s with classified reason)

The unreachable-backend error names the failure class (auth / host_key / network / timeout / configuration / unknown) and prints the last ssh stderr.

Flags:

• --skip-target NAME — DELIBERATE lane skip (no probe runs). Use when you know a target is irrelevant for this PR.
• --allow-unreachable-targets — proceed despite an unreachable backend. Prints a loud ⚠︎ VALIDATION GAP: <target> skipped banner. Use only when you genuinely cannot reach a backend and accept the validation gap.

Automation (crons, agents) should branch on exit code 3 specifically rather than parsing error strings.

Tool selection: Shipyard (primary)

Shipyard is Pulp's primary CI tool. All merges, validations, and ship cycles should use Shipyard. local_ci.py remains in the repo as a fallback but is scheduled for removal after a 2-week observation period (see danielraffel/pulp#120).

Prefer Shipyard for GitHub work — it dodges the personal gh rate limit. Shipyard authenticates with its own GitHub App token (higher rate budget), so PR-create / check-watching / merge aren't bound by the developer's personal 5,000/hr gh budget — which is shared across interactive gh, Shipyard, and every self-hosted runner authenticating as the same user. Operational rules:

• gh "The token in keyring is invalid" / gh auth status failing / gh api 403 "rate limit exceeded" is almost always a rate-limit FALSE POSITIVE, not a broken token. gh auth status validates by calling the API; a rate-limited 403 gets mislabeled as an invalid token. Do not gh auth refresh — it wastes a round-trip and fixes nothing. First run gh api rate_limit --jq .resources. High core.remaining (e.g. 4900/5000) with a 403 ⇒ the secondary/abuse limit (fires on bursts / concurrent / expensive calls like gh api .../git/trees/ ...?recursive=1), which clears in ~1–5 min; a low core.remaining ⇒ the primary limit, resets at the top of the hour.
• git push/clone over HTTPS use a separate budget and keep working when gh api 403s — so push branches with git, then let shipyard pr open the PR via its App token.
• shipyard pr's local target-reachability probe still shells out to local gh auth status, so a rate-limited probe prints "Target 'mac' (cloud) is unreachable / gh auth status failed". Push past it with --allow-unreachable-targets (the required GitHub-Actions macos gate still guards the merge). Do not --skip-target mac to dodge it — mac is the only validation target, so skipping leaves none ("No targets remain after --skip-target filtering").
• Reduce burst: git clone --depth 1 instead of recursive tree-API dumps; space gh calls; never tight-loop gh (back off / schedule).

# Primary: Shipyard
shipyard run                              # validate current branch
shipyard pr                               # create, track, validate, and merge on green
shipyard ship --resume                    # pick up an interrupted ship (v0.3.0+)
shipyard ship --no-resume                 # discard stale state, start fresh
shipyard ship-state list                  # in-flight ships (title, url, sha)
shipyard ship-state show <pr>             # full state for one PR
shipyard ship-state discard <pr>          # archive stale state
shipyard cleanup --ship-state --apply     # prune closed-PR + aged state
shipyard run --targets windows --smoke    # fast Windows-only check
shipyard run --resume-from test           # skip configure+build, run tests only
shipyard cloud run build <branch>         # dispatch the GHA build workflow
shipyard rescue <PR>                      # recover a wedged PR by redispatching queued runs
shipyard rescue <PR> --rerun-failed       # v0.67.0+: also re-dispatches FAILED/timed-out runs (not just cancelled), and — with --to omitted — RE-RESOLVES the provider local-first (overflow-aware) instead of forcing github-hosted. This is the lever to recover a saturated/timed-out macOS leg (re-run it on a real local runner). Pass --to <provider> to force.
shipyard rescue <PR> --rerun-failed --to local   # force a re-run onto the local runner
shipyard ship --pr N --base main --adopt-head     # recover "ship state SHA drift" after a force-push (Shipyard #346): adopt the current branch head + re-validate, instead of re-shipping from scratch