| name | ci |
| description | Local and cloud CI for Pulp — validate branches, create PRs, merge on green. Handles "push to main", "ship this", "run CI", "check PR", and "list PRs". |
| requires | {"scripts":["tools/local-ci/local_ci.py"],"tools":["gh"]} |
CI Skill
Validate branches and ship code safely. This skill handles all CI workflows for Pulp across local machines and VMs.
Runner timing metrics
When asked whether Pulp's local runners are fast, stuck, regressing, or worth
monitoring, query Shipyard's metrics surface before guessing. Pulp does not
mirror these records into pulp CLI or pulp-mcp; Shipyard is the metrics
store and tartci is an optional VM runtime emitter.
This metrics surface requires a Shipyard build that includes the
shipyard metrics subcommand. Pulp's current pinned source-checkout Shipyard in
tools/shipyard.toml is v0.68.0, which does not include it yet. If a checkout
only has the pinned binary, use a newer Shipyard binary or source checkout for
the optional metrics workflow, or skip metrics and inspect live jobs directly.
Use these commands as the normal agent loop:
shipyard metrics import github --repo danielraffel/pulp --limit 50 --json
tartci runtime export --repo danielraffel/pulp --since-days 14 \
| shipyard metrics import tartci --json
shipyard metrics summary --project pulp --json
shipyard metrics watch --project pulp --since 14d --json
shipyard metrics advise --project pulp --json
Read watch as a triage signal, not as a hard failure. insufficient_samples
means the lane does not have enough history yet; drift, failure-rate, or slowest
findings are the useful prompts to inspect runner logs, VM boot behavior, cache
state, or GitHub queue time. If tartci is not installed for a repo, skip the
tartci runtime export import and still use the GitHub import plus Shipyard
summary/watch commands.
If a PR's required macos check has been queued >30 min or the
repo's PRs are all in mergeable_state=blocked with no movement,
jump to "Self-hosted runner ops" near the end of this file.
One-shot recovery is shipyard rescue <PR> (Shipyard v0.53.0+).
Continuous prevention is shipyard runner watch --kill-hung-workers
(v0.54.0+). Keep Shipyard itself current with shipyard update
(v0.55.0+; Pulp currently pins v0.56.2+). All three replace the
legacy planning/scripts/runner-watchdog.sh --fix workflow, which is
now an anti-pattern (cancels queued runs but registers failure on
required checks).
Pre-flight: plugin ↔ CLI skew check
Before shelling out to pulp (or shipyard pr, which ultimately
invokes pulp), source the shared skew-check helper so a user on an
outdated CLI sees a one-line hint rather than running into obscure
flag-missing errors mid-ship:
source "$(git rev-parse --show-toplevel)/tools/scripts/cli_version_check.sh"
pulp_cli_version_check
This is advisory only — the helper never blocks. See the upgrade
skill for the full banner contract and override knobs
(PULP_SKEW_CHECK_DISABLE, PULP_SKEW_CHECK_CACHE). Release-discovery
Slice 6 (#551).
Diagnosing a slow / stuck PR — investigate before assuming runner saturation
When a PR sits without merging, don't ASSUME "the macOS CI pool is saturated"
from a long-queued run — investigate. Saturation IS possible (a genuine burst, or
a wedged/dead runner — see the pulp-runner-ops skill), so it's worth checking;
just don't conclude it without evidence. In the 2026-06-18 case the cause turned
out to be non-hardware (a misdiagnosis worth not repeating). Check in this order:
- Did the required checks even register? A PR opened by the Shipyard GitHub
App does NOT auto-trigger
pull_request workflows, so the required macos
and Enforce version & skill sync checks never appear on the PR head SHA until
you dispatch them by hand:
ghapp workflow run build.yml --ref <branch>
ghapp workflow run version-skill-check.yml --ref <branch>
This is the most common reason a Shipyard PR "sits." (shipyard pr dispatches
build.yml itself but you may still need version-skill-check.yml.) After a
new push the head SHA changes — re-dispatch on the new SHA.
- Is it a version-bump race? The other concurrent agent re-bumping
main's
CMakeLists.txt VERSION makes the PR DIRTY (conflict on the VERSION line).
Merge origin/main in, re-resolve the VERSION to one above main, push,
re-dispatch the checks. (Longer-term fix for this whole class:
planning/2026-07-07-parallel-merge-land-coordination.md. In progress: the
version-at-land.yml bot — currently DRY-RUN — will make a single writer
assign the version from Version-Bump: intent trailers post-merge, removing
the per-PR race. Until it flips to --apply, keep hand-bumping as today.)
2b. Did a gate reject the push? Besides skill-sync / version-bump, the
pre-push + CI planning-gitlink gate (tools/scripts/planning_gitlink_guard.py)
fails if the PR moved the planning submodule pointer without a
Planning-Bump: trailer — usually an accidental bump from a git reset --hard
git add -A. Drop it with
git restore --staged --worktree planning && git submodule update planning,
or add Planning-Bump: reason="..." for a deliberate re-pin.
2b1. Config→doc drift? The config-doc gate (tools/scripts/config_doc_check.py,
map in tools/scripts/config_doc_map.json) fails if a mapped CI/release config
surface (.shipyard/config.toml, the shipyard pin/installer, the
build/auto-release/version-skill-check/coverage workflows) changed
without its guide doc (versioning.md / local-ci.md / release-watchdog.md).
Editing a workflow under .github/workflows/ therefore usually needs a matching
guide edit; a genuinely doc-irrelevant change bypasses with a
Config-Doc: skip reason="..." trailer on any commit in the range.
2b2. Hotspot growth? The hotspot-size gate is now net-delta vs merge-base:
it fails only if THIS PR grows a frozen hotspot past its reference size (main
growing the same file is NOT your fault and passes). If you must grow one,
make the change net-neutral (extract to a sibling file) or add
Hotspot-Grow: <path> reason="..." — do NOT raise max_loc in
hotspot_size_guard.json (that counter no longer gates growth, and raising it
races with other PRs).
2b3. Hotspot shrink? The ceiling is a one-way ratchet, and this is the
opposite instruction to 2b2. If your PR makes a frozen hotspot smaller, the
gate fails with missing hotspot ceiling reduction(s) until you lower
max_loc to the new LOC. Lowering is safe (it cannot race a concurrent PR into
a false pass); raising is not. So: never raise it, always lower it when you
shrink. A PR that both grows one hotspot and shrinks another needs a
Hotspot-Grow: trailer and a max_loc reduction.
2c. Is a RED check even your fault? Before investigating a failing check,
run python3 tools/scripts/pr_check_triage.py <PR#> — it labels each red
check REQUIRED vs advisory and PRE-EXISTING (also red / not run on main —
not your change) vs REGRESSED (green on main, red here). Advisory +
pre-existing red (e.g. a known-broken sanitizer lane on main) does NOT block
the merge and is not yours to fix; only a REQUIRED + REGRESSED row needs
action. This alone avoids chasing main-side breakage.
- Only THEN consider capacity — and verify, don't assume. The required
macos gate runs on the local self-hosted Mac Studios (pulp-studio-01/02/03,
- the M5 overflow), which are usually idle. Confirm with:
ghapp api repos/danielraffel/pulp/actions/runners \
| python3 -c "import sys,json;[print(r['name'],r['status'],'busy='+str(r['busy'])) for r in json.load(sys.stdin)['runners']]"
If the Studios show busy=False, the pool is NOT saturated — say so. What DOES
queue independently is the GitHub-hosted advisory lanes (Linux, Windows,
sanitizers, coverage, android) on GitHub's shared pool; those are advisory, not
the required gate, so a long queue there does not block merge.
- Is
shipyard run/ship failing at stage=configure with D external/skia-build/build? That's the Skia symlink loop (a tracked
machine-specific absolute symlink autofetch deletes at configure → tree-drift),
not capacity or a code failure — the local mac validation dies before posting
its macos status, so the PR stays BLOCKED with the required check absent.
Fixed at the repo level by PR #5588 (untracked + .gitignored), but a
pre-#5588 checkout keeps the stale looped symlink until it pulls main. Recovery
- full mechanics: the
skia-gpu-build skill's Gotchas section
(ln -sfn ~/.cache/pulp/skia-build/build <primary>/external/skia-build/build).
If you don't use Shipyard + the self-hosted Mac pool: steps 1 and 3 are
specific to that setup (App-dispatched workflows; local runners) — skip them. The
tool-agnostic rule still holds: distinguish the required checks from advisory
lanes, and verify a runner is actually busy before blaming capacity.
Caveat to step 3 — idle Studios do NOT prove the required gate is unblocked.
The required macos check is gated by a routing preamble (classify →
resolve-provider) AND is itself an alias job that reports the leg's outcome —
all three default to runs-on: ubuntu-latest. So when GitHub's shared ubuntu
pool saturates (repo-wide actions/runs?status=queued in the hundreds during a
PR burst), the preamble can't get a slot, the macOS leg is never dispatched, and
the required gate sits pending with the Studios idle. Tell this apart from a
real failure and from a wedged runner: queued preamble + online/busy=false
Studios + high repo-wide queued count = GitHub-hosted starvation, not a code
or hardware problem. Immediate unblock is admin-merge after local validation
(ghapp pr merge <n> --merge --admin, --merge not --squash to preserve the
bump marker); merging also drains that PR's ~11-workflow fan-out from the queue.
The structural fix is the PULP_PREAMBLE_RUNS_ON_JSON repo var: set it to a
self-hosted Linux selector (e.g. ["self-hosted","Linux","ARM64", "pulp-build-linux"]) to move those three jobs off the saturated pool — but only
once that pool is confirmed always-on, or the gate just starves elsewhere. It
defaults to ubuntu-latest (no-op) until set. A tartci launchd detector watches
for this triad; full design in
planning/2026-07-06-ci-queue-saturation-watchdog.md.
Host-vitals preflight — back off before a saturating CI host reboots
The self-hosted Mac Studio that runs the required macos gate ALSO hosts the
interactive agent session and a heavy MCP stack (RepoPrompt, Figma,
chrome-devtools, several pulp-mcp). When RAM fills, macOS jetsam starts killing
processes, the window server crashes, and the host reboots uncleanly — taking any
in-flight required-gate CI job down with it and reddening the leg for reasons that
have nothing to do with the code (the 2026-07-01 incident). This whole class of
failure is predictable ~20 minutes out from cheap metrics, so the agent, the
CI pool, and Shipyard should all read one shared signal and back off.
tools/scripts/host_vitals.sh is that signal. It reports green / warn /
critical (exit 0 / 10 / 20) keyed on memory pressure first
(kern.memorystatus_vm_pressure_level + fresh JetsamEvent-* reports), with load
only ever corroborating a warn — a healthy parallel build (load 1–2× cores, normal
pressure) never trips it. Use it before heavy work:
tools/scripts/host_vitals.sh
tools/scripts/host_vitals.sh --json
- Agent admission control: before launching a local build or a heavy MCP call
on a CI-host session, check
host_vitals.sh. If critical, don't pile on —
ship via shipyard pr / GitHub-native auto-merge (survives a restart) instead
of a foreground shipyard/ci watch, and shed idle load (close RepoPrompt/Figma
/idle MCP) before building. gates.sh prints this banner advisorily on every
pre-push.
- A "hung"/"stuck"
git push is almost always the pre-push diff-cover BUILD,
not the network. When the diff touches a coverage surface (core/,
tools/cli/, tools/scripts/), .githooks/pre-push runs a full local
configure + compile before the transfer. It prints a loud DIFF-COVERAGE BUILD running banner and a …still running (Ns elapsed) heartbeat every 30s so this
is unmistakable — if you see those, it is a BUILD (minutes), NOT a transport
hang. Tell: git fetch/ls-remote succeed while the push "hangs", so SSH/HTTPS
are fine. Do NOT diagnose the network, and do NOT background-and-kill the push
at a short timeout (that hides the banner/heartbeat). When the work is already
validated (e.g. via shipyard pr), push with PULP_SKIP_DIFF_COVER=1 git push
and it completes in seconds. (Learned 2026-07-07 after ~an hour lost mis-blaming
MTU/SSH for a diff-cover build during a deps-bump push.)
- Continuous sensor:
tools/scripts/install_host_vitals_sensor.sh installs a
per-user launchd agent (com.pulp.host-vitals, 60 s) that publishes the latest
reading to ~/.local/state/pulp/host_vitals.json and a rotating
host_vitals.log. It is observation-only — it never stops a runner or kills a
process — so it is safe to run on the required-gate host. Installed on the m3/m5
/m1 pool. install_host_vitals_sensor.sh --status shows the launchd + latest
reading; --uninstall removes the agent.
- A whole-pool-fails-at-once red leg is infra, not code (per
macos-required-leg-timeout-saturation): if windows + both macos legs fail
together and the diff can't explain it, correlate against host reboot / jetsam
time and re-run the required leg rather than debugging the change. Active
back-off in the CI pool (tartci auto-yield) and Shipyard (host-health dispatch
gate + infra-vs-code classification) consume this same host_vitals state.
GitHub workflow gotchas
-
Every on: schedule workflow must carry the fork guard. When someone forks
the repo, GitHub copies all workflows and runs the scheduled ones on the fork's
default branch — then emails the fork owner whenever one fails, which our
monitors reliably do on a fork (they probe this repo's state or use secrets the
fork lacks). So each entry job (a job with no needs: — dependents cascade-skip)
of a scheduled workflow gets:
jobs:
check:
if: github.event_name != 'schedule' || github.repository == 'danielraffel/pulp'
compose it with an existing condition as
if: (github.event_name != 'schedule' || github.repository == 'danielraffel/pulp') && (<existing>).
It only suppresses the schedule event on forks — push / pull_request /
workflow_dispatch are untouched, so PRs to this repo (which run in this repo's
context) and manual dispatches behave exactly as before. A workflow that
should run on forks' schedules opts out with a top-of-file
# fork-guard-exempt: <reason> comment. This is enforced:
tools/scripts/scheduled_workflow_fork_guard_check.py runs in gates.sh, the
pre-push hook, and workflow-lint.yml, so a new scheduled workflow missing the
guard fails the PR. Add the guard when you add the workflow.
-
Codecov "total lines/files dropped" is usually upload starvation, not config drift. Two distinct guard layers exist and they catch different things: test_codecov_components.py / test_codecov_config.py (PR-gate) guard the codecov.yml mapping (a subsystem matching no component → invisible); coverage-upload-watchdog.yml (hourly, main) guards the outcome — "main had a successful Coverage run in the last N hours." The watchdog exists because the dashboard degrades silently when fresh complete uploads stop arriving, from causes the config gate can't see: coverage runs cancelled by stale-run-reaper.yml when a CI dispatch throttle makes them queue past the cutoff (the 2026-06-28 incident — 8 consecutive main Coverage runs cancelled), an llvm-cov -object-set regression (a libpulp-*.a drops out → a whole subsystem vanishes while lines-valid stays > 0, which verify_cobertura_xml.py's lines-valid > 0 check cannot catch), or Codecov's after_n_builds waiting forever on a missing per-OS leg. When triaging a coverage-surface complaint: first check gh run list --workflow coverage.yml --branch main for recent successful runs (cancelled ≠ uploaded); only then suspect codecov.yml. Note the config gate is advisory (not in branch protection), so a fleet-auto-merged PR can land config drift despite a red gate — the watchdog is the main-branch backstop.
-
web-plugins.yml is the headless-browser web lane (advisory). It builds
Pulp's WAMv2 (Emscripten) and WebCLAP (wasi-sdk) web plugin formats on a Linux
GitHub-hosted runner and runs every web validation, including the browser
fixtures in headless Chrome (browser-actions/setup-chrome +
playwright-core, which drives the system Chrome — no browser download). It is
deliberately NOT on the self-hosted VMs: a headless-Linux browser lane needs
emsdk + wasi-sdk + Chrome, all of which install cleanly on GitHub-hosted, so no
golden VM or QEMU work is warranted. The WAM build vendors choc by cloning the
pinned fork (PulpWam.cmake needs PULP_WAM_CHOC_INCLUDE; the WebCLAP build
FetchContents choc/clap itself). The browser drivers are
examples/web-demos/*/{browser-test,browser-host}/validate.mjs; run them
locally with a system Chrome/Canary via node validate.mjs --screenshot out.png
(set CHROME_PATH or pass --browser). The WAM node validations also cover
the in-worklet rack path (wam_rack_runner.mjs against the
PulpPluckGainRack target — the only committed consumer that compiles
wam_chain_entry.cpp/WamChainBridge), the wire protocol + version-skew
bridge contract (core/format/src/wasm/wam-runtime.test.mjs, pure JS), and
hostile-state rejection (overflow/CRC/truncation) folded into
wam_feature_runner.mjs — so a regression in the state codec or chain runtime
fails here rather than only in a browser.
-
wclap-cloudflare.yml is WebCLAP's canonical-host lane (Cloudflare Pages).
Separate from web-plugins.yml: it builds the threaded WebCLAP PulpGain module
(pinned wasi-sdk), assembles a self-contained deploy dir, proves it headlessly
under the real _headers, then owner-gated-deploys. Two pages are assembled +
proven: the single-plugin isolation page (assemble.mjs → validate-deploy.mjs)
and the shared-player WebCLAP demo (assemble-player.mjs → player/ →
validate-player.mjs) — the latter mounts the SAME @danielraffel/web-player shell the
WAM demos use, driven by the WebCLAP adapter (a worklet-resident CLAP host), and
asserts crossOriginIsolated, real-time render (worklet quanta ≈ sampleRate/128
per wall-second), an audible param change that updates the shared widget, and a
clap.state round-trip. The lane also runs the @danielraffel/web-player adapter unit
tests (packages/pulp-web-player — npm test), which include an ABI-parity
check that the worklet's inlined CLAP struct offsets match wclap-abi.mjs.
Hard-won gotchas encoded there: posting a WebAssembly.Module INTO an
AudioWorklet is silently dropped in Chrome (transfer the raw bytes and compile
in-worklet), and transferring an ArrayBuffer OUT of an AudioWorklet is
unreliable (the receiver gets a detached buffer) — clone small payloads (state,
sysex) instead, and never transfer a caller-owned buffer (it detaches theirs).
-
screenshot-sync is a three-layer gate that mirrors skill-sync. A repo opts
in by committing a .pulp/screenshots.toml manifest (presence == opt-in);
tools/scripts/screenshot_sync_check.py then diffs the manifest's [trigger].paths
and fails when a triggered target's committed PNG/OG image wasn't refreshed.
Wired identically to skill-sync: PostToolUse hint in
hooks/scripts/cli-plugin-sync.sh, pre-push report in .githooks/pre-push, and
authoritative CI step in version-skill-check.yml. Bypass a single commit with a
Screenshot-Sync: skip target=<id|all> reason="..." trailer. Pulp core itself is
NOT opted in (no manifest), so the gate no-ops here; it exists for downstream
plugin repos (GPU NAM, example plugins, Bendr) and the WCLAP/WAM OG images.
-
Android native .cxx caches must be dependency-aware. The Android workflow
builds through Gradle's external native build, and android/app/.cxx can hold
FetchContent checkouts under _deps. Do not cache .cxx under a Gradle-only
key or a broad Gradle restore key: after a native dependency bump, CMake's
UPDATE_DISCONNECTED path can see the stale local checkout and fail because
the new git ref is not present. Keep the Gradle cache (~/.gradle) separate
from the native CMake cache, key .cxx on the native dependency inputs
(CMakeLists.txt, tools/cmake/PulpAndroid.cmake,
tools/cmake/PulpDependencies.cmake, tools/deps/manifest.json, plus Android
Gradle files), and do not give .cxx a restore key that ignores those inputs.
-
intent-bump-on-merge.yml is the merge-time half of the version-bump
intent-trailer model — and it ships DORMANT. It exists to kill the
version-bump merge treadmill (PRs editing CMakeLists VERSION /
plugin.json / marketplace.json re-conflict every time main advances). The
endgame: a PR declares Version-Bump: <surface>=<level> and touches NO
version files, and this workflow assigns the exact number after merge from
main's current version via tools/scripts/apply_intent_bump.py. Phase 1
(current): no-op. Nothing emits intent trailers yet (Shipyard still file-
bumps on the PR side; version-skill-check.yml still runs WITHOUT
--accept-intent-trailers), so every run finds no trailer and exits clean.
Two things must be verified before the phase-2 flip (a separate, reviewed
change): (1) RELEASE_BOT_TOKEN can push a commit to protected main
(it already pushes tags from auto-release.yml; a commit needs the bot on the
branch-protection bypass list), and (2) the Version-Bump: trailer survives
squash-merge into main's commit message. The workflow has a recursion guard
(skips its own chore: bump versions commit) and a concurrency group so
near-simultaneous merges bump the version line one at a time. The
chore: bump versions commit it pushes triggers auto-release.yml exactly
like a PR-side bump.
-
test/CMakeLists.txt is now an include hub, not a registration
manifest. Add new add_test, add_executable, and
pulp_add_test_suite blocks to the matching test/cmake/*_tests.cmake
manifest instead of rebuilding the old monolith. If no existing manifest
owns the subsystem, create a focused new test/cmake/<area>_tests.cmake
file and include it from test/CMakeLists.txt in dependency order. Focused
owner hubs may include smaller manifests for their own subsystem, but do not
hide sibling manifests inside an unrelated owner just to keep the top-level
file short. Keep the top-level hotspot_size_guard.json ceiling at the exact
tiny hub LOC; raising it for ordinary test additions is a regression. Before
burning a PR CI job on a manifest-only cleanup, run the local preflight:
python3 tools/scripts/docs_noise_lint.py --mode=report --base origin/main,
python3 tools/scripts/hotspot_size_guard.py --base origin/main --config tools/scripts/hotspot_size_guard.json --mode=report --require-ceiling-reduction, and a clean CMake configure/target-list compare
against origin/main.
-
Source hotspots (e.g. core/view/src/design_cpp_codegen.cpp) are frozen
too — bump the ceiling for a coherent feature, split when it's accretion.
hotspot_size_guard.json also freezes large source files. When a single,
cohesive feature legitimately grows one (e.g. teaching the C++ codegen to emit
faithful_svg as a DesignFrameView), raise that file's max_loc in the same
PR — splitting a tightly-coupled emitter mid-feature would hurt readability
more than the extra LOC. Reserve the split for genuine grab-bag growth.
When a PR shrinks a tracked hotspot, lower that file's max_loc in
hotspot_size_guard.json to the new exact LOC in the same PR; the
--require-ceiling-reduction gate compares the merge-base blob against HEAD
for branch-touched tracked hotspots, so leaving headroom after a shrink fails
the pre-push/CI guard.
Exception — a hotspot already ABOVE its ceiling. check_hotspots only
fails the PR that itself grew a file, so main can drift past a max_loc
while every individual PR stays net-neutral (e.g. window_host_mac.mm at 2780
LOC against a 2770 ceiling). Shrinking such a file cannot lower the ceiling —
setting max_loc to the new LOC would raise it — so the guard does not ask
for a reduction there, and prints still at/over ceiling; no reduction to make. The ratchet is demanded only when the shrink lands strictly under the
existing ceiling. If you see the guard demand a max_loc larger than the one
in the config, that is the bug this rule fixed, not a config you should edit.
Editing hotspot_size_guard.json itself trips the ci skill-sync gate; the
ceiling bump is normally part of a non-ci feature PR, so either fold this
note's rationale into that PR or skip the ci gate with a
Skill-Update: skip skill=ci reason="ceiling bump only" trailer on the tip
commit (note: a later chore: bump versions commit from shipyard pr displaces
the tip, so updating this SKILL is the more robust path).
-
Inspector hotspots are frozen too. hotspot_size_guard.json watches newly
added inspect/** files and freezes the current inspector overlay, window,
domain handler, and tweak-store hotspots. When an inspector extraction shrinks
a tracked file, lower its max_loc to the exact new line count in the same
change. When a new overlay companion file triggers the large-file warning,
split it before it becomes another hotspot.
-
Reskinnability ratchet (token-coverage-ratchet ctest). Driven by
tools/scripts/token_coverage_check.py: fails if a core/view/src paint file
gains a NEW colour literal that is not a resolve_color(...) fallback. Mark a
deliberate material-effect literal // token-lint:allow; after intentionally
lowering a file's count, re-freeze with --update-baseline.
-
SignalGraph single-backend governance (single_backend_guard.py +
processing_model_terms_lint.py). Whole-tree structural guards (not
diff-scoped) that keep the DSP runtime convergence from regrowing a second
surface: exactly one type (GraphRuntimeExecutor) may define
process_routed / process_parallel; pulp create offers no graph-plugin
authoring scaffold; the public generated-DSP ABI entry symbols stay the
sanctioned pair (pulp_native_core_entry_v1, pulp_node_v1_entry); and the
differential parity test stays registered as a built target. They run in
gates.sh, the pre-push hook, and the Versioning & Skill-Sync workflow
(hard-fail). To sanction a genuinely-new routing engine, authoring template,
or generated-DSP ABI, widen the allowlist in single_backend_guard.py in the
same PR — that diff is the architecture-review record. Both guards carry a
--selftest run in the workflow's fixture-test step. Hardening notes (from the
series adversarial review): a non-sanctioned Rival::process_routed is flagged
even when defined inside graph_runtime_executor.{hpp,cpp} (no TU-filename
escape), and the parity-registration check requires the test source to appear on
a real SOURCES/add_executable/target_sources/pulp_add_test_suite line —
a bare mention in a dead variable no longer counts.
-
Conflict-marker guard (conflict_marker_check.py). Whole-tree guard (not
diff-scoped): no tracked file may carry a git conflict marker. Born from the
incident where a squash-merge's stale side collided with an already-advanced
project() VERSION line and wrote <<<<<<< / ======= / >>>>>>> straight into
CMakeLists.txt (fixed in #5477), breaking every build until a human noticed.
Keyed on the start/base/end markers (<<<<<<< / ||||||| / >>>>>>> at
column 0, followed by whitespace/EOL) — verified zero-false-positive across the
whole tracked tree, external/ included; the ======= separator is reported
only inside a real conflict block, so Markdown headings and ASCII banners stay
clean. Runs in three layers: gates.sh + the pre-push hook (local), the
Versioning & Skill-Sync workflow's Conflict-marker guard step (which scans
the pull_request MERGE ref, so a marker born from the merge itself is caught,
not only one on the PR head), and conflict-marker-guard.yml — a push:main
backstop that reddens the branch and opens a tracking issue if a marker reaches
main by any path (the squash case, where GitHub had no clean mergeable ref for
the PR job to inspect). Carries a --selftest in the fixture-test step. A
vendored fixture that legitimately ships markers is a reviewable ALLOWLIST
edit in the script. If the push:main guard reddens main, run
python3 tools/scripts/conflict_marker_check.py for the file:line list, resolve,
and push — the tracker auto-closes on the next clean push. Exit codes are
meaningful: 0 clean, 1 markers found (the backstop opens the tracker), 2+
internal error (the backstop fails the run without opening a wrong "markers
committed" issue) — the selftest locks this contract. Documented scope
limitations (deliberate, to keep zero false positives): only default
seven-char markers (a non-default .gitattributes conflict-marker-size is
missed), submodule contents are out of scope (the superproject sees a
gitlink), and NUL-bearing/UTF-16 text is skipped as binary. The upstream fix
for the whole class — the merge tool refusing to commit a conflicted result —
is tracked in Shipyard #372; this guard is the consumer-side backstop.
-
Thread-safe-assertions guard (thread_assert_check.py). Catch2 3.7.1's
assertion macros are NOT thread-safe (thread-safe assertions are opt-in only
from Catch2 3.9.0), so a REQUIRE/CHECK/FAIL/SUCCEED inside a
std::thread / std::jthread / std::async lambda in test/ is undefined
behavior: it corrupts Catch's per-run assertion counters + section state. Bare
metal tolerated it, but the move of the required macOS gate onto Tart VMs made
the different scheduler timing trip it — a HotSwapSlot hammer test "failed" in
0.00s though the code under test was race-free by design. The correct pattern:
record the violation into an atomic/guarded value in the worker, join(),
then assert on the test thread (REQUIRE_FALSE(bad.load())). The lint is a
lexical scan (best-effort: it does not follow calls into helpers) with a
same-length string/comment-blanking pass that also skips C++ digit separators
(10'000); suppress a verified-safe line with a trailing
// thread-assert:allow. Runs as the thread-safe-assertions ctest case and
in gates.sh. When graduating any required lane to VMs, expect this class of
latent UB to surface — fix at the source, don't suppress.
-
Release builds must pass -DPULP_BUILD_EXAMPLES=OFF. The
pulp-design-tool example hard-fails CMake configure when PULP_HAS_SKIA
is FALSE (belt-and-suspenders, code 78). sign-and-release.yml builds on a
GitHub-hosted macOS runner with no Skia, so configuring the full tree
(examples included) aborts the entire run → no GitHub Release publishes
even when release-cli.yml (the CLI build) is green. This silently broke
Release publication (the release watchdog flagged the missing Releases). The
release ships the CLI + plugins
(build/VST3,/CLAP,/AU), never the example apps, so the release legs
build with -DPULP_BUILD_EXAMPLES=OFF (matching build.yml). If you add a
new release/packaging workflow, configure with examples OFF (or a populated
SKIA_DIR), or the design-tool Skia gate will block it.
-
Release/SDK builds must pass -DPULP_ENABLE_AUDIO_PROBES=OFF. The
standalone audio probe is a dev/example inspection surface and defaults ON
for local development, but shipped CLI, standalone, and SDK artifacts must
not export it. Keep release-cli.yml, release-path-pr-gate.yml,
release-dry-run.yml, sign-and-release.yml, release-cli-local.sh, and
checkout-backed SDK configure paths aligned when touching release CMake
flags.
-
sign-and-release.yml waits on release-cli.yml to create the release —
the wait must cover release-cli's FULL runtime. Since the release split
(release-cli is the sole creator of the GitHub release; sign-and-release only
attaches appcast.xml on top), sign-and-release polls gh release view "$TAG"
until the release exists. release-cli creates it only after its whole chain
(build-cli → smoke-cli → universal-arch-gate), which includes a self-hosted
darwin-arm64 leg that can queue behind a busy runner — routinely far longer
than 10 minutes. A too-short poll loses that race on every tag and leaves the
published release without its Sparkle feed (the watchdog flags it as a broken
release). Keep the wait window generous (currently 150 * 20s = 50 min); if
release-cli grows a slower leg, widen it rather than letting the appcast attach
silently time out. The SDK/CLI binaries are unaffected either way — they are
release-cli's output; only the appcast rides on sign-and-release.
-
Hooks inherit GIT_DIR — tests that shell out to git can corrupt the live
worktree. Git exports GIT_DIR/GIT_WORK_TREE into hook environments, and
a set GIT_DIR overrides git -C <dir> discovery. So when the pre-push
hook (or shipyard) runs the full ctest and a test does
git -C <tempdir> init/commit/checkout, those commands silently hit THIS
worktree's repo instead — stray initial commits, throwaway branches, and a
core.bare=true flip in the shared config (which makes the main checkout
look "not a work tree"). .githooks/pre-push and
tools/scripts/local_diff_cover.sh now unset GIT_DIR GIT_WORK_TREE …
before running gates/ctest, and the git-shelling tests scrub the same vars
in-process (see the regression test in test_mcp_server.cpp). If you add a
test that shells out to git on a temp repo, clear the inherited git env first
(or run from a context with no GIT_DIR), and never assume -C alone
isolates it. Recovery if a worktree was hit: git config core.bare false,
reset the branch off the stray initial commit, delete the throwaway branch.
-
The required macos alias in .github/workflows/build.yml mirrors the
macOS matrix leg by polling the Actions jobs API. Keep that poll retry-safe:
API failures or malformed JSON must log and continue the loop, not trip
set -euo pipefail before the macOS leg has a chance to report.
-
.github/workflows/release-dry-run.yml (P9-2, #2576) exercises the release
build → package_cli.py → pulp ship appcast chain on a synthetic version
(0.0.0-dryrun) WITHOUT publishing — no GitHub release, no signing/notarize,
no appcast upload; artifacts are throwaway. It's additive (does not touch
release-cli.yml / sign-and-release.yml) and runs weekly + on demand, so a
build/packaging/appcast-generation regression surfaces BEFORE a real tag.
Keep it credential-free (notarize/sign stay in the real path) so it can run
on a schedule without secrets.
-
Release-bot source refs must be SSH-signed. auto-release.yml creates
signed annotated v* tags with git tag -s, and the bot commit workflows
(intent-bump-on-merge.yml, post-tag-sync.yml) configure the same SSH
signing helper before committing. The required Actions secret is
RELEASE_BOT_SSH_SIGNING_KEY; it is a file-backed OpenSSH private key backed
up outside the repo. The workflow uses 25807+danielraffel@users.noreply.github.com
because GitHub verifies SSH signatures against the account that owns the
uploaded signing key. If this secret is absent, the signing setup must fail
closed rather than create unsigned release tags or bot commits.
-
Release-runner Xcode must be pinned (C++20 parity). sign-and-release.yml
runs on GitHub-hosted macos-14, whose DEFAULT Xcode is 15.4 — its Apple clang
lacks C++20 P0960 (parenthesized aggregate init, Type p(arg) for a
ctor-less aggregate). The self-hosted PR macos lane uses a much newer clang
that accepts it, so a CLI/import TU compiled on every PR but FAILED only in the
release build — silently breaking GitHub Releases v0.372–v0.391 (tags kept
being cut; only the release-cadence watchdog noticed). The job now selects the
newest installed Xcode 16.x via xcode-select (shell, no third-party action so
an actions-allowlist can't hold the release hostage), restoring C++20 parity
with the PR lane. When a release/packaging workflow builds C++ on a GitHub-
hosted macOS runner, pin a modern Xcode — the default lags and silently
diverges from the PR toolchain. (Code-side defense: always brace aggregate init
Type p{arg} in CLI/import code — see the import-design skill.)
-
The release build is NOT a test gate. sign-and-release.yml no longer
re-runs the unit suite (ctest). By the time a commit is tagged it has already
passed the FULL suite on the PR/merge gate (self-hosted lane, real
GPU/display/iOS-SDK). Re-running on the HEADLESS GitHub-hosted release runner is
redundant and yields false failures from environment-only tests (Skia-raster
screenshot → empty, cmake-require-gpu → timeout, cmake-ios-hostapp-links) that
pass on real hardware — that blocked Releases AFTER the Xcode-pin let the build
through. Principle: tests gate at PR on representative hardware; the release
builds + signs + notarizes + packages the validated commit (the Build step is
the release-config compile smoke; validate.yml gates format validators). The
replacement gate is a built-ARTIFACT smoke (the "Smoke built plugins" step):
it nm-reads each built build/CLAP/*.clap and FAILS only if a Mach-O was
produced without its clap_entry C-ABI export (a real linkage regression),
warning-and-passing when no artifact is found (a path/setup miss must never
re-block the release). Static symbol read = NO execution, so it's headless-safe
— never use dlopen+init (loads GPU/Skia libs the headless runner lacks) or
re-run the hardware-dependent ctest suite.
-
Sandbox E2E macOS has a long cold C++ CLI build.
.github/workflows/sandbox-e2e.yml builds the pulp-cli target before
running the Python sandbox harness. On GitHub-hosted macos-latest, a cold
C++ CLI build can exceed 30 minutes and be reported as cancelled with
The operation was canceled plus orphaned clang processes, before pytest
starts. Treat that as a job timeout/build-cost issue, not a sandbox assertion
failure. The workflow uses timeout-minutes: 60 so the cold macOS build has
room to finish while still bounding genuinely wedged runs.
-
.github/workflows/header-self-contained.yml (pulp #2576) is a BLOCKING gate
for the "compiles on Apple Clang, breaks on Linux" transitive-include class
(e.g. uint32_t without #include <cstdint> — broke the v0.197.4 release).
It compiles each PR-changed public header standalone with Linux Clang via
tools/scripts/check_headers_selfcontained.py. Unlike the advisory IWYU gate
it only fails on a header that genuinely won't compile alone (no "unused
include" false positives), so it is safe to block on. Headers whose module
isn't in the GPU-off compile DB are skipped, not failed.
-
Windows FetchContent subbuilds need a short base dir. GitHub-hosted
Windows runners can still route MSBuild metadata through the legacy
260-character path limit. The wgpu prebuilt-populate subbuild has deeply
nested stamp paths, so build-windows/_deps/... can exceed MAX_PATH during
configure even before compilation. build.yml passes
-DFETCHCONTENT_BASE_DIR="$PWD/fc" only on Windows to keep dependency
subbuilds short while preserving the normal build-windows artifact/test
directory.
-
.github/workflows/watchdog-reaper.yml (pulp #2576) sweeps ALL open release
watchdog trackers daily and closes any whose version is released or superseded
— the existing watchdogs only auto-close inside a recent window, so historical
per-version trackers orphaned (334 had accumulated). It only matches the exact
auto-generated tracker titles and only closes objectively-resolved ones.
SHA-keyed release: stuck trackers carry no version in their title; the reaper
skips them without failing the sweep. Close those only after verifying the
affected SDK/plugin surface shipped the stranded commit.
-
Keep watchdog/issue-maintenance workflows on REST gh api calls. Avoid
gh issue list / gh pr * helpers in those paths because they can use the
shared GraphQL quota; a watchdog must not fail while reporting that the
watched workflow already recovered.
-
Pulp's docs site is deployed by .github/workflows/docs-deploy.yml. If
GitHub Pages is set to legacy main/docs builds, the generated Pages
checkout tries to clone the private planning submodule with the default
token and fails before docs deploys. Fix that at the Pages configuration
layer (build_type=workflow), not by weakening normal submodule checkout.
-
.github/workflows/sanitizers.yml runs broad ASan/UBSan matrices, but its
exclude regex may carry narrow sanitizer-lane skips for non-memory-safety
failures that remain covered by regular Build/Test. Keep those skips named
exactly to the failing CTest cases and leave comments explaining the
alternate coverage path; do not use sanitizer excludes to hide new targeted
coverage tests.
-
Flaky-lane retry (de-flaking). The ASan/UBSan/TSan lanes
(sanitizers.yml) and the coverage lanes (scripts/run_coverage.sh) run
ctest with --repeat until-pass:2. Timing-sensitive tests intermittently
fail under sanitizer slowdown (a different test each run — RenderLoop
coalescing, ImageView fill, etc.), and a single failure aborts ctest and
cascades into the diff-coverage gate (partial profile → 0% on the diff),
leaving every PR UNSTABLE even when the required gates pass. until-pass:2
retries a failed test once so a flake passes on retry while a genuine
failure still fails both attempts (no masking). This complements the per-lane
--exclude-regex flake lists by catching not-yet-listed flakes generically —
prefer it over growing the exclude list for transient timing flakes.
Current Build-and-Test routing
As of the 2026-05-20 classify gate, build.yml runs a cheap
classify job before allocating the native matrix. PRs that touch only
skip-safe docs/planning paths report the native aliases green without
running macOS/Linux/Windows builds; PRs that touch C++/Swift/CMake or
workflow inputs must run the native matrix. A code PR whose native build
is skipped is a CI bug.
workflow_dispatch defaults runner_provider to github-hosted, not
Namespace; do not dispatch with runner_provider=namespace. Linux/Windows
use GitHub-hosted runners by default. macOS routes through the self-hosted
pulp-build pool (pulp-m1-01, pulp-m1-02) via
PULP_LOCAL_MACOS_RUNS_ON_JSON; repository variables control any overflow.
Do not push empty commits just to churn queued macOS checks. Cancel
superseded SHAs, rebase or push only when a PR needs current main, and
wait unless a check has actually failed.
coverage.yml's macOS leg resolves its runs-on via
resolve_runs_on.py --deny-labels pulp-build,pulp-build-vm: a coverage
selector (repo var PULP_COVERAGE_MACOS_RUNS_ON_JSON or a dispatch input)
that names the shared gate pool fails the resolver fast rather than
letting a long advisory coverage run contend with the required macos
check. The dedicated coverage lane uses pulp-coverage-vm-macos; a bare
GitHub-hosted label (macos-15) is never denied. (Push-triggered coverage
on a busy main is designed to be superseded while queued — the
supersession-immune scheduled run, cron 17 */8 * * *, is the one that
produces the green full-matrix upload that clears the coverage-stale watchdog.)
The os-windows coverage leg is best-effort. The instrumented MSVC build +
~9k instrumented tests + llvm-cov over 1000+ objects exceeds the 150-min job
cap on GitHub-hosted windows-latest (it is ~1h on Linux/macOS), and the
staleness watchdog keys off a successful run, not per-OS Codecov flags — so a
red Windows leg would otherwise keep a healthy full run red forever. The
subtle trap (verified by canary): job-level continue-on-error does NOT
neutralize a timeout-minutes cancellation — a cancelled job still makes the
run conclude cancelled. It DOES neutralize a normal job failure. So the
coverage suite step self-terminates at an internal budget (135 min) below the
job cap, turning the would-be cancellation into a normal non-zero exit that
the job-level continue-on-error: matrix.os=='windows' then absorbs → the run
concludes success. Don't "simplify" this to bare continue-on-error; it will
silently stop closing the watchdog. And the watchdog that enforces the budget
must separate its steps with ;, NOT && — if the kill is &&-gated behind
a : > marker write (which can fail on a Windows RUNNER_TEMP backslash path),
a failed marker write short-circuits the chain and the suite is never killed,
so the job hits the 150-min cap and is cancelled anyway. The kill is
mandatory; the marker is best-effort (cleanup of any partial Cobertura also
triggers on a 143/137 signal-kill exit, not just the marker). Real os-windows
correctness bugs are still worth fixing (the ARG_MAX response-file +
vanished--object existence-filter + mass-drop guard in run_coverage.sh were
real); only the runtime/timeout is accepted as best-effort.
release-cli.yml needs Rust; the VM golden does not have it
The macOS build gate and release-cli.yml's darwin leg can share the Tart VM
golden (pulp-build-runner), but that golden is provisioned only for the C++
"Build and Test" lane — Xcode, homebrew cmake/ninja/node, baked Skia, and
no Rust toolchain (manifests/pulp.macos.toml [brew] has no rust/rustup).
build.yml stays green on it because it never builds the Rust build/pulp CLI
(experimental/pulp-rs, pulled into the CMake graph only when PULP_ENABLE_GPU && top-level — which release-cli's GPU-enabled configure trips). So
release-cli is the first VM workload that needs cargo, and on a fresh
golden its "Ensure a working Rust toolchain on PATH" step exhausts every probe
(PATH cargo, ~/.cargo/bin, ~/.rustup/toolchains/*) and fails the darwin
release leg — while the bare-metal studios stay green because they have rustup
from operator setup. That step now bootstraps rustup as a last resort
(curl … | sh -s -- -y --profile minimal --default-toolchain stable --no-modify-path, then front-loads rustfmt+clippy per
rust-toolchain.toml), so the lane is self-sufficient on any runner; the pin is
stable, so no reproducibility drift vs the studios. Before flipping
PULP_RELEASE_MACOS_RUNS_ON_JSON to a VM label, either that fallback must be in
place or the golden must bake rustup — otherwise the darwin publish leg
fails and the whole release never publishes (the release job needs the full
build-cli matrix). Steady-state fix is to bake rustup into the golden (tartci
manifests/pulp.macos.toml); the workflow fallback stays as the portability
belt for any un-baked runner.
release-cli.yml has a native Intel (darwin-x64) leg
release-cli.yml's macOS matrix ships TWO slices: darwin-arm64 (routed through
resolve-macos-runner) and darwin-x64 on GitHub-hosted macos-15-intel. The
Intel leg deliberately does not go through the resolver — the runs-on
ternary only routes matrix.os == 'macos-15', so macos-15-intel runs on that
fixed hosted label (no self-hosted/Namespace/VM routing, and Rust is
preinstalled so the toolchain-bootstrap above is a no-op there). It's a native
build (no cross-compile, no Rosetta) with a CMAKE_OSX_DEPLOYMENT_TARGET=13.0
floor and an arch-asserting smoke. macos-15-intel is the pinned stable hosted
x86_64 macOS image (supported ~through Aug 2027; macos-13 retired Dec 2025;
macos-26-intel also exists but is newer/less-baked) — when it EOLs, the
successor is cross-compiling on the arm64 pool
(planning/2026-07-10-intel-mac-cli-support.md). Do NOT route this leg to the
self-hosted studios (the no Intel on the studios discipline in
docs/guides/intel-support.md).
Advisory cross-lane workflow: macos-cross-advisory.yml
.github/workflows/macos-cross-advisory.yml is a path-scoped advisory
job for the Linux-hosted macOS arm64 cross lane (Phase 5 scaffolding,
see planning/2026-05-24-linux-hosted-macos-arm64-cross-lane.md). It
runs on ubuntu-latest, does not bootstrap osxcross, and does
not download a macOS SDK — it only confirms the Pulp-side cross
scaffolding (tools/cmake/toolchains/macos-arm64-osxcross.cmake,
tools/scripts/verify_macos_cross_artifacts.py, the
PULP_RUST_CLI_TARGET / PULP_MACOS_CROSS_ALLOW_MISSING_ICON_TOOLS /
OTOOL / INSTALL_NAME_TOOL hooks) stays wired and that the verifier
unit tests still pass. It is non-gating by design; do not promote it to
a required check until a self-hosted Linux runner with pinned osxcross
- private SDK is provisioned and the matching full-build job lands.
Advisory compile-gate: windows-midi2-gate
build.yml's windows-midi2-gate job (continue-on-error: true, NOT a
required check, NOT part of the build matrix) compile-verifies Pulp's
opt-in WinRT MIDI 2.0 backend (core/midi/platform/win/winrt_midi_device.cpp,
gated by PULP_HAS_WINRT_MIDI). That backend consumes the
Microsoft.Windows.Devices.Midi2 C++/WinRT projection, which ships
out-of-band with the Windows MIDI Services SDK — a GitHub-only NuGet, NOT in
the base Windows SDK, so no other lane can compile it. The job provisions it
through Microsoft's official vcpkg port microsoft-windows-devices-midi2
(it downloads the SDK NuGet, runs cppwinrt to generate the projection headers,
and exports the Microsoft::Windows::Devices::Midi2 CMake target). Pins +
rationale live in tools/ci/midi2/ (vcpkg.json + README.md). The default
Windows build never sets PULP_HAS_WINRT_MIDI, so this is purely additive.
Watch points: the port requires Windows SDK >= 10.0.26100.0 (windows-latest is
right at that floor), and the drafted backend's API surface may still drift
from the real winrt::Microsoft::Windows::Devices::Midi2 namespace.
Advisory compile-gate: windows-ble-gate
build.yml's windows-ble-gate job (continue-on-error: true, NOT a required
check, NOT part of the build matrix) compile-verifies Pulp's WinRT Bluetooth-LE
scan backend (core/midi/platform/win/ble_midi_win.cpp). Unlike
windows-midi2-gate, the BLE GATT / advertisement APIs
(Windows.Devices.Bluetooth*) ship in the BASE Windows SDK cppwinrt
projection, so this gate needs NO vcpkg / out-of-band NuGet provisioning — the
default Windows build already compiles the backend (it links WindowsApp +
runtimeobject). The job is a fast, isolated cmake --build … --target pulp-midi so a blind (macOS-authored) Windows TU gets a compile signal without
waiting on the full matrix. Watch point: the backend's
winrt::Windows::Devices::Bluetooth::Advertisement API surface is written
without a local Windows compiler, so signature drift surfaces here first.
Advisory build-gate: tracing-build.yml
.github/workflows/tracing-build.yml (advisory, ubuntu-latest, NOT a required
check) is the only lane that builds the opt-in Perfetto tracing configuration
(-DPULP_TRACING=ON). Every other lane builds the default OFF config, so a
break in the ON path — the Perfetto amalgamation fetch/compile in
tools/cmake/PulpTracing.cmake or the trace macros lighting up in
<pulp/runtime/trace.hpp> — would otherwise land unnoticed. It configures
PULP_TRACING=ON -DPULP_ENABLE_GPU=OFF (Release), builds pulp-test-tracing,
pulp-test-tracing-session, pulp-test-offline-tracing, and
pulp-test-ship-tracing-guard, and runs them. GPU/Skia stay OFF because the
tracing tests only need pulp::runtime + pulp::audio, keeping the lane fast
and hostable on a stock GitHub runner. Watch point: test_tracing.cpp's
"tracing is off by default" case asserts kTracingEnabled == false and is
designed to fail under ON, so the lane excludes exactly that case (Catch2
~"tracing is off by default"); the other suites are config-agnostic and must
fully pass under ON. runs-on is a hard-coded ubuntu-latest — never route it
to a self-hosted label or add it to branch protection.
Governance is declared, and mirrors LIVE state — not aspiration
.shipyard/config.toml declares Pulp's branch-protection posture so
shipyard governance {diff,apply} can reconcile it, and it is checked in
alongside .github/rulesets/main-protection.json. Two hard rules:
- The declared required checks must match the LIVE GitHub ruleset, not what
we wish we enforced. Live
main requires exactly two contexts: macos and
Enforce version & skill sync. Both the [governance] required_status_checks list in .shipyard/config.toml AND the
required_status_checks array in main-protection.json are pinned to that
two-context set. [branch_protection."main"] require_strict_status = true
mirrors the ruleset's strict_required_status_checks_policy. Before editing
either, run shipyard governance diff — a clean run prints
OK main: no changes; any other output means the checked-in intent has
drifted from live and you must reconcile (don't just re-declare).
- Never promote
linux/windows into the required set. They validate as
advisory GitHub-hosted lanes and deliberately do NOT gate merge. Making an
advisory lane blocking craters throughput — a saturated or flaky advisory
lane would wedge every merge — without adding signal. This is why the ruleset
was trimmed from four contexts to two: so nobody can "fix drift" by pushing a
config that flips the advisory lanes blocking. test_ruleset_drift_config.py
asserts the two-context required set; keep it and the two config surfaces in
lockstep.
PR Review Thread Hygiene
Before opening a follow-up PR or declaring a phase complete, sweep review
threads for the PRs touched by that phase:
gh api graphql -f query='
query($owner:String!, $repo:String!, $number:Int!) {
repository(owner:$owner, name:$repo) {
pullRequest(number:$number) {
reviewThreads(first:100) {
nodes {
isResolved
isOutdated
comments(first:20) {
nodes { url body author { login } }
}
}
}
}
}
}' -F owner=danielraffel -F repo=pulp -F number=<PR>
For every unresolved thread, either fix it in the follow-up branch or verify
that current main already fixed it. Leave a reply on the original thread with
the fixing commit/PR and the validation that proved it, so future sweeps can
distinguish addressed-but-unresolved GitHub state from actual pending work.
Shipping a PR: route through shipyard pr
When the user says any of: "push to main", "ship this", "ship it",
"we're done", "merge this", "push it", "run CI", "push a PR" —
run shipyard pr (not gh pr create + shipyard ship separately).
shipyard pr is the single orchestrator (Shipyard v0.19.1+; currently pinned
in tools/shipyard.toml). It:
- Calls
tools/scripts/skill_sync_check.py (resolved via Shipyard's
[validation] path-discovery, explicit in .shipyard/config.toml) and
hard-fails if a mapped skill path was touched without a SKILL.md update
or a Skill-Update: trailer.
- Calls
tools/scripts/version_bump_check.py --mode=apply to bump SDK,
Claude plugin, and marketplace versions consistently, honoring any
Version-Bump: trailers. This applies patch/minor/major bumps —
including patch, which every fix: PR gets (fixed in pulp #3626;
before that, patch bumps were silently skipped and fix:/feat: PRs
stranded at the gate, forcing a manual chore: bump versions commit — no
longer needed).
- Runs the no-build source-contract registry gate:
tools/import-validation/check-source-contracts.py --strict plus
tools/import-validation/test_source_contracts.py. This mirrors the
GitHub Versioning & Skill-Sync workflow and the local pre-push hook.
- Commits the bump (if any) as
chore: bump <surfaces>.
- Pushes the branch, creates the PR, and records Shipyard tracking state.
- Runs cross-platform validate + merge on green.
- Auto-release workflow (
.github/workflows/auto-release.yml) tags and
publishes binaries on merge. The full pipeline (tag → 5-platform build
→ sign + notarize → 11-asset publish) is documented end-to-end in
docs/guides/release-pipeline.md.
Keep that doc in sync when you touch any release workflow.
Auto-supersede behavior: when a new SDK tag is created, the workflow
cancels any in-flight release-cli.yml / sign-and-release.yml runs for
strictly-older SemVer tags and deletes their draft releases. This prevents
the 2026-05-15 v0.101.x saga from recurring (5 bumps each queueing a full
macos-15 darwin-arm64 build, latest waiting hours behind already-obsolete
builds). Opt out via vars.PULP_RELEASE_MODE=land-all (repo-wide) or
Release-Supersede: skip reason="..." trailer on the merging commit
(per-release). See #2076 + the release-draft-stuck-check.yml newest-SemVer
skip for the watchdog side of the cleanup.
Never run gh pr create + shipyard ship separately for a normal ship
cycle. Never invoke the two version/skill scripts by hand — shipyard pr
wires them together with the right flags.
After opening/merging a material PR, sweep its review comments. merge on green fires before the automated reviewers (Codex, and cubic on Shipyard)
finish, so a PR can land with unaddressed P1s. For any logic-bearing or
destructive-path PR, follow the pr-review-sweep skill: read
pulls/<n>/comments (via ghapp), verify each finding against the code, and
ship a follow-up with a test for anything confirmed. Docs-only PRs can skip it.
Direct gh pr create is an explicit emergency/manual bypass only. If the
user asks for that path, state the tracking gap up front: the PR may not
appear in Shipyard-managed state or the macOS GUI until it is reconciled or
re-shipped through Shipyard.
pulp pr is a Pulp-side wrapper that delegates to shipyard pr; both are
valid, agents should prefer shipyard pr for directness. Humans can opt out
of Shipyard for their own checkout with pulp config set pr.workflow github
or manual, but agents should not choose those workflows unless the user
explicitly asks for a manual/emergency bypass. pulp status reports the
effective workflow and whether its required local tool is installed.
Gotcha: shipyard-merged PRs don't reliably auto-close linked issues
Always verify an issue actually closed after a shipyard merge; close it
with gh issue close if it didn't. Don't trust closing-keyword auto-close
to do it for you here.
What was actually observed (#3299, twice): both Closes #3299 (no colon,
PR #3420) and Closes: #3299 (with colon, PR #3413) were correctly
recognized as closing keywords — GitHub registered the closing reference in
both cases (gh api graphql … closingIssuesReferences returned
[{number:3299}] for each). Yet neither shipyard-local[bot] merge
auto-closed the issue. So:
- The colon is a red herring for auto-close.
Closes #N and Closes: #N
both register a closing reference; GitHub accepts either after the keyword.
(An earlier version of this note wrongly blamed the colon — corrected here.)
- The real failure is that the app-performed merge didn't trigger GitHub's
issue-closing automation. Verify with
gh issue view <N> --json state
after merge and gh issue close <N> -r completed if still open.
Separately — and this is a real colon rule, but for a different system
— git trailer bypasses (Skill-Update:, Version-Bump:,
Release-Supersede:) are parsed by git interpret-trailers --parse, which
requires the trailer paragraph to be all Key: value lines. A
colon-less Closes #N dropped into that paragraph breaks the parse and
silently voids every bypass trailer in it. So keep issue references like
Closes #N in the body prose, above an all-colon trailer block
(Skill-Update: …, Co-Authored-By: …); inside the trailer block, only
Refs: #N (colon) is safe. Dry-run git log -1 --format='%(trailers)' to
confirm the bypass trailers still parse — that part is unchanged and true.
Backward compatibility: raw shipyard ship / shipyard run still work for
diagnostics, experimental branches, existing Shipyard-managed PRs, or when
shipyard pr itself is being debugged. Do not use them as the primary ship
path.
A ship must survive the session dying — arm GitHub auto-merge as a backstop
shipyard pr performs merge-on-green inside the CLI/worker process. If
that process dies before the merge — the cmux app relaunching under resource
starvation, or this Claude session running out of quota — the validated PR is
stranded unmerged and its ship-state record orphans. This is a recurring
real failure (521 orphaned records were reaped across the CI Macs on
2026-06-30). The merge must not depend on any interactive session staying alive.
Standing policy — after creating/validating a pulp PR, arm GitHub-native
auto-merge as a server-side backstop:
ghapp pr merge <PR> --auto --merge
GitHub then merges the moment required checks (macos + Enforce version & skill sync) go green, regardless of whether shipyard pr, cmux, or this
session survive. Use --merge (merge commit), never --squash: a squash folds
the chore: bump versions commit into the PR-title commit, which trips the
auto-release watchdog into a false "merged without bump." It is safe alongside
shipyard pr — whichever merges first wins; the other no-ops on already-merged.
(The Shipyard repo has no branch protection, so GitHub auto-merge is
unavailable there; the host-side queue janitor below covers it.)
Host-side backstop (both repos + orphan reaping): a launchd queue-tick on
each CI Mac (tartci scripts/shipyard_queue_tick.sh) periodically drives
in-flight ship-state to completion via shipyard's own fail-closed auto-merge
and reaps records whose PR GitHub reports merged/closed — independent of any
session. Reap a stale local pile by hand with shipyard ship-state list →
shipyard ship-state discard <pr> for each merged/closed PR (never discard an
OPEN one). Full design: pulp
planning/2026-06-30-ship-queue-resilience-design.md.
Stale-SHA merge race — DO NOT push onto a PR that's being shipped
The failure mode (observed 2026-05-29): Shipyard's can_merge()
validates the exact merge-candidate SHA and then merges that SHA. If a
developer/agent pushes new commits to the same branch while a shipyard ship
is mid-flight, Shipyard merges the already-validated older SHA and the new
commits are stranded on the branch — the PR squash-merges without them. In
the observed case a PR merged at its pre-fix SHA, dropping a whole review-fix
push; the fixes had to be re-landed via a fresh fast-follow PR.
expected_head_oid alone would NOT have caught it — the validated SHA was
GitHub's PR head at the merge instant; the new push only advanced the branch
ref moments later. The only true prevention is the merger re-checking the
branch tip immediately before merging and aborting if it advanced past the
validated SHA — that lives in Shipyard (tracked upstream as Shipyard issue
321). .
Operational rules (enforce these; they are the practical fix):
- Never push commits onto a PR that has an in-flight
shipyard ship or
armed auto-merge. Check shipyard ship-state list (shows PR/url/sha) and
GitHub's auto-merge state first. If a ship is running, let it finish.
- Land post-review fixes as a fresh PR off
main, not as a push onto the
already-green PR you just reviewed. (Review comes after green; the green PR
is exactly when auto-merge fires.)
- After any ship, verify the merge actually carried your latest commits.
Don't trust "merged" — confirm the merge SHA's tree contains your changes.
git fetch origin main then check a file you changed is present on
origin/main (e.g. git show origin/main:<path> | grep <marker>). If
you cross-check via gh api repos/<o>/<r>/pulls/<n> --jq .merge_commit_sha,
note that the SHA alone does NOT prove your push made it — that field
returns the PR's squash/merge commit on the base branch and exists for any
merged PR, including the stale-SHA case. The SHA is only useful if you
then inspect its tree/diff (git show <sha>:<path> | grep <marker> or
git show <sha> --stat | grep <expected-file>). If your commits are
missing, re-land them as a fresh PR immediately.
- A degraded/rate-limited
gh pr view ... headRefOid read can return a stale
SHA — corroborate branch state with git ls-remote / a real git fetch
before concluding the branch moved or was reset.
Gotcha: a long-lived integration branch drifts from main's whole-tree gates
The hotspot-size guard (tools/scripts/hotspot_size_guard.json frozen ceilings)
and the codecov-config gates (codecov.yml flags/components mirroring the live
core/* tree) are whole-tree, not diff-scoped — they assert a property of the
entire checkout, so they fail on any push to a branch whose tree violates them,
regardless of what that push changed. A long-lived develop/* integration branch
accumulates this kind of drift relative to main:
main may lower a frozen hotspot ceiling (e.g. after a comment-hygiene
cleanup) while the integration branch has legitimately grown the same file
(new test targets), so the merged tree exceeds the ceiling.- The integration branch may add a whole subsystem (
core/<new>/) and its
upload flag without registering the matching codecov.yml component, so the
flag/component-alignment and "every first-party file maps to a component" tests
fail.
Either way the integration branch cannot pass its own pre-push/CI gates, and
the failure surfaces on the first unrelated PR that tries to land on it (looks
like the PR's fault; it is not). Fix by reconciling the gate config to the
branch's real state in the landing PR: raise the hotspot ceiling to the file's
current LOC, and add the missing codecov flag+component (paths: core/<new>/**).
Run tools/scripts/gates.sh <integration-branch> locally first — the fast gates
are sub-second and catch all of this before a multi-minute ship cycle. Reconciling
config you didn't author trips skill-sync/version-bump (config maps to the ci
skill; examples/config-only diffs still need a Version-Bump: skip trailer under
a feat:/fix: title) — expect to add those trailers too.
Shipyard pin and behaviour notes
Pin bumps must go through shipyard pin bump --to vX.Y.Z, not a hand edit.
Shipyard v0.50.0+ is Rust-backed and macOS ships as an Apple-Silicon-only
signed/notarized .dmg, so the version and asset metadata must move together.
-
Pinned at v0.68.0+: a killed shipyard pr/ship worker no longer wedges
the PR. Before v0.68.0, a worker that died (crash, kill, launching a
second shipyard pr for the same PR) left its job stuck running in the
durable queue, and every later same-PR ship was refused with
SamePrShipRunning — recoverable only by hand-editing queue.json. At this
pin the queue auto-reaps a dead-worker job (heartbeat stale >180s) at
ship-submit time and on each drain pass. So if a same-PR ship is refused as
"already running" and nothing is actually live, just retry after ~180s.
Still: never run two shipyard pr for the same PR concurrently.
-
Installing/upgrading Shipyard on macOS uses the GitHub API, which is
rate-limited at 60/hr unauthenticated. A "No binary found for
shipyard-macos-arm64" error from install.sh / shipyard update is almost
always that rate limit, not a missing asset — re-run with GITHUB_TOKEN set
(./tools/install-shipyard.sh runs in an authenticated context).
-
Release SDKs are expected to include desktop WebView symbols
(pulp #695). .github/workflows/release-cli.yml now configures the
release build with -DPULP_BUILD_WEBVIEW=ON, installs Linux's
libgtk-3-dev + libwebkit2gtk-4.1-dev, and verifies the staged
pulp-view-core archive still contains WebViewPanel and
make_webview_embedded_resource_fetcher. If you touch the release
workflow or tools/scripts/release-cli-local.sh, preserve that
contract or WebView-using downstream SDK consumers will link-fail.
-
Release bodies are composed in release-cli.yml. The release job
runs tools/scripts/compose_release_notes.py for grouped Highlights and
calls GitHub's generated-notes API for the native "What's Changed" /
"Full Changelog" block before the Install section. CHANGELOG.md
remains Shipyard-owned via shipyard changelog regenerate; do not route
the Release body back through the deleted in-tree changelog generator.
-
Phase 8 CLI flip ships two CLI binaries. Release CLI jobs must
preserve Rust pulp as the user-facing binary and C++ pulp-cpp
as the fallthrough delegate in the same archive. Smoke both names:
pulp version --json for the Rust path, and at least one C++-owned
command through PULP_RS_CPP_BINARY=/path/to/pulp-cpp pulp ... or a
direct pulp-cpp ... invocation. Do not resurrect pulp-rs as the
shipped binary name.
-
macOS binary is signed + notarized (Shipyard v0.29.0). On
macOS 26.3+ XProtect skips the deep scan for notarized binaries,
cutting shipyard pr cold-start ~4-5x (from ~5-6s to ~1-1.5s).
No pulp-side action; transparent.
-
Format baseline diff is a plugin-only gate. Preserve
-DPULP_ENABLE_GPU=OFF in .github/workflows/format-baseline-diff.yml:
the self-hosted macOS runner may not have the pinned Skia archive, and this
workflow only needs the PulpEffect AU/VST3/CLAP bundles, not GPU examples
such as pulp-design-tool.
-
Build-and-Test workflow_dispatch is Shipyard PR validation. Preserve
-DPULP_ENABLE_GPU=OFF -DPULP_BUILD_EXAMPLES=OFF on the
workflow_dispatch configure path in .github/workflows/build.yml: the local
self-hosted macOS runner may not have the pinned Skia archive, and no-GPU
dispatches must not link example bundles that require the GPU plugin view
host. Pull-request validation also disables example bundles, while nightly /
release workflows own full example/product and GPU coverage.
When adding optional shell arguments in build.yml (for example macOS-only
-G Ninja), use bash arrays and expand them as "${args[@]}"; scalar
$args trips actionlint/shellcheck word-splitting checks.
-
Heartbeat line during long validation (Shipyard v0.29.0). A
20-minute lane now prints periodic progress instead of leaving a
silent terminal. Helpful when watching shipyard ship interactively.
-
Backend errors are surfaced under the summary table (Shipyard
v0.28.0). A bare ubuntu error ssh 12s row used to
give zero diagnostic signal — the captured stderr tail (bundle
upload failure, remote cmake apply failure, SSH transport
error, etc.) now prints below the table. Closes pulp #665's
diagnosis-blind-spot complaint.
-
Worktree-local .shipyard.local/ falls back to the primary
checkout (Shipyard v0.27.2). Pulp uses worktrees heavily for
parallel agent work; without this every new worktree had to
manually cp .shipyard.local/config.toml from the primary repo
before shipyard pr could see the SSH host config. Now it
inherits automatically.
-
Ship preflight runs BEFORE gh pr create (Shipyard v0.27.1).
Earlier the PR was opened first and ship aborted on unreachable
SSH backend, leaving stranded PRs with no validation (the
Apr 22 pattern that left several pulp PRs mid-flight). Now an
unreachable target fails fast and the PR is never opened.
-
Daemon tunnel supervisor (Shipyard v0.27.0). Tailscale Funnel
transients no longer kill the daemon — the supervisor restarts
the funnel on backoff. Periodic reconcile loop runs independently
of per-PR polls. Both apply to the macOS GUI's webhook delivery.
-
Long-running daemons keep accepting fresh subscribers (Shipyard
v0.26.0). The subscribe-replay path now uses blocking put()
instead of put_nowait(), so once the replay ring grows past 64
events the daemon no longer silently stops handing new IPC clients
their initial snapshot. This was the root cause of the macOS GUI
falling back to "polling" mode and showing no active PRs after
enough ship-state churn.
-
Daemon/CLI drift is now diagnosable over IPC (Shipyard v0.26.0).
IPC protocol is bumped to 2, daemon hello/status frames advertise
the daemon's own shipyard_version, and shipyard doctor flags a
daemon-vs-CLI version mismatch explicitly instead of leaving stale
subscribers to fail mysteriously.
-
Auto-PR titles and bodies use the feature commit (Shipyard v0.24.0 /
Shipyard #151). The orchestrator walks past the mechanical
chore: bump versions commit when composing the title/body, and scrubs
the Automated by shipyard pr. tool-branding text. Pulp PR #624 was the
canonical repro before the fix. Previously the auto-PR pointed at the
bump commit — generic and uninformative. Shipped PRs now read as
first-party.
-
Version-Bump: trailers are authoritative, not ceiling-raising
(Shipyard v0.25.0 / Shipyard #152). An author-declared
Version-Bump: <surface>=patch reason="..." is no longer silently
raised to minor when the conventional-commit heuristic on other
subjects classifies the diff as minor. This matches the pulp-side
behaviour in tools/scripts/version_bump_check.py at this pin.
-
shipyard ship-state list is served from the daemon via IPC when
shipyard daemon is running (Shipyard v0.25.0 / Shipyard #154). The
PyInstaller cold-start (~5-6s) is bypassed. Callers that tight-loop
over ship-state — the macOS GUI polls every 7s; pulp pr preflight
calls it indirectly — see a meaningful CPU saving. Nothing to do at
the pulp side; it's transparent.
SSH preflight (v0.20.0+ / Shipyard #106)
Exit codes are distinct:
| Exit | Meaning |
|---|
| 0 | Success |
| 1 | Validation failed |
| 2 | Configuration error |
| 3 | Backend unreachable (new; surfaces within 10s with classified reason) |
The unreachable-backend error names the failure class (auth / host_key /
network / timeout / configuration / unknown) and prints the last ssh stderr.
Flags:
--skip-target NAME — DELIBERATE lane skip (no probe runs). Use when
you know a target is irrelevant for this PR.--allow-unreachable-targets — proceed despite an unreachable backend.
Prints a loud ⚠︎ VALIDATION GAP: <target> skipped banner. Use only when
you genuinely cannot reach a backend and accept the validation gap.
Automation (crons, agents) should branch on exit code 3 specifically rather
than parsing error strings.
Tool selection: Shipyard (primary)
Shipyard is Pulp's primary CI tool. All merges, validations, and
ship cycles should use Shipyard. local_ci.py remains in the repo as
a fallback but is scheduled for removal after a 2-week observation
period (see danielraffel/pulp#120).
Prefer Shipyard for GitHub work — it dodges the personal gh rate
limit. Shipyard authenticates with its own GitHub App token
(higher rate budget), so PR-create / check-watching / merge aren't bound
by the developer's personal 5,000/hr gh budget — which is shared
across interactive gh, Shipyard, and every self-hosted runner
authenticating as the same user. Operational rules:
gh "The token in keyring is invalid" / gh auth status failing /
gh api 403 "rate limit exceeded" is almost always a rate-limit FALSE
POSITIVE, not a broken token. gh auth status validates by calling
the API; a rate-limited 403 gets mislabeled as an invalid token. Do
not gh auth refresh — it wastes a round-trip and fixes nothing.
First run gh api rate_limit --jq .resources. High core.remaining
(e.g. 4900/5000) with a 403 ⇒ the secondary/abuse limit (fires on
bursts / concurrent / expensive calls like gh api .../git/trees/ ...?recursive=1), which clears in ~1–5 min; a low core.remaining ⇒
the primary limit, resets at the top of the hour.git push/clone over HTTPS use a separate budget and keep working
when gh api 403s — so push branches with git, then let shipyard pr open the PR via its App token.shipyard pr's local target-reachability probe still shells out to
local gh auth status, so a rate-limited probe prints "Target 'mac'
(cloud) is unreachable / gh auth status failed". Push past it with
--allow-unreachable-targets (the required GitHub-Actions macos gate
still guards the merge). Do not --skip-target mac to dodge it —
mac is the only validation target, so skipping leaves none ("No
targets remain after --skip-target filtering").- Reduce burst:
git clone --depth 1 instead of recursive tree-API
dumps; space gh calls; never tight-loop gh (back off / schedule).
shipyard run
shipyard pr
shipyard ship --resume
shipyard ship --no-resume
shipyard ship-state list
shipyard ship-state show <pr>
shipyard ship-state discard <pr>
shipyard cleanup --ship-state --apply
shipyard run --targets windows --smoke
shipyard run --resume-from test
shipyard cloud run build <branch>
shipyard rescue <PR>
shipyard rescue <PR> --rerun-failed
shipyard rescue <PR> --rerun-failed --to local
shipyard ship --pr N --base main --adopt-head
