release-artifact-verification

# A Green Release Pipeline Is Not a Shipped Artifact — Go Look For the Artifact Promoted from the always-loaded `release-artifact-verification.md` portfolio rule, whose stub keeps the gate lines. A release "succeeded" when the thing you released is **downloadable by someone else**. Every intermediate signal — a green workflow badge, a published GitHub release, a version tag, a passing check — is a claim about *mechanics*, and each can be true while nothing shipped. Verify the artifact at its public address, from outside your own tree, in the same breath you announce the release. > Canonical break (2026-08, pal-mcp-server): `publish-pypi` had failed on > **every** release since it was added — 10.2.2, 10.3.0, 10.4.0, 10.4.1, > 10.4.2 — and the package had never existed on PyPI. A month of releases, each > one green where anyone would look. ## When to Use This Skill | Use this skill when... | Use something else when... | |---|---| | About to say "released" or "shipped" | A release-please PR is stuck or conflicting → `git-plugin:release-please-pr-workflow` | | Wiring or reviewing a publish job | A red check needs its error extracted → `github-actions-inspection` | | Checking an installed version by hand | Choosing Dockerfile structure → `container-plugin:container-development` | | Adding a container build that pushes to a registry | | ## 1. A release is not atomic — the failing job is not the one you watch release-please (and most release pipelines) cuts the tag and the GitHub release in **one** job and publishes artifacts in **another**. When the publish job fails, the tag exists, the GitHub release page renders perfectly, the changelog is updated, and the notification says a release happened. The only red is one job inside a run nobody opens. - **Check**, always, per artifact class: ```sh curl -s -o /dev/null -w '%{http_code}\n' https://pypi.org/pypi/<pkg>/json # 200 curl -s https://pypi.org/pypi/<pkg>/json | jq -r .info.version # the version you cut gh api users/<owner>/packages/container/<pkg>/versions --jq '.[0].metadata.container.tags' npm view <pkg> version ``` - **Look at the run's jobs, not its conclusion**, when a release "succeeds": ```sh gh api repos/<o>/<r>/actions/runs/<id>/jobs --jq '.jobs[] | {name, conclusion}' ``` - **Fix the design, not just the release**: make the publish job's failure visible — gate the release on it, or add a post-release check that asserts the version resolves publicly. A pipeline that can half-succeed silently will. ## 2. Verify from outside the source tree, or you may be reading the source An install check run **inside the project directory** can resolve the local working copy instead of the artifact you just published, and report a plausible-but-wrong version. `python -c` puts the cwd on `sys.path`, so a stale `*.egg-info/` or a source package in the repo root shadows the installed distribution. ```sh # Wrong — cwd is the repo root; reads the stale local egg-info $ .venv/bin/python -c "import importlib.metadata as m; print(m.version('pkg'))" 10.4.1 # ← the version from ./pkg.egg-info/PKG-INFO # Right — neutral cwd, and confirm the dist-info the venv actually holds $ cd /tmp && /path/.venv/bin/python -c "import importlib.metadata as m; print(m.version('pkg'))" 10.4.3 $ ls /path/.venv/lib/python*/site-packages | grep 'pkg.*dist-info' pkg-10.4.3.dist-info ``` Generalises past Python: `node_modules` resolution walks upward, `cargo` prefers a path override, `go` honours a `replace` directive. **The rule is to verify in a directory that has no relationship to the source**, and to confirm against the installed metadata (`dist-info`, `package.json` in `node_modules`) rather than an import that could have come from anywhere. The failure is silent and reads as a real result — same family as `agent-patterns-plugin:probe-input-integrity`: a broken harness and a genuine finding produce identical-looking output. ## 3. A check that never ran reports `pass` Path-filtered CI checks (`file-patterns`, `paths:`) complete in **4–6 seconds** having analysed nothing, and report success. A sibling PR's green tick is not a control unless it actually executed. Full treatment in `github-actions-plugin:ai-review-max-turns` § *"a check that never ran looks exactly like a pass"*; noted here because it is the third way a release looks verified and isn't. ## 4. A build job that never *runs* the image publishes a broken one, green A `docker build` that succeeds proves the layers assembled, not that anything inside can start. A pipeline that builds and pushes without ever executing the entrypoint will publish an image whose every import fails, report success, and deploy it. > Example: a dependency-bot PR moves a multi-stage image's runtime stage to > `python:3.14-alpine` while the builder stage stays on 3.12. The venv copied > between stages keeps its packages in `lib/python3.12/`, so the 3.14 > interpreter drops `/app/.venv` from `sys.path` entirely. Every service built > from it crash-loops for days, hundreds of restarts each, while build, push, > release, and every CI check stay green — nothing in the pipeline ever ran the > image. - **Gate the push on a smoke step, not the other way round.** Build with `load: true`, run the entrypoint, *then* push. A broken image never reaches the registry, so there is nothing to roll back. - **Assert the environment, not just an import.** `import <pkg>` alone can pass for the wrong reason; assert the venv is actually on the path: `assert [p for p in sys.path if "/app/.venv/" in p]`. **`docker run` needs `-i` or the check is vacuous.** Without it docker does not forward stdin, so a heredoc-fed `docker run --rm IMG python -` reads an **empty script** and exits **0 having asserted nothing** — the gate then passes on every image, including a deliberately broken one: ``` docker run --rm -i "$IMAGE" /app/.venv/bin/python - <<'EOF' ``` This is the same family as §2 and `agent-patterns-plugin:probe-input-integrity` § *an inert stub means the real tool ran*: the harness is green while measuring nothing. It was caught only by **running the finished gate against a known-bad image** and requiring it to fail — do that before trusting any gate you just wrote. ## The litmus test Before saying "released", answer: **"What URL would a stranger fetch to get this, and have I fetched it?"** If the answer cites a workflow badge, a tag, or a GitHub release page, you have verified the mechanics and not the outcome. ## Related - `github-actions-plugin:multirepo-ci-cd` — a re-run replays the *stale* workflow, so re-running a failed publish after merging its fix does not test the fix - `github-actions-plugin:ai-review-max-turns` — the skipped-check-reports-pass trap in full - `migration-patterns-plugin:tool-migration-cutover` — the same law before the fact: removal is gated on a positive *operational* signal, not on config being present - `git-plugin:git-merge-hazards` — "exit 0 is a claim about mechanics, not content", of which this skill is the release-time instance