upstream-release-docs

name	upstream-release-docs
description	Analyze an upstream project's new release, verify changes against source code, and update documentation. Covers discovery, deep-dive into PRs/issues, docs audit, source-verified implementation, and review feedback handling.
argument-hint	<owner/repo> [tag]

Upstream Release Documentation

Analyze a new release of an upstream project and update the documentation site to reflect verified changes.

Core Principle

Verify everything against source code at the release tag. Never trust release notes, PR descriptions, PR review comments, or issue descriptions at face value. Always check the actual source code.

How you reach that source depends on what the caller gave you:

• A local clone at the tag (an automated caller usually provides one, and names its path in the invocation): read it with the Read, Grep, and Glob tools. Prefer this; it costs no API quota. See Reading the upstream clone for the constraints.

• No clone: fetch individual files from the API, and decode the base64 response.

  gh api repos/<OWNER>/<REPO>/contents/<PATH>?ref=<TAG>
  

Claims from any human-written source (release notes, PR bodies, review comments) may be inaccurate, outdated, or aspirational. The source code at the tag is the single source of truth.

Input

Parse the argument to extract:

• <OWNER>/<REPO>: the upstream repository (e.g., stacklok/toolhive-registry-server)
• <TAG> (optional): the release tag (e.g., v0.6.3). If omitted, fetch the latest release.

Output conventions

Any output that may be rendered as a GitHub comment, PR body, or Markdown file in the docs-website repo (progress narration, SUMMARY.md, GAPS.md, commit messages, PR descriptions) must fully qualify references to the upstream repo. GitHub auto-links bare #NNN relative to the repo the text lives in, so a bare #777 in a docs-website comment links to docs-website PR #777, not the upstream PR.

• Refer to upstream PRs and issues as <OWNER>/<REPO>#NNN (e.g., stacklok/toolhive#777), never bare #NNN.
• When the release notes body contains bare #NNN, expand them to <OWNER>/<REPO>#NNN before echoing them back.
• Full PR/issue URLs are also fine.

Execution modes

This skill runs in one of two modes. The caller signals the mode; absent an explicit unattended signal, assume interactive. Never infer unattended mode from surrounding context.

Interactive (default): a human is present. At decision points that need product context you cannot derive from source (Phase 2 step 4), ask the user. Never write the GAPS.md, SUMMARY.md, NO_CHANGES.md, or REVIEWERS.json artifacts described below in interactive mode; surface that information conversationally instead. Those files are machine-readable handoff artifacts for an automated caller, and writing them during a local run just litters the repo root.

Unattended: no interactive user, for example a CI workflow that invokes /upstream-release-docs ... in unattended mode. Never ask clarifying questions; proceed best-effort at every decision point, and route anything genuinely unresolvable into the artifacts below.

Unattended decision-point behavior

When Phase 2 step 4 would normally ask the user for a major feature's "why", instead:

1. Fetch the PR body and author with gh pr view <NUMBER> --repo <OWNER>/<REPO> --json title,body,author. The PR body usually carries the "why" the author wrote at open time: motivation, intended consumers, design decisions.
2. If the PR body references linked issues ("Closes #N", "Fixes #N", "Refs #N"), fetch the likely-context-bearing ones with gh issue view <N> --repo <OWNER>/<REPO>.
3. Write the "why"/consumer narrative directly into the relevant page using what you learned, translated into reader-facing language rather than the PR's engineering shorthand. This is best-effort; reviewers refine it later.
4. Defer to GAPS.md only when the rationale demonstrably cannot be derived from available sources: the PR points to an internal design doc you cannot access, multiple plausible consumer narratives exist and choosing one would mislead readers, or a release timeline or commitment needs product-team confirmation.

Reading the upstream clone

When a caller provides a local clone of the upstream repo at the release tag, read it with the Read, Grep, and Glob tools. Prefer it over gh api contents?ref=<tag>: it is already at the tag and costs no API quota.

Do not reach for Bash to explore it. An automated caller typically clones to a scratch directory outside the session's working directory, so ls, find, and grep run through Bash are refused there, and git -C <path> is refused everywhere because it can execute untrusted hooks from the target repo. Read, Grep, and Glob have no such restriction and are the right tools regardless.

Two consequences worth internalizing, because working around them by retrying Bash variants wastes turns and never succeeds:

• Don't try to run git log on the clone. When the caller supplies .release-meta.json, its commits array is the release range's commit list. Otherwise derive the range from the release notes and gh api.
• Don't append 2>&1 or chain with && on any Bash call. That splits the command into parts that no longer match the caller's tool allowlist, so the call is denied even when the underlying command is permitted.

Artifacts (unattended mode only, written at repo root)

These files are read by the automated caller and spliced into the PR body. The filenames and the repo-root location are a contract with the caller; do not rename or relocate them.

GAPS.md - only if you genuinely need to defer (see above). An empty GAPS.md is worse than none; do not create it if every feature's "why" was resolvable from available sources.

• Include only content gaps a human reviewer must fill. Exclude environment or sandbox limitations (for example, "couldn't run npm build"); the PR's CI handles those. Exclude "documented for clarity, not a gap" commentary.

• Each entry must @-mention the PR author, skipping bot authors (renovate[bot], github-actions[bot], stacklokbot).

• Each entry must include a paste-ready "Helper prompt for local Claude" referencing the specific file(s), the PR number for context, and the narrow piece of information the human must supply or confirm.

  Entry format:

  ### <Feature name> (PR <OWNER>/<REPO>#123 by @alice)
  
  <One paragraph: what's missing and why it couldn't be resolved from available sources.>
  
  **File(s):** path/to/file.mdx
  
  **Helper prompt for local Claude:**
  
  > <Self-contained, paste-ready prompt referencing the file(s), PR number, and the narrow piece of info needed.>
  

REVIEWERS.json - written whenever .release-meta.json exists at the repo root and lists contributors. This one is JSON, not markdown, because the workflow parses it with jq to decide who gets a review request.

Read .release-meta.json first (the caller writes it before invoking you):

{
  "repo": "stacklok/toolhive",
  "prev_tag": "v0.42.0",
  "new_tag": "v0.43.0",
  "owner": "jerm-dro",
  "owner_source": "merged release PR stacklok/toolhive#6333",
  "contributors": ["alice", "bob", "carol"],
  "commits": [
    {
      "sha": "8343851e9f06c0d67e315eb6aa4e9371d6ef76cf",
      "subject": "Push skills unsigned until keyless signing lands (#6334)",
      "author": "alice"
    }
  ]
}

commits is the commit list for the release range: use it instead of trying to run git log against the upstream clone, which is refused (see Reading the upstream clone). It is also what tells you which commits belong to which contributor for the classification below.

When commits_truncated is true, the range exceeded what the caller could fetch in one request, so both commits and contributors are partial. Treat the release notes as the authoritative list of changes for that run, and say in SUMMARY.md that the commit list was truncated so a reviewer knows the classification may have missed someone.

Classify every login in contributors as docs-facing or not, and write REVIEWERS.json at the repo root:

{
  "contributors": [
    {
      "login": "alice",
      "docs_facing": true,
      "docs_facing_shas": ["8343851e9f06c0d67e315eb6aa4e9371d6ef76cf"],
      "note": "Confirm the ai-plugin timeout flag section matches what you shipped."
    },
    {
      "login": "bob",
      "docs_facing": false,
      "reason": "CI workflow and test-fixture changes only"
    }
  ]
}

• docs_facing: true means at least one of this person's commits in the release range changed something a reader of the docs can observe: a CLI flag or subcommand, a CRD or config field, an API route, a default, an error message, a user-visible behavior, or anything you documented or corrected in this run. When you are unsure, classify as true. A needless review request is a minor annoyance; a missing one means a wrong page ships.
• docs_facing_shas (docs-facing only): every full commit SHA from commits that made the contributor docs-facing. If GitHub cannot request the contributor as a reviewer, the workflow uses all of these SHAs to find the human mergers of the relevant upstream PRs. Include only commits you verified as reader-visible; do not list an unrelated commit merely because the same contributor authored it.
• docs_facing: false is for changes with no reader-visible surface: CI and build plumbing, dependency bumps, tests and fixtures, internal refactors, lint fixes, comment-only edits. Base this on the actual diff you read in Phase 2, not on the commit message. A commit titled "refactor" that changes a default value is docs-facing.
• note (docs-facing only): one short sentence naming the specific thing that person should check, in their terms. Not a summary of the release, and not a restatement of their PR title. Skip the note rather than pad it.
• reason (non-docs-facing only): a short phrase naming what their changes actually were. This keeps the classification auditable when the run artifact is inspected; the workflow counts these contributors without rendering their names or reasons in the PR body.
• Include the owner in the list with an honest classification. The workflow requests a review from the owner either way, so classifying the owner false costs nothing and keeps the classification truthful.
• Bot logins are already filtered out of contributors; if one appears anyway, omit it.

Write this file even on a NO_CHANGES.md run: "no doc-relevant changes for this release" means every contributor is non-docs-facing, which is exactly the case where suppressing review requests matters most. Classify them all false with a reason.

The consequence of skipping this file is concrete: the workflow falls back to requesting a review from every contributor, which is the noisy committee behavior this artifact exists to prevent.

NO_CHANGES.md - if the Phase 3 impact map is empty (no doc-relevant changes for this release), write this at repo root with a one-line explanation and stop. Do not hand-edit any file.

SUMMARY.md - before the final commit, write a concise list of the hand-written doc changes you made. The caller surfaces it as the PR's "Summary of changes" so reviewers see what shipped without reading the diff. Skip it only when you wrote NO_CHANGES.md or made zero hand-edits. Keep it to 3-8 bullets, each formatted as one of:

• Added <what> at <path> - new pages/sections
• Updated <what> in <path> - meaningful prose edits
• Swept <what> across N files in <area> - repo-wide renames, apiVersion bumps, etc.
• Removed <what> from <path> - deletions

Describe one logical change as one bullet (don't enumerate each file in a sweep), and exclude auto-generated reference files the caller's refresh step updates: describe only your hand-written edits.

Phase 1: Discovery

1. Fetch the release:

   gh release view <TAG> --repo <OWNER>/<REPO> --json tagName,name,body,publishedAt
   

   If no tag was provided:

   gh release view --repo <OWNER>/<REPO> --json tagName,name,body,publishedAt
   

2. Extract PR numbers from the release notes body (look for #NNN patterns or full PR URLs). When referring to these PRs in any subsequent output, always fully qualify them as <OWNER>/<REPO>#NNN (see "Output conventions" above).

3. Categorize changes into:

   • New features: entirely new capabilities
   • Changed defaults: existing behavior that now works differently
   • New/changed configuration: new config options, env vars, CLI flags
   • Deprecations: features or options being phased out
   • API changes: new/modified endpoints, request/response formats
   • Bug fixes: corrections that may affect documented behavior
   • Internal/infra: CI, dependencies, refactoring (usually no docs impact)

4. Log the categorized summary for transparency, then proceed to Phase 2.

Phase 2: Deep Dive

For each PR identified in Phase 1 (skip internal/infra unless user requests):

1. Fetch PR details:

   gh pr view <NUMBER> --repo <OWNER>/<REPO> --json title,body,files
   

2. Fetch linked issues if referenced:

   gh issue view <NUMBER> --repo <OWNER>/<REPO> --json title,body
   

3. Understand the "why": for new features, look beyond the code to understand motivation and intended usage:

   • Check linked issues for user stories, acceptance criteria, and "definition of done"
   • Follow references to RFCs, design docs, or PRDs linked from issues or PR descriptions
   • Identify the intended user workflow: who uses this, why, and what happens after?
   • Map the full lifecycle: if the feature has a publish/produce side, actively search the source code for the consume/discover side. Check CLIs, client libraries, and related repositories. If consumption tooling doesn't exist in this release, note that explicitly. This gap must be documented rather than silently omitted.
   • If the "why" and consumption story aren't clear from any source, flag this gap. Documentation that only covers the API surface without explaining purpose or workflow is incomplete

4. For major new features: when a change introduces an entirely new capability (not just a config change or incremental addition), the "why" and consumer workflow often cannot be derived from source code alone. In this case, ask the user for additional context before writing documentation:

   • Request user stories, PRDs, RFCs, or design documents that explain the motivation and intended usage
   • Ask who the target users are and what workflow they're expected to follow
   • Ask how consumers are expected to discover and use the feature (CLI, IDE extension, API, etc.)
   • Do not attempt to fabricate the "why" or consumer story from code structure alone. This produces documentation that covers the "what" and "how" but misses the perspective and voice that only comes from understanding the product intent
   • Incremental changes (new config options, default changes, annotation additions) can proceed without this step
   • In unattended mode, do not ask. Follow the unattended decision-point behavior in Execution modes: derive the "why" from the PR body and linked issues, write it best-effort, and defer to GAPS.md only when it is genuinely underivable.

5. Check related repositories: components often span multiple repos. For example, a server's CRD/operator may live in a different repo than the server itself. When a release changes config structures, API surfaces, or deployment models, check whether related repos (operators, CLIs, client libraries) have also released changes that affect the documentation. Ask the user which repos are related if unclear (in unattended mode, infer related repos from the release notes and proceed best-effort).

6. Read the actual source code at the release tag to verify every claim made in the PR description. Use the local clone when the caller provided one, per Core Principle; otherwise fetch the file from the API and decode the base64 response:

   gh api repos/<OWNER>/<REPO>/contents/<PATH>?ref=<TAG>
   

7. Note discrepancies between PR descriptions and actual code. Trust the code.

   A PR body describes the moment it was written, not the release. This is a distinct failure from a PR body simply being wrong: the body was accurate when opened, then later commits, review changes, or a follow-up PR in the same release moved past it. stacklok/docs-website#1037 documented a field as "not wired up" because an upstream PR note said so; by the time the release shipped it was wired up, and the note was months stale. A caveat, limitation, or "not yet supported" claim in a PR body is the single least trustworthy kind of statement you will read, because it is a claim about absence, and absence is exactly what a later commit silently fixes. Before writing any limitation into the docs, verify in source at the tag that the limitation still holds. If you cannot confirm it, leave it out rather than repeating it.

8. Deep-verify behavioral claims: these are the most common source of documentation inaccuracy. For each feature, verify not just struct definitions but actual runtime behavior:

   • API routes: Check the actual route registration code (e.g., r.Get, r.Post, r.Delete), not just handler names. Docs often claim endpoints exist at paths where no handler is registered.
   • Required fields: Check validation code (e.g., if field == "" checks), not just struct definitions. A field present in a struct is not necessarily required; only fields checked in validation logic are enforced.
   • Default values: Check the actual defaulting code or fallback logic, not comments or struct tags. For example, "defaults to main" may actually mean "defaults to the remote's HEAD" in practice.
   • Precedence rules: Read the actual if/else chain. For example, if commit != "" { ... } else if branch != "" { ... } else if tag != "" { ... } means commit > branch > tag, not commit > tag > branch.
   • Delete/cleanup behavior: Check whether the code reassigns pointers, cascades deletes, or leaves orphans. Delete behavior is frequently mis-documented.
   • Query parameters: Check whether parsed parameters are actually wired to the service layer and database queries. Parameters can be parsed from the URL but silently ignored if no service option or SQL filter exists for them.
   • Removed surface: a release takes things away as well as adding them, and a removal is invisible in a release-notes list of features. Diff the previous tag against the new one for deleted user-facing symbols (CRD fields, status subresources, flags, subcommands, routes, config keys) and grep the docs for every one you find. stacklok/docs-website#1086 documented status.referencingWorkloads in the same release that removed it, because the PR that added the surrounding feature described only what it added. Treat every deletion in the diff as a docs defect until you have confirmed no page mentions it.
   • Command and subcommand strings: never assemble a command from a pattern. Every command, subcommand, and flag you write into a page must be traced to its actual registration in source (AddCommand, flag registration, route table). stacklok/docs-website#1095 documented thv ai-plugin sync and thv ai-plugin upgrade because sibling commands had them; neither existed. If you cannot point at the line that registers it, it does not go in the docs.
   • Containment/authorization direction: When documenting subset/superset checks, verify which argument is the caller and which is the resource. Getting the direction wrong produces examples that show the opposite of actual behavior.

9. Identify:

   • Auto-generated content: files generated from upstream (OpenAPI specs, CLI reference docs, JSON schemas). Do not manually edit these; flag them for automated update instead. However, auto-generated reference docs (e.g., API endpoints from a swagger spec) do not replace the need for conceptual explanations, guide content, or cross-references in existing pages. A new feature with auto-generated API docs still needs: (1) a conceptual explanation of what it is and why it exists, (2) mentions and cross-references in related existing pages (intro pages, feature lists, related guides), and (3) guide content if the feature has non-trivial workflows. Only skip creating a duplicate API reference page: never skip the surrounding documentation.
   • Hidden, flag-gated, and experimental features: see the readiness gate in step 11 below. This is the single most common over-documentation failure in this workflow, and the release notes will not warn you about it.

10. Inventory the new public surface. As you read the source and the regenerated reference assets, list every new or changed user-facing symbol the release introduces: CRD/struct fields, enum values, CLI flags and subcommands, env vars, config keys, and API routes. Most are already enumerated in the auto-synced reference assets (CLI .md, CRD *.schema.json, Swagger YAML) and the diff, so this is mostly transcription, not discovery. This list is the checklist the completeness pass in Phase 5 verifies against. It is the difference between documentation that is accurate and documentation that is complete: a release can ship five new config fields, and a section that explains one of them correctly passes every accuracy check while silently omitting the other four.

11. Apply the readiness gate before documenting anything. A release ships code; it does not thereby declare that code ready for readers. Three separate releases leaked experimental features into published docs (skills sync/upgrade and the lockfile, Sigstore signing for skills, desktop theme customization), and each cost a reviewer round-trip to remove. The pattern is always the same: the PR is well-written and enthusiastic, the code is real and complete, and nothing in either says "not yet". Readiness lives in the code, not the narrative.

    For every feature you are about to document, check the source at the tag for these signals:

    • CLI: Hidden: true on the cobra.Command, or registration behind a build tag or an if on an env var / config flag. A hidden command is not user-facing no matter how complete it is.
    • Feature flags and gates: a config key, env var, or CRD field that defaults to off/false and guards the code path. Search for the feature's entry point and read what has to be true for it to execute.