release-test-scope

<!-- SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. --> <!-- SPDX-License-Identifier: Apache-2.0 --> # Release Test Scope Generate one evidence-backed local artifact for QA, Engineering, PM, TPM, and developer review. It combines the feature list, source evidence, validation paths, risks, decisions, and draft release notes. Do not treat every PR as a separate feature. ## Easy invocation Prefer this complete prompt: ```text Use $release-test-scope for version <version>, previous tag <previous-tag>, release ref <release-ref>, and main verification ref <main-ref>. Generate the local shared release test scope. Do not create or update a GitHub issue. ``` Example: ```text Use $release-test-scope for version 0.6.0, previous tag 0.5.0, release ref origin/release/0.6, and main verification ref origin/main. Generate the local shared release test scope. Do not create or update a GitHub issue. ``` The normal invocation needs only those four values. Ask for input only if a boundary is missing or ambiguous. ## Required inputs - `version`: proposed release version. - `previous_ref`: previous released SemVer tag. - `release_ref`: release branch or proposed cut commit. - `main_ref`: main snapshot used only to trace release-derived work. Resolve all refs to immutable 40-character SHAs and record them in the artifact. Optional inputs include manual additions, agreed exclusions, and a GitHub issue to refresh. ## Operating invariants - The only automatic candidate-producing range is `previous_ref..release_ref`. - Main is used only to trace or verify release-derived commits or PRs. Never enumerate unrelated main development as release candidates. - Release reachability, not presence on main, determines whether work shipped. - Substantive documentation in the release range that is published through the release snapshot's Fern navigation is the default automatic inclusion gate. - Exclude `docs/about/release-notes/**` from automatic candidate qualification when it only summarizes the same release, but always inspect its content and history as release-note coverage evidence. - Group related implementation, documentation, fix, and polish PRs into one user-visible capability. - Do not invent behavior, Studio navigation, CLI commands, expected results, owners, or documentation links. - Mark incomplete evidence `Needs clarification` and name the missing evidence. - Keep manual additions and exclusions explicit; never silently discard them. - Generate locally first. Creating or updating a GitHub issue requires explicit user authorization. - Never trigger a release, modify a tag, publish release notes, or alter release branches. ## Required capabilities - Read access to repository history. - Local Git history with the three fetched refs is sufficient to generate the release boundary, commit evidence, documentation state, and main-forwarding checks. - Network access and authenticated `gh` access only enrich the artifact with PR metadata that is not stored in Git, such as owners, labels, review context, and linked issues. - GitHub write access only when the user explicitly authorizes issue creation or update. If `gh auth status` or API access fails, continue from local Git unless the user explicitly required GitHub enrichment. Mark PR metadata unavailable, use commit SHAs instead of guessed PR metadata or owners, and identify only the affected enrichment fields as incomplete. ## Workflow ### 1. Validate boundaries and topology Run the deterministic boundary checks before summarizing anything. Prefer the checked-in helper because it records the raw evidence files for review: ```bash bash .agents/skills/release-test-scope/references/collect-git-evidence.sh \ "${previous_ref}" "${release_ref}" "${main_ref}" \ "release-artifacts/<version>/evidence" ``` If the helper is unavailable, run the equivalent commands directly: ```bash git rev-parse --verify "${previous_ref}^{commit}" git rev-parse --verify "${release_ref}^{commit}" git rev-parse --verify "${main_ref}^{commit}" git merge-base --is-ancestor "${previous_ref}" "${release_ref}" git merge-base --is-ancestor "${previous_ref}" "${main_ref}" git rev-parse "${previous_ref}^{commit}" git rev-parse "${release_ref}^{commit}" git rev-parse "${main_ref}^{commit}" git merge-base "${release_ref}" "${main_ref}" ``` Keep this command output as mechanical evidence. Do not ask the LLM to infer ref topology from prose or PR titles. Stop if a ref does not resolve. If the previous release is not an ancestor of either snapshot, report the topology and ask for corrected boundaries. Do not substitute a triple-dot range or merge base silently. Run `gh auth status` before GitHub enrichment or issue operations. Authentication failure does not invalidate the local Git comparison; apply the fallback under **Required capabilities**. ### 2. Collect the release range Use only: ```text previous_ref..release_ref ``` 1. Enumerate all commits in that range. 2. Preserve commit SHAs as the baseline source evidence. When GitHub is available, also associate each commit with PRs through GitHub commit-to-PR metadata. Do not rely only on commit-message PR numbers. 3. When GitHub is available, fetch PR number, title, URL, body, labels, author, base branch, merge commit, linked issues, and changed files. Otherwise retain commit-backed evidence and mark those fields unavailable. 4. Record release commits that have no associated PR. 5. Identify PRs with changed paths under `docs/**`. 6. Exclude release-note-only changes from automatic qualification. Collect the authoritative documentation delta with: ```bash git diff --name-status "${previous_ref}..${release_ref}" -- docs/ git diff --find-renames "${previous_ref}..${release_ref}" -- docs/ git diff --find-renames "${previous_ref}..${release_ref}" -- \ docs/about/release-notes/ git log --format='%H%x09%cs%x09%s' \ "${previous_ref}..${release_ref}" -- docs/about/release-notes/ git show "${release_ref}:docs/fern/versions/latest.yml" ``` The release-note diff is coverage evidence, not a candidate source. Compare every user-visible release capability with the actual note content. A change being reachable from the release branch or represented by a release-note commit does not prove that the capability itself is described. A documentation source file counts as published only when its page is reachable from the release snapshot's `docs/fern/versions/latest.yml`; record gated pages as missing publication evidence. ### 3. Verify release-derived work on main For each release-range commit or PR, classify: - `Exact commit on main` - `Forwarded by PR` - `Included by aggregate forward-merge` - `Not found on main` - `Different on main` Use evidence in this order: 1. Commit ancestry at the immutable main SHA. 2. Commit-to-PR associations for release and merge commits. 3. Explicit release-PR references in a forward-merge PR. 4. Membership in an aggregate release-to-main merge. 5. Patch equivalence plus matching documentation paths and behavior. Do not match similar titles alone. A forward-merge PR is supporting evidence, not a separate feature. If the release snapshot is an ancestor of main, classify its release-derived items as exact and do not enumerate main-only PRs. Use a cherry-marked comparison to expose patch-equivalent changes that different commit IDs would otherwise hide: ```bash git log --left-right --cherry-mark --no-merges \ --format='%m%x09%H%x09%cs%x09%s' \ "${release_ref}...${main_ref}" ``` `=` establishes patch equivalence, not release-note coverage. Keep user-visible equivalence groups in the evidence ledger until their documentation and release-note disposition are checked. For aggregate forward merges, inspect the commits introduced from the release parent and never treat the merge wrapper as a feature. ### 4. Build and consolidate evidence For every candidate capability, capture: - Plain-language feature description and user impact. - Source commits, plus PRs and linked issues when GitHub enrichment is available. - Release documentation paths and relevant headings. - Component and surface: Studio, CLI, API/SDK, Deployment, or Backend. - Exact documented Studio navigation. - Exact documented CLI commands. - Documented expected outcomes and failure paths. - Upgrade, compatibility, permissions, and security considerations. - Evidence-backed owner when available. - Confidence, disposition, and forward-merge state. - Release-note coverage: `Covered`, `Partial`, or `Missing`, with the exact note path and heading when present. - Release evidence separately from main-forwarding evidence. Keep command blocks verbatim except for clearly labeled placeholders. Do not infer Studio paths from component names or convert SDK examples into CLI commands. Group PRs that implement, document, fix, or polish the same user-visible journey. Keep separate features when user journeys, owners, risk, or validation differ. ### 5. Report gaps without broadening scope Explicitly identify: - Release-range feature/fix PRs without documentation evidence. - Documentation whose implementation is outside the release range. - Release-derived work missing or materially different on main. - Commands or Studio paths that cannot be sourced. - Conflicting or incomplete descriptions. - Critical security, migration, or compatibility work needing manual inclusion. - User-visible release or cherry-equivalent changes missing from the release-note diff and resulting note text. Do not add undocumented work automatically. Put it in a warning or manual-decision section. Before finalizing draft release notes in an interactive run, present all otherwise-includable capabilities with missing substantive documentation or `Partial`/`Missing` release-note coverage as one grouped decision. Let the user include, omit, or defer each item. In headless or explicitly non-interactive runs, retain the items in gaps and warnings but omit them from draft release notes. ### 6. Generate the shared local artifact Read [the shared release test-scope template](references/ticket-template.md). Use [the deterministic Git evidence collector](references/collect-git-evidence.sh) when available instead of manually retyping the Git collection commands; the generated files are analysis inputs, not the user-facing artifact. Write the reviewable test plan into the repository: ```text tests/plans/<version>.md ``` Use the exact release version string for `<version>`, including the patch component when provided. The artifact must be readable by all release stakeholders and must follow the section order of the template exactly: metadata table, release collection summary, forwarding exceptions, **Highest Priorities**, numbered **Proposed test scope**, manual additions, exclusions, gaps and warnings, draft release notes. Do not add a status blockquote, approver fields, or generation notes. Write **Highest Priorities** after the consolidated capabilities are settled, so every item traces to a capability section. Group by Platform, Agents, Evaluation, Fine-tuning, and Other; order by release risk; name Studio/CLI surfaces in parentheses; mark renames and rewrites `please regression test`. Items with no supporting capability go under **Manual additions**, not here. Use one numbered section per consolidated feature (`### 1. <name>`) and include the description, user impact, release-snapshot docs, applicable Studio or CLI entry points, expected result, additional validation, and draft release-note text. Cite sources as `<short SHA>` (#PR) and docs as repo paths. Write expected results as outcomes a tester can observe, including Studio/CLI parity when both surfaces exist. State clearly that it is a consolidated feature scope, not an exhaustive PR manifest. Draft release notes must use one structured block per included user-visible capability: `Description`, `Documentation`, and `Use it`. Under `Use it`, include verified CLI commands and/or Studio navigation as applicable; state explicitly when neither surface applies, and flag unverified Studio labels as needing final UI verification. Documentation links must be absolute URLs with a real host. Keep internal-only maintenance compact and do not invent an interface to fill the template. In the metadata table, write dates as `YYYY-MM-DD` and set `Review deadline` to the report generation date. Keep the artifact as a draft review input: if approvals or final scope decisions are needed, point reviewers to the external release tracker. For a pre-cut run where the release branch does not exist yet, `release_ref` may be the main ref. Record that in **Boundary warnings** and state in the forwarding section that both refs resolve to the same SHA. When the user supplies a release branch rather than a SemVer tag as `previous_ref`, accept it and record the same kind of boundary warning. If the artifact already exists, read it first and preserve human decisions, manual additions, exclusions, notes, and execution status. For maintainers reviewing this skill, [the 0.6.0 example report](references/0.6.0-example-report.md) shows a complete generated artifact in the earlier format, without **Highest Priorities** or numbered sections. The template takes precedence over it wherever they differ. Treat it as a dated review snapshot, not as input or an authoritative scope for later releases. ### 7. Verify before handoff Verify: - All refs still resolve to the recorded SHAs. - Candidate discovery used only `previous_ref..release_ref`. - Main was queried only for release-derived work. - Every **Highest Priorities** item maps to a numbered capability section. List unsupported items under **Manual additions**, not **Highest Priorities**. - Every capability has one forward-merge state. - Every user-visible cherry-equivalent group has a documented disposition. - No unrelated main-only PR appears. - Every automatic capability cites at least one release commit, a source PR when GitHub enrichment is available, and one published release-snapshot docs path. - All cited release docs exist at the release SHA. - Every capability has `Covered`, `Partial`, or `Missing` release-note coverage backed by the release-note diff and content. - Commands and Studio paths are source-backed or explicitly need clarification. - Draft release notes omit excluded and unresolved capabilities. - Manual additions and exclusions have rationale and owner placeholders. - Rerunning unchanged inputs produces the same candidate set and preserves review state. State any verification that could not be completed. ### 8. Optional GitHub issue Only after explicit authorization: - Search for an existing open issue for the same version and purpose. - Prefer updating it over creating a duplicate. - Use the shared release test-scope artifact as the issue body. - Apply only existing repository labels. - Report whether the issue was created or updated and return its URL. ## Completion report Report: - Version and the three resolved SHAs. - Release-range commit and PR counts. - Docs-touched PR and consolidated feature counts. - GitHub enrichment status and release-note coverage counts. - Counts by forward-merge state and disposition. - Path to the local shared artifact. - Missing evidence and review decisions. - State that the output is a draft for stakeholder review and link or name the external place where approvals should be tracked when known. - GitHub issue URL only when creation or update was authorized. Represent the generated artifact as supporting evidence for review; approval and scope finalization happen outside it.