| name | review-library-bulk-update |
| description | Review automated pull requests with the `library-bulk-update` label in graalvm-reachability-metadata. Use when asked to review or triage a PR that bumps tested versions for existing libraries, including approve vs close decisions, CI checks, diff-scope validation, and verification of `source-code-url`, `test-code-url`, `documentation-url`, and `repository-url` changes. |
Review library-bulk-update PRs
These PRs are created by the scheduled compatibility workflow and are expected to update supported library versions by changing metadata/**/index.json. They may also refresh the generated root COVERAGE.md when tested-version changes affect the coverage table.
Historical Review Logic
Previous reviews follow a simple pattern:
- Approve silently when the PR only updates the expected metadata index files, optionally with generated root
COVERAGE.md, and CI is green, then enable auto-merge.
- Close the PR when it contains unrelated changes such as workflow files, Gradle files, generated sources, or test code that clearly did not belong in a bulk version update.
- Close stale PRs with merge conflicts instead of repairing them manually if a newer bulk-update PR will supersede them.
- If CI looks flaky or inconsistent with the diff, ask for or trigger a rerun rather than approving blindly.
- Treat generated-source changes as suspicious in this PR type. They should normally not appear here.
Workflow
-
Inspect the PR summary.
- Confirm the PR has label
library-bulk-update.
- Read the body and verify it is a bot-generated "Added tested versions" summary.
- Gather files, reviews, issue comments, inline comments, and status checks.
-
Validate the diff scope first.
- Expected files:
metadata/<group>/<artifact>/index.json, plus the generated root COVERAGE.md when the PR updates tested versions.
- Normal change shape: new values appended to
tested-versions, with formatting-only line movement around the surrounding JSON.
- Be suspicious of changes to
latest, metadata-version, test-version, allowed-packages, requires, or URL fields unless the PR clearly needs them.
- A pure pre-release-to-release promotion produced by
addTestedVersion is allowed when it is limited to the matching library and consists only of:
metadata/<group>/<artifact>/<oldVersion> -> <newVersion>
tests/src/<group>/<artifact>/<oldVersion> -> <newVersion>
stats/<group>/<artifact>/<oldVersion> -> <newVersion>
- matching
index.json changes to metadata-version, tested-versions, and exact-version artifact URLs
- Reject or close the PR if it changes:
.github/workflows/**
ci.json
gradle/**, build.gradle, settings.gradle, gradle.properties
tests/src/** outside a pure matching pre-release promotion rename
stats/** outside a pure matching pre-release promotion rename
- generated Java sources
- any other non-
metadata/**/index.json file, except the generated root COVERAGE.md
-
Check that the PR body matches the diff.
- Every library listed in the body should correspond to an
index.json diff.
- Added tested versions should match the actual JSON changes.
- A
COVERAGE.md refresh is allowed as generated fallout from the listed library updates; it does not need to be listed as a separate library change.
- There should not be hidden updates outside the listed libraries, except the generated root
COVERAGE.md.
-
Check CI before deciding.
- Required baseline: JSON validation, metadata tests, and style checks must be green.
- If the PR unexpectedly triggered build-logic or workflow-heavy jobs, that is a sign the diff scope is wrong. A root
COVERAGE.md change alone is not workflow-heavy.
- If checks failed because the PR is contaminated by unrelated changes, close it.
- If checks failed due to likely infra flakiness or a suspicious runner issue, request a rerun or rerun all relevant tests before approval.
-
Review each changed index.json.
- Confirm the new versions were added under the correct
metadata-version entry.
- Preserve existing ordering conventions; new tested versions are typically appended in version order.
- Do not accept dropped tested versions, entry reshuffling, or unrelated field edits without a clear reason.
- For pre-release promotions, confirm
metadata-version moved from the matching pre-release to the corresponding full release, old pre-release values were removed from tested-versions, and any test-version references were updated only when they pointed at the promoted test directory.
-
Apply URL verification when URL fields changed.
- This is higher scrutiny than a normal bulk-update PR.
- Review
source-code-url, test-code-url, documentation-url, and repository-url with the same rules used by PopulateArtifactURLs.
- For pre-release promotions, verify the
source-code-url, test-code-url, and documentation-url values were rendered from the old version to the promoted metadata-version or otherwise corrected to an exact-version URL. Do not accept stale URLs that still point at the old pre-release.
-
If the PR is clean and approved, enable auto-merge.
- In this repository, approved automation PRs are expected to auto-merge after approval.
- Do this after the approval is submitted, and only when the PR is otherwise mergeable.
URL Verification Rules
If any URL field changed, verify all of the following:
repository-url is the canonical repository root URL.
repository-url must not include a versioned tree path such as /tree/<tag>.
source-code-url, test-code-url, and documentation-url must point to the exact library version in the changed entry.
- Do not accept unversioned docs,
latest, current, or branch-based docs unless there is no versioned source and the PR clearly justifies it.
- Prefer Maven
-sources.jar, -test-sources.jar, and -javadoc.jar when they exist and are valid.
- If a Maven
-sources.jar or -test-sources.jar is used, verify it contains real source files such as .java, .kt, .scala, or .groovy, not only metadata or license files.
- If the URL points to a repository tree or archive instead of Maven, verify that it resolves to real source or test files for the exact version tag.
- If a candidate source or test URL cannot be verified with confidence, it should not be approved as-is.
- For pre-release promotions, template rendering is expected: old-version URL tokens may be replaced by the promoted version only when the rendered URL resolves and archive/source checks pass.
Use the logic from tests/tck-build-logic/src/main/groovy/org/graalvm/internal/tck/harness/tasks/PopulateArtifactURLs.java, especially the urlUpdateInstructions and sourceArtifactVerificationInstructions rules.
Decision Rules
Approve when all of these are true:
- The PR is limited to the expected
metadata/**/index.json files, optionally with generated root COVERAGE.md.
- Or the PR is a pure verified pre-release promotion rename limited to the matching
metadata/, tests/src/, stats/, index.json, and optional root COVERAGE.md changes.
- The diff is only tested-version maintenance or clearly justified URL maintenance.
- CI is green, or any rerun confirms green status.
- Any changed URLs were verified against the exact version.
- Auto-merge was enabled after approval when the PR is mergeable.
Close or reject when any of these are true:
- The PR contains unrelated files or generated code.
- The PR is stale, conflicted, or clearly superseded by a newer automation run.
- CI failures point to a real regression or invalid update.
- URL changes cannot be verified confidently.
Ask for rerun or deeper investigation when:
- The diff looks correct but CI failed in a way that smells like infrastructure noise.
- A generated file changed and you need to confirm whether the workflow accidentally regenerated something.
Output Style
Match the historical style:
- Clean PR: approve with no comment or a very short confirmation, then enable auto-merge.
- Contaminated PR: leave a short factual comment explaining why it is being closed or should not be merged.
- Flaky CI: leave a short comment requesting or noting a rerun.