github-pr-reviewer

# GitHub PR Reviewer Automation ## Agent Canvas catalog For new Agent Canvas installations, use the **GitHub code review** catalog entry. Its deterministic `worker.py` delegates each requested exact head to a stable conversation using the selected agent profile. It supports GitHub reviewer-request events and scheduled label scans. The manual upload flow below remains for existing deployments and is deprecated for new installations. Create a cron automation that watches one or more GitHub repositories for pull requests with a review trigger label, starts an OpenHands review conversation once per label event, and publishes the AI review to GitHub. Windows PowerShell equivalents for the setup, packaging, upload, and API-check shell snippets are in `references/windows.md`. The automation script is deterministic: PR discovery, label-event tracking, head eligibility, state persistence, stale-result suppression, the repository checkout, and its removal are all handled in Python. The LLM is invoked only for the review itself. Before any review conversation is created, a scheduled scan evaluates the current head's GitHub-required checks, read through the GraphQL `isRequired` signal for the pull request. That signal is the merge policy's own source of truth and is pull-request-scoped, so it stays correct for a stacked PR whose symbolic base branch carries no branch rules of its own: - Only checks GitHub reports as required decide the scheduled gate. An optional workflow that fails - including one that fails before creating any check run - does not block a head whose required checks pass. - A completed required run on the exact head whose conclusion is `failure`, `cancelled`, or `timed_out` blocks the review. Any other unrecognized conclusion fails closed as a block rather than silently approving. - A completed required run whose conclusion is `success`, `neutral`, or `skipped` does not block. A required `commit status` context is classified alongside required check runs. - A `queued` or `in_progress` required run on the exact head, and a required context that has not reported a current-head check run at all, make the run exit with a **waiting on checks** outcome. Unfulfilled is never approval. The worker does not hold a slot polling; the next scheduled scan or explicit review request retries. - When the required-check signal cannot be read or is empty, the gate falls back to every current-head check run and workflow run, so a red head still blocks. In that fallback, workflow runs are read as well as check runs, because a workflow can fail before creating any check run - a workflow-level error, or a `pull_request` run whose jobs never start. Such a run leaves a failed check suite with no check runs under it, so the commit's check-run rollup and `gh pr checks` both report success and only the workflow run reveals the red CI. A workflow run whose check suite already reported check runs is left to those runs, so a workflow is never counted twice. - A scheduled scan considers the trigger label **and** every open PR, including drafts, that still holds an outstanding `all-hands-bot` review request. That is what resumes a request made while CI was running: the request is keyed by its own `review_requested` event, so repeated scans reuse one conversation and one review instead of creating duplicates, and a PR whose request was answered or withdrawn simply drops out of `requested_reviewers`. - The scheduled scan also reviews open, non-draft PRs that **nobody requested** and that carry no trigger label, once their current head has no completed review by this account. That is what keeps reviewing PRs after the outstanding requests are exhausted. The delivery key is the stable `scan:{repository}:{number}:{head}`, so repeated scans reuse one conversation and one native review, and a changed head becomes eligible again under its new SHA. - Each scheduled scan classifies the **whole open backlog**, including unrequested PRs. Blocked, pending, draft, or already-reviewed heads therefore cannot hide an eligible head behind an inspection window. The separate `max_new_per_run` quota limits only newly created review conversations, not the number of PRs inspected; explicit requests and trigger labels retain priority when eligible candidates are drained. - A gate stop on an **unrequested** head posts no managed comment. A PR nobody asked about that is merely red or pending is simply skipped without starting an agent, which is what keeps a scan over a large backlog from posting a gate comment on every PR. The managed comment is the answer to an explicit request, so a requested or labeled head that is red or pending still gets its explanation. - Workflow runs are read as well as check runs, because a workflow can fail before creating any check run - a workflow-level error, or a `pull_request` run whose jobs never start. Such a run leaves a failed check suite with no check runs under it, so the commit's check-run rollup and `gh pr checks` both report success and only the workflow run reveals the red CI. A workflow run whose check suite already reported check runs is left to those runs, so a workflow is never counted twice. - Runs attributed to any other (obsolete) head SHA are ignored, so a stale failure cannot block the push that fixed it. This applies to workflow runs too. - Only the latest run of each logical check or workflow counts. A logical check is its name plus the reporting app identity, and a logical workflow is its name plus workflow ID. The latest is chosen by the run ID (the reliable creation sequence) with the start time as a tie-break, so a re-run that fixed a check supersedes its earlier failure on the same SHA while a newer queued or in-progress re-run supersedes an earlier success and makes the head wait. Ordering by the run ID keeps a run whose start time is still absent in its true creation position in both directions. An explicit `all-hands-bot` review request is the intake-policy exception: the caller asked for that head by name, so the request dispatches even when required CI is red or pending, and the gate leaves no explanatory comment. Draft, scope, exact-head, and delivery-deduplication safeguards still apply. A blocked or waiting scheduled stop consumes no trigger: once the head's required checks are non-blocking, the scheduled scan, or a new `all-hands-bot` review request, starts the normal review. The gate needs no configured list of check names. When it stops a review it leaves one concise explanation on the PR, identified by a hidden marker carrying the head SHA and gate category, so a later run for a different head updates that comment instead of posting another. Only a marker this reviewer account authored counts as its own comment; every other PR comment is untrusted and can neither suppress the explanation nor be edited. If the repository's own workflow already posted a deterministic remediation comment that names the same current-head checks, the gate adds nothing; a disclosure about some other check does not suppress it. The gate applies to scheduled label scans; an explicit `all-hands-bot` review request bypasses it by design. Each explicit review request - a re-applied trigger label, or a new reviewer request in event mode - is a fresh review. The conversation for a PR is reused, but its earlier turns must not be trusted as current: before deciding a verdict the reviewer re-fetches the mutable GitHub state (the exact head, the PR body, review comments and threads, review requests, the linked issues' bodies and labels, and the current-head Actions results) and ignores any earlier finding, verdict, or label/priority claim that state no longer supports. Repository analysis the conversation already did, such as reading `AGENTS.md`, stays useful and is not repeated. The waiting and blocked explanations name the retry the deployment actually has. A scheduled run proves a scan is configured, so it says the next scan will retry. An event-only run does not, so it tells the reader to remove the outstanding `all-hands-bot` request and request `all-hands-bot` again instead of promising a scan that does not exist: GitHub will not accept a second request while the first is still outstanding. The gate leaves only one explanation per head. When the same head keeps the same hidden marker but the retry wording changes -- an event-only comment on a head whose automation is later switched to a cron scan -- the scheduled run rewrites that managed comment in place, so the comment always names the retry that is actually deployed, and an unchanged body is left untouched. ## Bounded intake per scheduled scan A scheduled scan drains outstanding reviewer requests fairly, but starting an agent for every eligible pull request at once would overload the deployment, so the scan starts at most `MAX_NEW_PER_RUN` new review conversations, counted **across every configured repository** rather than per repository. The default is `2`, and the rendered `config.json` overrides it through the `max_new_per_run` key that `github-issue-to-pr` and `gitlab-issue-to-mr` already use. - Eligible candidates are ordered deterministically: an explicit `all-hands-bot` request (or a trigger label) is ordered before an unrequested PR, then by the oldest `all-hands-bot` `review_requested` event for a requested PR or the PR's own creation time (oldest first) for an unrequested one, then by repository, then by pull-request number, so the oldest work drains first and a later scan reaches the remainder. - The bound counts the conversations a scan **starts**. A delivery that only deduplicates or reports an already-running conversation reuses a runtime and consumes no slot, so repeated scans make progress on the backlog instead of re-spending the bound on work already in flight. - Reaching the bound never cuts the scan short. The scan still evaluates the exact-head checks of the remaining candidates, reconciles completed reviews, and runs the maintainer handoff. A candidate whose checks are pending or failing starts no conversation and consumes no slot, so it cannot block a later green candidate. Only an explicit request or a trigger label gets a waiting or blocked gate comment; an unrequested head that is merely red or pending is skipped silently. - A dispatch that raises is reported and does not consume a slot or abort the scan, so the candidates behind it are still considered. - The explicit `review_requested` event path and the trigger-label scan are unchanged: an explicit request still starts its conversation immediately, and only the scheduled scan's new conversations are bounded. The review prompt starts with a scope gate: using the repository's own guidance (its scope categories and ownership boundaries, not a list of individual PR numbers), the reviewer decides whether the change belongs in this repository and has the product/architecture direction it needs. When it does not, the review stops with a single `event: COMMENT` review that says whether the change should move repositories, close, or receive a maintainer decision, and ends with the `🛑 MAINTAINER DECISION REQUIRED` verdict. That outcome is **neither an approval nor a change request**: it does not approve or merge the PR. The completion handler recognizes the verdict and requests one configured maintainer through the same handoff used after an approval. Both paths require a successfully fetched same-repository closing issue labeled `priority:medium` or `priority:high`; a missing link or priority label, low/normal priority, unresolved reference, or PR reference does not authorize handoff. One qualifying issue is sufficient even with other low-priority or missing references. Existing manual review requests are unchanged. An in-scope change continues the existing review unchanged. Before approving a change to user-visible UI behavior, the prompt requires live evidence from a real running application when the repository's guidance demands it: a screenshot, screen recording, or equivalent capture of the running app that exercises the production-facing path. Unit tests, CSS-token or contract assertions, generated mockups, and reconstructed captures may support the review but cannot substitute for that evidence. When the required evidence is missing, the reviewer publishes one `event: COMMENT` review naming exactly what is missing and ends with `🔄 CHANGES REQUESTED`, so approval is withheld and the deterministic maintainer handoff does not fire. Non-UI changes keep the existing requirement of the real command and its observed output, with tests insufficient as sole proof. The script prepares each review's workspace before the agent starts: the pull request's head commit is downloaded as a tarball and extracted to a directory of its own, which becomes the conversation's working directory. The agent is told not to clone, fetch, check out, or delete anything, and the script removes the checkout once the conversation has stopped. Nothing accumulates between runs. --- The script imports shared GitHub transport from `scripts/github_client.py`, installed with this skill. Include it beside `main.py` when packaging manually, as shown below; catalog bundles include it automatically. ## Prerequisites ### Required secret Verify that the following secret is set in **OpenHands Settings -> Secrets**: | Secret name | Token type | Minimum permissions | |---|---|---| | `GITHUB_PERSONAL_ACCESS_TOKEN` | Classic PAT | `repo` for private repos or `public_repo` for public repos | | `GITHUB_PERSONAL_ACCESS_TOKEN` | Fine-grained PAT | Contents: Read, Metadata: Read, Pull requests: **Read and Write**, Issues: Read and Write | Pull-request **write** access is required because the agent publishes a pull request review, not just an issue comment. The Agent Canvas catalog worker may also request a configured human reviewer after approval. A token with only Pull requests: Read will poll happily and then fail at the point of publishing or requesting the handoff. When several repositories are monitored, the token must cover all of them. Check with: ```bash curl -s https://api.github.com/user \ -H "Authorization: Bearer $GITHUB_PERSONAL_ACCESS_TOKEN" \ | python3 -c "import json,sys; d=json.load(sys.stdin); print(d.get('login') or d.get('message'))" ``` If the token is missing or invalid, inform the user and stop. --- ## Setup Workflow Follow these steps in order. ### Step 1 - Verify `GITHUB_PERSONAL_ACCESS_TOKEN` Run the `curl` check above. - If absent: *"GITHUB_PERSONAL_ACCESS_TOKEN is not set. Please add it in OpenHands Settings -> Secrets."* Stop. - If the API returns `{"message": "Bad credentials"}`: tell the user the token is invalid and ask them to update it. Stop. ### Step 2 - Collect repositories Ask: *"Which GitHub repositories should be monitored? (Format: `owner/repo`, e.g. `myorg/backend`. List several separated by commas to review them all from one automation.)"* Validate access to **each** repository: ```bash curl -s "https://api.github.com/repos/{owner}/{repo}" \ -H "Authorization: Bearer $GITHUB_PERSONAL_ACCESS_TOKEN" \ | python3 -c " import json, sys d = json.load(sys.stdin) if 'message' in d: print('ERROR:', d['message']) else: print(f\"Accessible. Private: {d.get('private')}. Permissions: {d.get('permissions')}\") " ``` Record every accepted repository into `REPOS = ["{owner}/{repo}", ...]`. If one repository fails the check, say which and ask whether to continue without it. Each repository is polled independently and keeps its own state, so pull-request numbers never collide between them. The trigger label, tone, and schedule are shared by all of them; a repository needing different settings wants its own automation. ### Step 3 - Collect trigger label Ask: *"Which PR label should trigger a review? (Press Enter for the default: `openhands-review`.)"* Record the answer as `TRIGGER_LABEL`. If the label does not exist yet, tell the user that GitHub will still record the event once the label is created and applied to a PR. The automation reviews a PR when it sees the latest matching `labeled` event for that label. To request another review later, remove and re-apply the label. ### Step 4 - Collect review tone Ask: *"What review tone should the reviewer use? 1. Thorough (default) - comprehensive coverage of correctness, security, tests, style 2. Concise - high-signal only, skips minor style feedback 3. Friendly - constructive and encouraging (Press Enter for Thorough, or type your choice or any custom style description)"* Map the choice to `REVIEW_TONE`: | Answer | `REVIEW_TONE` | `REVIEW_STYLE_INSTRUCTIONS` | |---|---|---| | 1 / Enter | `"thorough"` | `""` | | 2 | `"concise"` | `""` | | 3 | `"friendly"` | `""` | | Custom text, e.g. `strict but kind` | `"thorough"` | the custom text verbatim | ### Step 5 - Collect cron schedule Ask: *"How often should the automation poll for labeled PRs? (Press Enter for the default: every 5 minutes. Use a cron expression for a different interval, e.g. `0 * * * *` = hourly)"* Default: `*/5 * * * *`. Record as `CRON_SCHEDULE`. ### Step 6 - Generate the automation script Read `scripts/main.py` from this skill's directory. Apply exactly seven constant substitutions near the top of the file: > The script also reads a `config.json` shipped beside it, if there is one, over > these constants. That is how the catalog entry > (`automations/catalog/github-pr-reviewer/`) configures an unmodified copy, > since a declarative host cannot rewrite Python. This setup path substitutes the > constants and ships no `config.json`, so the two never collide. | Placeholder | Replace with | |---|---| | `REPOS = ["owner/repo"]` | `REPOS = ["{owner_repo}", ...]` - one entry per repository collected in Step 2 | | `TRIGGER_LABEL = "openhands-review"` | `TRIGGER_LABEL = "{trigger_label}"` | | `REVIEW_TONE = "thorough"` | `REVIEW_TONE = "{review_tone}"` | | `REVIEW_STYLE_INSTRUCTIONS = ""` | `REVIEW_STYLE_INSTRUCTIONS = "{style_instructions}"` | | `REPO_REVIEW_GUIDE_PATH = ".agents/skills/custom-codereview-guide.md"` | leave unchanged to auto-load a repo review guide at this path, or set to `""` to disable | | `MAX_NEW_PER_RUN = 2` | leave unchanged to bound a scheduled scan to two new review conversations across all repositories, or raise it if the deployment can hold more agents at once | | `DEFAULT_OPENHANDS_URL = "http://localhost:8000"` | leave unchanged unless the user has a preference | Use a safe string writer such as `json.dumps(value)` when inserting user-provided repository names, labels, or style instructions into Python string literals. `json.dumps(list_of_repos)` produces the whole `REPOS` list safely in one step. Run these commands from this skill's directory and write the customized script to a temporary build directory: ```bash mkdir -p /tmp/pr-reviewer-build cp -L scripts/github_client.py /tmp/pr-reviewer-build/github_client.py # write the customized main.py to /tmp/pr-reviewer-build/main.py ``` Validate syntax before packaging: ```bash python3 -m py_compile /tmp/pr-reviewer-build/main.py && echo "Syntax OK" ``` Fix any syntax errors before proceeding. ### Step 7 - Package and upload Determine the Automation backend URL and auth from the `<RUNTIME_SERVICES>` block in your system context: - **OPENHANDS_HOST**: the Automation backend `url_from_agent` - **Auth**: `X-Session-API-Key: $OPENHANDS_AUTOMATION_API_KEY` ```bash tar -czf /tmp/pr-reviewer.tar.gz -C /tmp/pr-reviewer-build . TARBALL_PATH=$(curl -s -X POST \ "${OPENHANDS_HOST}/api/automation/v1/uploads?name=github-pr-reviewer" \ -H "X-Session-API-Key: $OPENHANDS_AUTOMATION_API_KEY" \ -H "Content-Type: application/gzip" \ --data-binary @/tmp/pr-reviewer.tar.gz \ | python3 -c "import json,sys; print(json.load(sys.stdin)['tarball_path'])") echo "Uploaded: $TARBALL_PATH" ``` ### Step 8 - Register the automation ```bash curl -s -X POST "${OPENHANDS_HOST}/api/automation/v1" \ -H "X-Session-API-Key: $OPENHANDS_AUTOMATION_API_KEY" \ -H "Content-Type: application/json" \ -d "{ \"name\": \"GitHub PR Reviewer: {repo_summary} label {trigger_label}\", \"trigger\": {\"type\": \"cron\", \"schedule\": \"{cron_schedule}\"}, \"tarball_path\": \"$TARBALL_PATH\", \"entrypoint\": \"python3 main.py\", \"timeout\": 600 }" | python3 -m json.tool ``` Use the single repository as `{repo_summary}` when there is one, and something like `3 repos` when there are several. A poll now downloads a tarball per queued review, so the timeout allows for that; a run never waits for a review to finish, only for it to be started. Record the returned `id`. ### Step 9 - Confirm Tell the user: > ✅ **GitHub PR Reviewer** is running! > > - Automation ID: `{id}` > - Repositories: `{owner}/{repo}`, ... (one line each) > - Trigger label: `{trigger_label}` > - Review tone: `{tone}` > - Polling schedule: `{cron_schedule}` > - State file per repository: