review-reviewers

# Review Reviewers Analyze tend's CI behavior on the target repo over the window Step 1 returns. Focus on **outcomes** — what the bot produced publicly and whether it was accepted — rather than internal session mechanics. Create PRs or issues on tend when outcomes reveal behavioral problems. ## First steps Load `/tend-ci-runner:running-in-ci` first — it contains CI security rules, PR/comment formatting (line wrapping, heredoc hazards), and polling conventions. This skill opens PRs and issue comments on tend, so those rules apply. ## Cost discipline: smaller, cheaper models for exploration Session log parsing and outcome checking are token-heavy. Delegate all broad exploration to a **smaller, cheaper model**. Keep the main agent for judgment: evaluating findings against gates, deciding whether to act, and drafting PRs. Pattern: 1. Main agent sets up context (bot identity, repo guidance, run list) 2. Main agent delegates the run survey to a subagent using a smaller, cheaper model → receives structured summary 3. Main agent evaluates the summary against gates 4. If needed, main agent delegates specific session logs to another subagent using a smaller, cheaper model → receives diagnosis 5. Main agent drafts fix PR if warranted ## Core principle: outcomes over internals The bot's job is to produce useful outputs: reviews, triage comments, fix commits, issue responses. The cheapest way to evaluate quality is to check whether those outputs were **accepted** (merged, kept, acted on) or **rejected** (reverted, closed, corrected, disagreed with). Session logs are expensive to download and parse. Only escalate to session-log inspection when outcome signals indicate a real problem worth diagnosing. ## Core principle: repo-specific guidance is primary Each adopter repo has its own guidance (`running-tend` skill or equivalent) that shapes how the bot should behave in that repo. This repo-specific guidance **takes precedence** over tend's default rules. The bot's job is to follow the repo-specific guidance first, falling back to tend's defaults only where the repo doesn't specify. ## Non-issues: do not flag these Some patterns look suspicious but are intentional — flagging expected behavior creates maintainer churn and costs trust. Three structural rules cover them: - **Designed no-ops.** Many events correctly end with nothing posted, at whatever layer catches them: a pre-boot gate skip (`tend-mention`'s verify gate on the bot's own comments and reviews — though targets on older pinned releases still boot sessions for those), or a session that boots and exits silently (`tend-triage` on the bot's own monthly tracking-issue creation; the `issue_comment.edited` retrigger after a commenter refines their comment — the edit can change relevance, so the retrigger must re-evaluate; `tend-notifications` mark-reading a cross-repo `ci_activity` notification from an abandoned fork). These cost compute, not correctness — Gate 3 classifies them waste-class: record and move on; do not propose a skip-gate, label filter, pre-check, or occurrence threshold to save the boot. A loop that produces *wrong outward actions* (duplicate comments, spurious reviews) is different — that passes Gate 3, and a label-based skip is preferred over an authorship filter where a label can express it. - **Designed silence.** The bundled `review` skill authorizes posting nothing when there is nothing actionable: on a self-authored PR (GitHub rejects self-approvals, so APPROVE isn't an option), on a draft PR (COMMENT-only mode; GitHub blocks approving drafts), or when the PR closed or merged while its run was queued. GitHub reports drafts as `state: OPEN`, so before reading a missing review as omission — or escalating to session logs to explain it — check `gh api repos/OWNER/REPO/pulls/N --jq '{state, draft}'` and the PR's literal author (`gh pr view <n> --json author --jq '.author.login'`; owner-authored PRs are approved normally and are no bot-authored-APPROVE precedent). - **The reviewer role is independent of authorship.** `tend-review` re-reviewing — and re-approving — after any tend workflow pushes a fix commit is the design, not a re-approval loop; authorship-keyed guards that skip re-review drop real work and are not an accepted shape. Stacked approvals from racing runs are a *concurrency* artifact (cancelled runs POSTing before the SIGTERM arrived), not a review-rule problem. ## Target repo **Target repo:** $ARGUMENTS Analysis targets an adopter repo whose CI runs are analyzed. Findings result in PRs/issues on the current repo (tend) to improve skills and workflows. Use `-R $ARGUMENTS` for commands that access the target repo (querying runs, PRs, issues). Commands without `-R` default to tend. @review-gates.md ## Evidence accumulation Evidence lives in one secret gist per target repo and month, indexed by the monthly `review-reviewers-tracking` issue on Tend. Prepare it and read the current and previous month's evidence: ```bash uv run --script \ "${CLAUDE_PLUGIN_ROOT}/scripts/review_reviewers.py" \ prepare-evidence "$ARGUMENTS" ``` The command finds or creates both index and gist, announces a new gist once, persists their ids, and prints both evidence windows. After applying the gates, write this run's findings in the format from `@review-gates.md` to `$TMPDIR/findings.md`. Append a `## Run $GITHUB_RUN_ID` heading every run, including an all-clear window; the heading is the audit trail future runs use. Then append it: ```bash uv run --script \ "${CLAUDE_PLUGIN_ROOT}/scripts/review_reviewers.py" append-evidence ``` The command refuses a findings file that does not name this run, fetches the latest gist content, and appends without replacing prior evidence. ## Step 1: Setup Resolve the **target repo's** bot login and load repo-specific guidance upfront — both are needed throughout. `gh api user` returns the *analysis* bot (e.g., `tend-agent` when review-reviewers runs on tend), which is typically **not** the target repo's bot — filtering reviews/comments by the wrong login produces false "no bot output" negatives. Read `bot_name` from the target repo's `.config/tend.yaml`: ```bash BOT_LOGIN=$(gh api "repos/$ARGUMENTS/contents/.config/tend.yaml" --jq '.content' 2>/dev/null \ | base64 -d 2>/dev/null \ | yq '.bot_name // ""' 2>/dev/null) if [ -z "$BOT_LOGIN" ]; then echo "ERROR: could not resolve bot_name from $ARGUMENTS/.config/tend.yaml" >&2 exit 1 fi echo "BOT_LOGIN=$BOT_LOGIN (target: $ARGUMENTS)" ``` Read the target repo's repo-specific guidance to understand what the bot was told to do: ```bash gh api "repos/$ARGUMENTS/contents/.claude/skills/running-tend/SKILL.md" \ --jq '.content' | base64 -d ``` If the file doesn't exist, try the legacy overlay paths and the repo's root project instructions (`.claude/skills/running-tend.md`, `.claude/CLAUDE.md`, `CLAUDE.md`, `AGENTS.md`). Understanding the repo's guidance is essential context for evaluating outcomes — without it, you'll misjudge authorized behavior as a violation. Then list recently completed tend CI runs on the target repo: ```bash TARGET_REPO=$ARGUMENTS uv run --script \ "${CLAUDE_PLUGIN_ROOT}/scripts/list_recent_runs.py" review-reviewers ``` The script discovers `tend-*` workflows by default. Pass additional prefixes as arguments to include other workflows (e.g., `review-reviewers` when analyzing tend itself). If empty, record the run as all-clear per "Recording below-threshold findings" above, then skip to Step 6. If the script printed a `WARNING:` on stderr, the list is known-incomplete — the window was clamped, no anchor was found, or a workflow hit the fetch limit. Record a coverage gap naming the missing span instead of an all-clear, whether or not the list came back empty; the next run's floor advances past that span regardless, so an unrecorded gap is never revisited. If the script *fails* (non-zero exit, e.g. a transient API error), re-run it once; if it fails again, record the window as a coverage gap the same way — this run still concludes green, so the next tick anchors on it and never revisits the span. **State the window you analyzed.** Its floor is the previous successful run of this workflow, or 6h back when that is older. This workflow is dispatch-only, so runs sit further apart than a cron's and the floor moves accordingly: scope every claim to it — "no problems since 08:12Z", never "no problems" — and say plainly when the run was dispatched to check on something that landed before it. ## Step 2: Survey outcomes via a smaller, cheaper model Spawn a subagent using a smaller, cheaper model to check outcomes across all runs from Step 1. The subagent does the token-heavy work of mapping runs to PRs/issues and checking acceptance signals. Use a smaller, cheaper model for the subagent and a prompt like: > Survey bot outcomes on `$ARGUMENTS` for the following runs: [run IDs from Step 1]. > The bot's login is `$BOT_LOGIN`. > > For each run, determine: > 1. Did the bot produce visible output (review, comment, issue action, commit)? > 2. If yes, was the output accepted or rejected? > > **Sweep the window repo-wide before mapping any run, and report the row counts.** The per-run mapping below walks run → branch → PR → endpoint; a break anywhere in that chain returns empty for *every* run at once, and uniform silence reads as a quiet hour rather than as a broken query. These calls take no run ID, so they fail independently of it: > > ```bash > WINDOW_START=<window start, ISO 8601> > WINDOW_END=<window end, ISO 8601> > IN_WINDOW="[.[] | select(.user.login == \"$BOT_LOGIN\") > | select((.created_at // .submitted_at) >= \"$WINDOW_START\" > and (.created_at // .submitted_at) <= \"$WINDOW_END\")] | length" > > gh api "repos/$ARGUMENTS/issues/comments?since=$WINDOW_START&per_page=100" --jq "$IN_WINDOW" > gh api "repos/$ARGUMENTS/pulls/comments?since=$WINDOW_START&per_page=100" --jq "$IN_WINDOW" > > CANDIDATES=$(gh -R $ARGUMENTS pr list --state all --limit 100 \ > --search "updated:>$WINDOW_START" --json number --jq '.[].number') > echo "$CANDIDATES" > for pr in $CANDIDATES; do > gh api "repos/$ARGUMENTS/pulls/$pr/reviews?per_page=100" --jq "$IN_WINDOW" > done | jq -s add > ``` > > Filter the comment calls on `created_at` at both ends. `since` is an `updated_at` floor with no ceiling, so unfiltered it also returns comments written days earlier and merely edited in the window, plus everything posted between the window end and now — the session's own runtime keeps that gap open, so on a busy repo it catches rows in most windows. Those are rows the per-run walk was right not to reach, and a check that contradicts a correct walk every run stops being believed. Leave `CANDIDATES` unbounded above: `updated:` matches the PR's own `updated_at`, so a range drops any PR that got bot output inside the window and was touched after it, and over-inclusion in a candidate list costs nothing. `--limit 100` is load-bearing — `gh pr list` defaults to 30 and truncates silently. > > The comment endpoints cover conversation and inline-review comments only; neither returns review submissions, and an empty-body `APPROVE` is `tend-review`'s most common output. Without the fourth count an approvals-only window reports `0, 0` and satisfies the gate below with two zeros carrying no signal. Report all four numbers at the top of your summary. If the sweep found in-window rows your per-run walk did not reach, the walk is wrong — re-map from the PR numbers the sweep found rather than reporting those runs as silent. > > **Acceptance needs a non-bot actor — name it.** For every acceptance or rejection signal, report the login that produced it: who merged, who reviewed, who replied, who pushed the follow-up. `$BOT_LOGIN` reviewing, replying to, or pushing to its own PR is the bot talking to itself, not acceptance. A non-bot merge *is* acceptance even when every commenter is the bot — the most common shape here is a maintainer merging a bot PR without ever posting. Reserve `bot-only — no human signal` for threads where `$BOT_LOGIN` is the only login across every surface. One `gh pr view` covers them all — merge actor, reviews, inline comments, conversation comments, commits: > > ```bash > gh -R $ARGUMENTS pr view <pr> --json number,state,author,mergedBy,reviews,comments,commits --jq '{ > pr: .number, state, author: .author.login, merged_by: .mergedBy.login, > reviews: [.reviews[] | {login: .author.login, state}], > logins: ([.mergedBy.login, .reviews[].author.login, .comments[].author.login, > .commits[].authors[].login] | map(select(. != null and . != "")) | unique) > }' > ``` > > `logins` is the bot-only test: `["$BOT_LOGIN"]` or empty means no human touched the thread. Anything else, name that login and say which surface it came from. Every inline review comment belongs to a review record — including a standalone reply posted through the replies endpoint — so `reviews` covers inline commenters as well as submitted reviews, and its `state` gives the accept/reject direction. `comments` and `reviews` paginate in full. `commits` does not: `gh pr view` selects `commits(first: 100)` and never fetches a second page, oldest-first, so past 100 commits the newest drop out — exactly where a human follow-up push sits. On a PR that long, read the authors from `gh api "repos/$ARGUMENTS/pulls/<pr>/commits" --paginate` (itself capped at 250) before calling the thread bot-only. > > A comments-only check is not enough on its own: it misses both a silent maintainer merge (`mergedBy` set, nobody commenting) and a human `CHANGES_REQUESTED` that carries no inline comments. > > **How to map runs to outputs:** > - `tend-review`: `gh -R $ARGUMENTS run view <run-id> --json headBranch` → find PR via > `gh -R $ARGUMENTS pr list --head <branch> --state all` → check bot reviews via > `gh api repos/$ARGUMENTS/pulls/<pr>/reviews` > - `tend-notifications`: check for bot comments/issue-close events inside the window from Step 1 > - `tend-mention`: map run to issue/PR from triggering comment, check for bot replies > - `tend-mention` on `repository_dispatch` (the relay path for review events): there is no triggering comment and `headBranch` is the default branch, so neither route above resolves it. Read the target off the `verify` job's log, where the step env block prints the relayed payload: > ```bash > JOB=$(gh api "repos/$ARGUMENTS/actions/runs/<run-id>/jobs" --jq '.jobs[] | select(.name == "verify") | .id') > gh api "repos/$ARGUMENTS/actions/jobs/$JOB/logs" | grep -E 'PAYLOAD_(KIND|PR|ID):' > ``` > `PAYLOAD_PR` is the issue/PR number and `PAYLOAD_KIND` is the relayed event (`pull_request_review`, `pull_request_review_comment`). Read the gate's verdict off the `handle` job, which is gated on `should_run`: `handle` with conclusion `skipped` means the engagement gate declined and no agent booted — expected silence, not missing output. Do not read it off `React to mention`, which is skipped on every relayed `pull_request_review` regardless of the verdict (a review submission has no single comment to react to) and on a comment relay admitted for participation rather than a mention. > - `tend-ci-fix`: map run → PR via `headBranch`, check for bot commits > > **Negative outcome signals** — report any sign the bot's output was rejected, corrected, or ignored. Common shapes (use judgment for signals not listed): > - Human reviewer posted CHANGES_REQUESTED after bot approved > - PR closed without merge shortly after bot approved > - Bot posted no review despite a `tend-review` run completing on an open PR — first check whether designed silence applies (draft or self-authored PR), then inspect the session logs if it does not. > - Subsequent commits reversed changes the bot approved > - Bot-closed issue was reopened > - Fix commit was reverted or CI still failing after bot pushed > - Human replied to bot with correction or complaint > - Bot comment contains corruption (literal `${`, unescaped bangs, backslash-backticks, broken heredoc markers) > > **Corruption-scan recipe.** Save bot bodies to a file, then scan with `grep`: > > ```bash > mkdir -p "$TMPDIR/bot-output" && : > "$TMPDIR/bot-output/all.txt" > # Issue/PR comments (issue_comment endpoint) > for n in <pr-or-issue-numbers>; do > gh api "repos/$ARGUMENTS/issues/$n/comments?per_page=100" \ > --jq ".[] | select(.user.login == \"$BOT_LOGIN\" and .created_at > \"<window-start>\") | \"=== #$n issue-comment \(.id) ===\n\(.body)\n\"" \ > >> "$TMPDIR/bot-output/all.txt" > done > # Issue bodies (when bot opened the issue this window) > for n in <bot-opened-issues>; do > gh api "repos/$ARGUMENTS/issues/$n" \ > --jq "select(.user.login == \"$BOT_LOGIN\" and .created_at > \"<window-start>\") | \"=== ISSUE #$n body ===\n\(.body)\n\"" \ > >> "$TMPDIR/bot-output/all.txt" > done > # PR bodies (only when bot opened the PR this window) > for n in <bot-opened-prs>; do > gh api "repos/$ARGUMENTS/pulls/$n" \ > --jq "select(.user.login == \"$BOT_LOGIN\" and .created_at > \"<window-start>\") | \"=== PR #$n body ===\n\(.body)\n\"" \ > >> "$TMPDIR/bot-output/all.txt" > done > # PR reviews + inline review comments — any PR the bot reviewed/commented on, not just > # bot-opened. tend-review's output ships on human-authored PRs (the most common surface) > # which would never appear in <bot-opened-prs>. > for n in <pr-numbers-bot-reviewed>; do > gh api "repos/$ARGUMENTS/pulls/$n/reviews" \