review-runs

# Review Runs Analyze the previous night's tend CI runs in this repository. Identify behavioral problems, skill gaps, and workflow issues — then propose improvements to the repo's local skills and workflows. This skill runs **in the consumer repo**, not in tend. Improvements target `.claude/skills/` and `.config/tend.yaml` in this repository. ## First steps Load `/tend-ci-runner:run-tend` first — it contains CI security rules and comment formatting. This skill opens PRs and issue comments, so load `/tend-ci-runner:post-to-github` and `/tend-ci-runner:open-pr` with it. ```bash ls .claude/skills/ ``` Load any repo-specific skill overlay before proceeding. @review-gates.md ## Evidence accumulation Each run contributes to a monthly `review-runs-tracking` issue. Prepare the current tracker and read the current and previous month's evidence: ```bash uv run --script \ "${CLAUDE_PLUGIN_ROOT}/scripts/review_runs.py" prepare-evidence \ > "$TMPDIR/evidence.json" jq -r '.current_comments[].body' "$TMPDIR/evidence.json" > "$TMPDIR/evidence-current.md" jq -r '.previous_comments[].body' "$TMPDIR/evidence.json" > "$TMPDIR/evidence-previous.md" jq -r '"tracker #\(.tracking_number) \(.month)"' "$TMPDIR/evidence.json" ``` The command creates this month's tracker when needed, closes older open trackers, persists the current issue id, and returns both evidence windows. Redirect it rather than reading it inline: an established tracker's two windows run to hundreds of kilobytes, past what a tool result carries. The files are also how you count a finding's prior occurrences. Each entry carries its own session's wording, so grepping for this run's phrasing undercounts: list the finding headings first (`grep -h '^### ' "$TMPDIR"/evidence-*.md`), match on meaning, then read the `## Run` entry behind each candidate. After analysis, write the new findings in the format from `@review-gates.md` to `$TMPDIR/findings.md`. Include a literal `## Run $GITHUB_RUN_ID` heading. Then append them: ```bash uv run --script \ "${CLAUDE_PLUGIN_ROOT}/scripts/review_runs.py" append-evidence ``` The command refuses a findings file that does not name this run. It appends to the latest bot evidence comment while the combined body remains below 60 KB, then starts a new comment. Prior entries are never replaced. ## Step 1: Find recent runs List tend CI runs that completed since the previous `review-runs` run (nominally 24 hours — the cron runs daily): ```bash uv run --script \ "${CLAUDE_PLUGIN_ROOT}/scripts/list_recent_runs.py" review-runs ``` If no runs are found, report "no runs to review", complete **Reconcile live work** below, then exit. Report the run census as the count this returns. `.total_count` counts the wider `FETCH_FROM` fetch, so it bounds the census from above rather than matching it — but a census that lands on a round page boundary (30, 100) is still the signature of a page that was never followed, so check that one against `.total_count` before trusting it. Then, for each run ID from above, pull its jobs and classify them: - **Long-running** (>30 min): Tend runs typically finish in single-digit minutes. Anything over 30 is worth a look — download session logs in Step 3 and diagnose where the time went (long background waits, push-wait-fix cycles, a stuck tool call). - **Near-timeout** (within 90% of the cap): A job that consumed most of its timeout budget is one slow external check away from being killed. Structural, but rate it by what the kill would leave on the record. Usually nothing — a cron-driven run a later tick repeats. It reaches **Critical** where the killed session had already taken an outward action it was still gated on: a `tend-review` job killed mid-poll leaves its approval standing over red CI. To determine the timeout cap for a workflow, read `timeout-minutes` from that workflow's own file under `.github/workflows/` — the census admits workflows named outside the `tend-` prefix, so don't glob for one. Tend's generated workflows do not set `timeout-minutes`, so GitHub's 360-minute default applies unless the consumer has overridden it via `workflows.<name>.jobs.<job>.timeout-minutes` in `.config/tend.yaml`. ```bash # Flag long-running and near-timeout jobs gh api "repos/$GITHUB_REPOSITORY/actions/runs/$RUN_ID/jobs" \ --jq '.jobs[] | ((.completed_at | fromdateiso8601) - (.started_at | fromdateiso8601)) as $dur | select($dur >= 1800) # 30 min | {name, conclusion, duration_min: ($dur / 60 | floor), url: .html_url}' ``` After retrieving the timeout cap from the workflow file, flag any job whose duration exceeded 90% of it as a near-timeout. For the default 360-min cap, that threshold is 324 min. ### Reconcile live work The failed-run census and the `tend-outage` issue diagnose availability. Do not replay historical workflow runs to recover their event payloads: issue and PR recovery belongs to the unread notification queue, which applies the current workflow and current repository state. As a daily backstop for delayed notifications, retention, edited activity, and repaired subscriptions, inspect the live repository for: - an open issue with no bot response to the latest human activity; - an open PR whose live head has no bot review, or whose latest comment, review, or inline review comment directed at the bot has no response; this includes replies to the bot's review on a fork PR; - failing default-branch CI with no bot fix in progress. A live-state check like the two above it: scoped neither to `ci-fix`'s watched workflows — Dependabot security updates, cron releases and doc builds fail there with no PR attached, and nothing else looks for them — nor to this run's window. ```bash uv run --script \ "${CLAUDE_PLUGIN_ROOT}/scripts/red_default_branch_runs.py" ``` `live` holds the red rows that no later green run of the same subject — the same workflow, or for a generated run the same `path` and `name` — closed, newest first. `latest_green_by_path` and `reached_back_to` are the scope the claim rests on: `latest_green_by_path` is closure evidence rather than a set of fixed paths — per path, the newest green read under any subject that carried a red row, whether or not it closed one — so a path can be published with a green that closed nothing, and on a generated path carrying several subjects a row *older* than the published green can still be live because that green closed another name; a path is absent when no green was read under any of its red subjects, which is weaker than the workflow never passing — Dependabot's path is always absent and its workflow does pass — so name the workflows checked from `live`'s paths too. `reached_back_to` is the newest floor among the listings that filled their page, so a row older than it may be missing — `live` can still carry rows older than it, both from an untruncated listing's whole history and from what an earlier read of a truncated one returned from an older window. `null` means no listing was truncated and the sweep covers the branch's whole history. A non-empty `unconverged_listings` means a listing never settled — report that rather than publishing the sweep as complete. The script re-reads each listing until two consecutive answers agree, because the API answers one URL from more than one snapshot, and its two kinds of read fail in opposite directions. A stale red listing drops the *newest* rows, so "`main` is green" can ship while a failure stands on it; a stale closure read serves a green older than the true latest, so a path already fixed reads as still red. Unwindowed on purpose: a failure nobody fixed is still live on the nights after it ran, so anchoring on `$TMPDIR/review-runs-since` would surface each one the night it happened and read as an all-clear afterwards. The listing reaches back weeks, so most rows are already fixed and the closure read is what separates them. What it cannot close stays live: each Dependabot security update's `name` carries a per-update ID that never recurs, so no later run repeats its subject and those rows close only through a fix PR or a tracker. Step 1's census reaches back 49h at most, so skip only the tend rows inside its window — a tend workflow red for longer than that, with no green since, is news here like any other row. Report the scope the claim rests on — "`main` is green" is read later as covering every workflow — naming the workflows checked and how far back the listing reached. - a tend workflow whose queue is dead. A run parked in GitHub's pre-job `waiting` state holds its concurrency group without ever concluding, so under `cancel-in-progress: false` every later tick takes the single pending slot and is replaced by its successor — the workflow stops running and nothing fails. Step 1's census cannot see the parked run, which admits a row only on a non-null `conclusion`, and the replacements it causes read there as ordinary concurrency. ```bash gh api "repos/$GITHUB_REPOSITORY/actions/runs?status=waiting&per_page=50" \ --jq '.workflow_runs[] | {id, name, created_at, html_url}' ``` A tend run still `waiting` after several of its own scheduling intervals — or, on an event-driven workflow, long after the event that created it — is wedged. Every generated workflow but the secretless `tend-mention-relay` carries the `tend` environment, so any of the rest can park. `pending_deployments` on it confirms which kind: `wait_timer: 0` with an empty `reviewers` is an environment gate with nothing left to release it, so `gh run cancel <id>` is the remedy and the pending successor starts. `queued` is a different state and not this shape — `gh run cancel` there answers `Cannot cancel a workflow run that is completed` while the runs API still reports the run `queued`. That is GitHub bookkeeping holding nothing live; leave it rather than fighting it. - an open Dependabot security alert with no PR or tracker proposing its fix — same closure as the red rows above, so an alert whose fix needs a maintainer decision stops re-surfacing once it is tracked. Dependabot opens that PR itself for most alerts, so the ones that reach this sweep are the ones where it could not — and nothing else in tend looks: `weekly` reviews the dependency PRs that exist, and the defining property here is that none was created. ```bash gh api "repos/$GITHUB_REPOSITORY/dependabot/alerts?state=open&per_page=50" \ --jq '.[] | {number, dep: .dependency.package.name, manifest: .dependency.manifest_path, sev: .security_advisory.severity, created_at, fix: .security_vulnerability.first_patched_version.identifier}' ``` The PAT's `repo` scope covers this. Where alerts are disabled the call fails with `403 Dependabot alerts are disabled for this repository.` — that is the check not applying, not a missing scope; only an empty result means no open alerts. Otherwise run it unconditionally. An alert open for more than a few days with no PR naming its package is live work in the same sense as a red default-branch run. A red `dynamic/dependabot/...` row above naming that package is the mechanism: Dependabot is erroring, so waiting will not produce a PR and the manifest or lockfile has to be bumped directly. `gh run view <id> --log-failed` ends with an error table naming the dependency and the error type; `security_update_not_possible` also reports the lowest non-vulnerable version beside the highest the dependency tree currently resolves, which is the constraint to relax. Handle live work through the normal triage, review, or CI-fix instructions. Keep failed runs in the report as diagnostic evidence. After the exhaustive live scan, read every row on every outage tracker this sweep still owes a drain. Each tracker's `rows` holds them in order: the issue body carries the first, comments the rest. Fail the sweep if the script exits non-zero; that is different from it returning no trackers: ```bash uv run --script \ "${CLAUDE_PLUGIN_ROOT}/scripts/review_runs.py" outage-trackers \ > "$TMPDIR/review-runs-outage.json" ``` It returns the open trackers plus any closed since Step 1's anchor by someone other than the bot, since only this drain closes a drained tracker. Read the closed ones too: the live scan above reads current repository state, where a merged PR whose review died in the outage is indistinguishable from one the maintainer merged without waiting. Use every row to identify what the failed run may have missed. Diagnose it and handle any applicable current work. Then close the trackers still open; one someone else already closed stays closed: ```bash jq -r '.trackers[] | select(.state == "OPEN") | .number' \ "$TMPDIR/review-runs-outage.json" \ | while read -r number; do gh issue close "$number" --reason completed; done ``` ## Step 2: Token usage report Run the token report script to get per-run token counts: ```bash # Step 1's own anchor, so the spend prices exactly the band the census counts: # the script fetches with a cushion and admits a run on completion, the way # Step 1 does. A window in hours would drop a run that started before the # anchor and finished inside it — the longest and costliest runs there are. uv run --script \ "${CLAUDE_PLUGIN_ROOT}/scripts/token_report.py" \ --since "$(cat "$TMPDIR/review-runs-since")" \ > "$TMPDIR/token-report.json" ``` Pass the same extra prefixes Step 1 censuses, as positional arguments after `--since`, so the two steps agree on what the fleet is — the repo's `running-tend` skill is the source for both, naming any workflow that uses the tend action but isn't named `tend-*`. Include the total cost and the per-workflow breakdown in the summary (Step 7). Escalate outliers to Step 3 — for example a run far above its workflow's usual cost, or a subject the subject table shows several runs against. ## Step 3: Download and analyze session logs Load `/tend-ci-runner:read-session-logs`, which picks the runs and sends you to `/install-tend:debug-tend-run` for the download commands and JSONL parsing queries. Skip runs without artifacts. Trace decision chains: what did tend decide, what evidence did it use, what was the outcome? ## Step 4: Cross-check outcomes For each analyzed run, compare what the bot did against what happened next. The same "did it stick?" question applies to every tend workflow — ask it of whatever ran. For example: - **Review**: did subsequent commits undo something the bot approved? Did human reviewers flag issues it missed? - **Triage**: was the classification correct? Did the issue get relabeled? - **Nightly**: did the bot's PRs merge, or get closed as unhelpful? - **CI-fix**: did the fix actually resolve the failure? mention, notifications, and weekly runs get the same treatment: find the bot's output and check whether it was accepted. Dispositions — merged, closed, relabeled, reverted — are only half the signal. A maintainer replying in-thread that a bot claim was wrong, or requesting changes on a bot PR, leaves labels and state untouched and is equally a correction; where the bot authors most of the PRs, a review body is the *first* place a maintainer writes. The script collects all three — dispositions, thread comments, review bodies — for the window: ```bash uv run --script \ "${CLAUDE_PLUGIN_ROOT}/scripts/review_runs_corrections.py" \ "$(cat "$TMPDIR/review-runs-since")" ``` Read every row: a correction is a maintainer contradicting a bot claim, not merely replying. Every comment and review row names its `author` — prose style doesn't decide, since a maintainer can post through an agent; an empty `author` is a deleted account, never the bot. Comment rows carry both timestamps because the window filters on `updated_at` — a `created` before the anchor is an older comment edited inside the window, a real hit rather than a broken filter. Empty `dispositions`, `comments`, and `reviews` is the all-clear. Write "no maintainer corrections" into the tracking issue only after the script ran and returned empty — future runs read the phrase as ground truth when counting occurrences under Gate 1, so an unchecked all-clear suppresses the evidence it exists to accumulate. The script exits non-zero rather than reporting an empty window when the anchor or the bot login is missing, since both filters fail open. ## Step 5: Deduplicate Before creating issues or PRs, check for existing ones: ```bash gh issue list --state open --limit 200 --json number,title,body gh issue list --state closed --json number,title,closedAt --limit 200 # --state all: a merged PR is the most common way a finding is already fixed gh pr list --state all --limit 200 --json number,title,state # Bundled-skill defects are filed upstream (Step 6), and the queries above only # see this repo — dedup against tend before filing there. gh pr list --repo max-sixty/tend --state all --limit 200 --json number,title,state gh issue list --repo max-sixty/tend --state all --limit 200 --json number,title ``` Search the titles for related keywords, then read the bodies of the candidates (`gh pr view <n> --json body`). Your workflows call a pinned action ref, so a skill fix merged upstream stays dormant here until the next release tags. Observing the bug is therefore not evidence the fix is missing: read these results before filing, or the report is churn on something already landed. ## Step 6: Act on findings Improvements target **repo-local** files by default: - **`.claude/skills/`** — update or create skill overlays with instructions that prevent the identified problem. Prefer updating existing skill files over creating new ones. - **`.config/tend.yaml`** — adjust workflow configuration if the problem is structural (e.g., wrong cron schedule, missing setup step). - **Project instruction file (`CLAUDE.md` or `AGENTS.md`)** — add project-specific instructions if the problem is about code conventions or patterns the bot keeps getting wrong. **Bundled-skill defects.** If the root cause is a gap or bug in a bundled skill (`plugins/tend-ci-runner/skills/...` in `max-sixty/tend`) — the same pattern would fire in every consumer — file the fix against tend per `/tend-ci-runner:act-in-other-repos`. Signal: the fix reads as generic instructions that would apply to any consumer. **Prefer PRs over issues.** A PR with a clear description is immediately actionable. Editing `.claude/skills/` requires the read-only-mount workaround (bind-mounted read-only, plus a harness write-guard on `.claude/skills/` paths) — see `/tend-ci-runner:propose-instructions`. Adapted for review-runs (base on `HEAD` since this runs on a schedule, not a PR checkout; move each edited file into place): ```bash git worktree add "$TMPDIR/review-runs-fix" -b daily/review-runs-$GITHUB_RUN_ID HEAD # Author each edited skill file at $TMPDIR/<name>.md. # Then move the files into place. Both `cd`s stay inside subshells, so the # session's own cwd never enters the worktree: the last line deletes it, and a # session standing in it has no working directory for Step 7 or anything after. ( cd "$TMPDIR/review-runs-fix/.claude/skills/running-tend" && mv "$TMPDIR/running-tend.md" SKILL.md ) # Repeat per skill file being updated. ( set -e cd "$TMPDIR/review-runs-fix" git add .claude/skills/ git commit -m "skills(running-tend): ..." git push -u origin daily/review-runs-$GITHUB_RUN_ID gh pr create --title "..." --body-file "$TMPDIR/pr-body.md" --head daily/review-runs-$GITHUB_RUN_ID ) && git worktree remove "$TMPDIR/review-runs-fix" --force ``` `.config/tend.yaml` and project instruction files are not under the read-only mount, but if you're already in the worktree for a `.claude/skills/` edit, do those edits there too so the branch stays self-contained. - **PR** (default): Branch `daily/review-runs-$GITHUB_RUN_ID`, fix, commit, push, create with label `review-runs`. Write the description for a maintainer deciding whether the current change fixes the general behavior gap, following **Reader-facing prose** in `/tend-ci-runner:run-tend`. Link the tracking issue where it holds prior observations of the same behavior, and carry the evidence that justified promoting this finding — the run IDs, the log excerpt, and the gate assessment — in the body or a `<details>` block. - **Issue** (fallback): Only for problems too large or ambiguous to fix directly. **Limit to a couple of PRs per run.** Pick the highest-confidence findings; note the rest in the tracking issue. ## Step 7: Summary If no problems found (or none passed the gates), report "all clear" with: runs analyzed, sessions reviewed, brief quality assessment, and any below-threshold findings recorded in the tracking issue. Save the summary to `$GITHUB_STEP_SUMMARY` (a later workflow step copies this into the GitHub Actions step summary): ```bash cat > "$GITHUB_STEP_SUMMARY" << 'EOF' ## Review-runs summary ... EOF ```