| name | renovate-pr |
| description | Test and assess an open Renovate dependency-bump PR. Picks the first open Renovate PR, checks out the branch, starts the app, exercises code paths affected by the upgraded package, reviews the changelog and (if needed) the upstream source diff, and reports whether the bump is safe to merge. Use when asked to "test a renovate PR", "triage renovate", "assess a renovate bump", or "check a dependency upgrade". |
Renovate PR Triage
Pick the first open Renovate PR, run the app against it, exercise the affected code paths, and tell the user whether the bump is safe.
Iron Law
The recommendation MUST be evidence-backed. "Looks fine" is not a recommendation. Each verdict must cite: changelog read, codebase usage grepped, and at least one runtime check (page loaded / endpoint hit / log inspected). If you can't gather evidence for an area, say so — never paper over it.
Phase 0: Pick the PR
Find open Renovate PRs and take the first one:
gh pr list --state open --json number,title,headRefName,author,createdAt --limit 100 \
| python3 -c "
import json, sys
prs = json.load(sys.stdin)
renovate = [p for p in prs if p['author']['login'] == 'app/lightdash-renovate-bot' or p['headRefName'].startswith('renovate/')]
renovate.sort(key=lambda p: p['createdAt'])
if not renovate:
print('NONE')
else:
p = renovate[0]
print(f\"{p['number']}\t{p['headRefName']}\t{p['title']}\")
"
If NONE, stop and tell the user there are no open Renovate PRs.
Otherwise, take the first one as $PR_NUMBER and report to the user:
Triaging Renovate PR #<number>: <title>
Branch: <branch>
Phase 1: Identify the dependency change
gh pr view $PR_NUMBER --json title,body,headRefName,additions,deletions,files,labels
gh pr diff $PR_NUMBER
Renovate PR bodies always contain a markdown table that looks like:
| Package | Change | Age | Confidence |
|---|---|---|---|
| [pkgname](homepage) ([source](https://github.com/owner/repo)) | [`1.2.3` → `2.0.0`](renovatebot.com/diffs/...) | ... | ... |
Parse this to extract for each package:
- Name (e.g.
nodemailer)
- Old version → New version
- Bump type: patch (z), minor (y), major (x) — derive from semver
- Source repo URL (the
[source] link, e.g. github.com/owner/repo) — needed for changelog & source-diff lookup
- Is this a security advisory? (label
security on the PR, or [security] in the title)
Show the user the parsed list before proceeding:
Dependencies in PR #<number>:
1. nodemailer 7.0.13 → 8.0.5 (MAJOR, security) github.com/nodemailer/nodemailer
2. ...
Phase 2: Look up changelog & release notes
For each package, fetch the upstream release notes spanning oldVersion..newVersion. Try in order, stop at the first that yields useful content:
- GitHub Releases (best for most JS packages):
gh api "repos/<owner>/<repo>/releases?per_page=100" --jq '.[] | select(.tag_name | test("<oldVersion>|<newVersion>")) | {tag: .tag_name, name: .name, body: .body}'
Or list releases between the two tags:
gh api "repos/<owner>/<repo>/compare/v<oldVersion>...v<newVersion>" --jq '{commits: .commits | length, ahead_by: .ahead_by}'
- CHANGELOG.md in the repo via WebFetch:
https://raw.githubusercontent.com/<owner>/<repo>/<default-branch>/CHANGELOG.md
- Renovate's own diff page (linked in the PR body):
https://renovatebot.com/diffs/npm/<pkg>/<oldVersion>/<newVersion>
- WebSearch as last resort:
"<package-name> <oldVersion> <newVersion> breaking changes"
For each package, extract:
- Breaking changes — explicit
BREAKING: entries, removed APIs, behavior changes
- Security fix details (if security advisory) — what CVE, what attack vector
- New required configuration — env vars, options that must be set
- Deprecations that may affect us soon
Skip noise (typo fixes, internal refactors, doc-only changes).
Phase 3: Map upgraded package → our usage
For each package, find every place we use it in this monorepo:
grep -rEn "(from ['\"]<pkg-name>['\"]|require\(['\"]<pkg-name>['\"]\))" packages/ --include='*.ts' --include='*.tsx' --include='*.js'
grep -rEn "\"<pkg-name>\":" packages/*/package.json
Categorise hits:
- Direct API consumers — code that calls into the package
- Transitive only — appears in lockfile but no direct imports (much lower risk surface, but still worth confirming)
- Type-only imports —
import type { ... } (compile-time only)
- Test fixtures — usage only inside
*.test.ts / *.spec.ts
Hold this list — Phase 6 will exercise the user-facing flows that touch these files.
Phase 4: Upstream source diff (only if needed)
If Phase 2's changelog is vague, missing, or claims "no breaking changes" but a major version was bumped, drop down to the source diff:
gh api "repos/<owner>/<repo>/compare/<oldTag>...<newTag>" \
--jq '.files[] | select(.filename | test("^(src|lib|index)") and (test("test|spec") | not)) | {filename, status, additions, deletions}'
For files that look load-bearing for our usage in Phase 3, fetch the patch:
gh api "repos/<owner>/<repo>/compare/<oldTag>...<newTag>" --jq '.files[] | select(.filename == "<file>") | .patch'
Read the diff and check whether any API our codebase calls has changed signature, behavior, or default values.
Don't read the entire upstream diff blindly. Use the codebase-usage list from Phase 3 to target only files that match the APIs we actually call.
Phase 5: Start the app on the PR branch
5a. Decide: worktree or in-place checkout?
Check your Claude memory (and any project- or user-level instruction files you've been given) for a stated preference about worktrees for this project. The lookup is environment-agnostic — use whatever memory or instructions surface for you in this session.
If a worktree preference is found, follow whatever workflow that preference describes. The branch name to use is the PR's headRefName:
gh pr view $PR_NUMBER --json headRefName -q .headRefName
After the worktree is created/entered, cd into it before continuing with the rest of Phase 5. Renovate branches usually contain a / (e.g. renovate/npm-foo-vulnerability) — preserve the name as-is; don't sanitize it.
If no worktree preference is found, stay in the current directory and check out the branch in place:
gh pr checkout $PR_NUMBER
Report to the user which mode you picked and the source of the preference (or that none was found).
5b. Regenerate lockfile if missing
git status pnpm-lock.yaml
sfw pnpm install
5c. Start the dev stack
The running app is what we're testing against, not just the diff:
/docker-dev start
Wait for the State Detection to report all OK: lines and PM2 processes to be online. If the build or PM2 startup fails, that's the first signal — the bump may have broken the install or runtime resolution. Report this immediately and stop.
When using a worktree, /docker-dev start claims a fresh port slot for this instance (per the docker-dev port-allocation flow), so it won't conflict with another running Lightdash instance in the main checkout or another worktree.
Phase 6: Test the change with /debug-local tooling
Use the /debug-local skill workflow — but inverted. Instead of investigating a known symptom, we're fishing for symptoms in the code paths that touch the upgraded package.
For each package, design 1–3 focused checks based on what the package actually does. Examples:
| Package category | What to exercise |
|---|
Email (e.g. nodemailer) | Trigger an invite or password reset; verify the email lands in Mailpit (http://localhost:8025) |
Auth / OAuth (e.g. @node-oauth/oauth2-server) | Login flow via demo@lightdash.com / demo_password!; OAuth client flow if applicable |
HTML sanitization (e.g. sanitize-html) | Render a markdown tile / dashboard description containing rich content |
Rich text editor (e.g. @tiptap/*) | Open a page that uses the editor (dashboard tile description, comment) and type into it |
Translation / i18n (e.g. i18next-locize-backend) | Switch language; verify strings render and no console errors |
Warehouse drivers (pg, snowflake-sdk, etc.) | Run a query via curl -H "Authorization: ApiKey $LDPAT" "$LIGHTDASH_API_URL/api/v1/projects/<uuid>/explores" and against the SQL runner |
Frontend UI library (@mantine/*, react-*) | Open the dashboard view; check for console errors and visual regressions |
For each check, use these tools in parallel:
What counts as a failure signal:
- Stack traces or unhandled rejections in PM2 logs
- Spotlight errors with timestamps after we triggered the flow
- HTTP 500s on previously-working endpoints
- Console errors in the browser that weren't there on
main
- Behavior change visible to the user (e.g. an email body now missing headers, an editor that won't accept input)
What is NOT a failure:
- Pre-existing errors unrelated to the bumped package
- Deprecation warnings that don't affect behavior
- Successful response with different but valid output shape (note it, but don't fail on it)
If a test fails: stop, capture evidence (log lines, trace ID, screenshot), and move that finding to the report. Continue testing the other packages — one failure doesn't invalidate triage of unrelated bumps.
Phase 7: Verdict
Score each package independently using this rubric, then roll up to an overall PR verdict.
| Verdict | Criteria |
|---|
| 🟢 SAFE | Patch or minor bump, no breaking changes in changelog, no usage requires changes, runtime checks pass cleanly |
| 🟡 LIKELY SAFE | Minor/major bump, changelog has breaking changes but none touch APIs we use, runtime checks pass, recommend a quick human glance at the affected area |
| 🟠 NEEDS CODE CHANGES | Breaking changes affect our usage — list the call sites that must be updated before merge |
| 🔴 UNSAFE / BLOCKED | Runtime check failed, install failed, or the change clearly breaks a flow |
| ⚪ CANNOT ASSESS | Couldn't run the app, no test path for this package, or changelog missing and source diff too large to read meaningfully — escalate to user with what blocked you |
Output to the user (and only to the user — do NOT post a PR comment unless they explicitly ask):
RENOVATE PR TRIAGE — #<number>
════════════════════════════════════════
Title: <PR title>
Branch: <branch>
Bump type: <patch|minor|major> [security]
Packages
────────
1. <pkg> <old> → <new> Verdict: 🟢|🟡|🟠|🔴|⚪
Changelog: <link or summary>
Our usage: <count> direct imports across <N> files
Tested: <what flow you exercised>
Evidence: <log line, trace ID, or screenshot path>
Notes: <breaking changes that mattered, or "none">
2. ...
Overall verdict: 🟢|🟡|🟠|🔴|⚪
Recommendation: <merge / merge with quick review / fix code first / do not merge>
════════════════════════════════════════
Phase 8: Cleanup
Leave the app running unless the user asks otherwise — they may want to poke at it further.
- If you used a worktree (Phase 5a): leave it in place. Do not delete it automatically — that destroys evidence the user may want to inspect. Tell them where the worktree lives so they can return to it or clean it up later.
- If you checked out in place: return to the original branch with
git checkout -.
If you regenerated pnpm-lock.yaml and committed it during Phase 5, note that in the final report so the user knows there's a new commit on the PR branch.
Notes
