verify-vuln-issues

name

verify-vuln-issues

description

Triage the repo's open "[Vulnerability]" GitHub issues — for every issue lacking a true-positive/false-positive label, fetch the referenced commit, judge whether it is a genuine security fix or a false positive, post an analysis comment, and apply the verdict label. Pure dependency bumps get a `dependency` label and are closed. Use when asked to "verify", "triage", or "review" the vulnerability issues / findings in the tracker.

Verify vulnerability issues

This repo's monitoring action files a GitHub issue for every commit it thinks is a security patch. Many are genuine; some are false positives (routine dependency bumps, feature commits, build-tooling changes, AI-misclassified commits). This skill verifies the unverified ones, comments, and labels each.

Scope & disposition

• Target: open issues authored by github-actions[bot] that carry none of the verdict labels — true-positive, false-positive, or dependency. An issue carrying any of those, or one already closed (e.g. a dependency bump closed in an earlier run, or a duplicate), has already been dispositioned — skip it. prepare.mjs applies exactly this filter.

• For every targeted issue: post one analysis comment, then apply the disposition:

  Finding	Action
  Genuine security fix	comment + add true-positive label
  False positive	comment + add false-positive label
  Pure dependency bump	comment + add dependency label + close (not planned)

• Pure dependency bump = the commit's entire diff is package manifests, lockfiles, or version changelogs — no hand-written fix code. Close it regardless of whether the upstream advisory is real; dependency bumps are out of scope for this tracker.

  • Exception — keep it open: if the commit also carries actual vulnerability-fix code (e.g. it backports/cherry-picks specific fix code into vendored sources, not merely a version number/lockfile entry), it is a real finding — treat it as Verified (true-positive), do not close it.

• Duplicate issues: the monitor sometimes files more than one issue for the same commit. prepare.mjs detects these (writes /tmp/vsa/duplicates.json). Keep and verify the canonical one (lowest issue number = first filed); the rest are duplicates — close them as duplicates, or delete with gh issue delete only after the user confirms, since deletion is permanent.

• Labels: add the verdict label. Never remove a label someone else applied — if an issue carries a conflicting one, stop and surface it. You may correct a verdict this skill applied earlier (e.g. true-positive → false-positive) when re-analysis warrants: remove the stale label, add the right one, and post a correction comment explaining why.

• Relay a final summary (counts per repo, false positives, issues closed as dependency bumps, duplicates found).

Prerequisites

gh must be authenticated. gh auth status reporting "not logged in" inside the sandbox is normal for git operations, but gh api/gh issue calls still need a real token. If gh calls fail with HTTP 401/Bad credentials, ask the user to run ! gh auth login in the prompt (the dummy-GH_TOKEN trick does not work — the proxy rejects it).

Ensure the dependency label exists (create once if missing):

gh label create dependency --repo <owner/repo> \
  --description "Issue is only a dependency/version bump — out of scope, ignored" \
  --color ededed

Procedure

1. Fetch & prepare

node .claude/skills/verify-vuln-issues/prepare.mjs

It lists issues, filters to unverified ones, parses each body for its repo + full commit SHA (via scripts/lib/parser.mjs), downloads every commit diff and metadata, and writes /tmp/vsa/:

• todo.json — [{number, state, repository, commitSha, ...}]
• ctx/<n>.txt — per-issue: the AI claim + commit message + file list + diff (lockfiles trimmed, huge files truncated)
• duplicates.json — groups of issues sharing one commit SHA (see "Duplicate issues" above)
• results.json — starts {}; record verdicts here as you go.

The --- FILES CHANGED --- list in each context file is what you use to spot a pure dependency bump (only package.json / *.lock / go.mod / go.sum / CHANGELOG* / doc/changelogs/* / toolchain files).

2. Analyze, in batches

Read ctx/<n>.txt files (~10–15 per turn; read large ones individually). For each issue decide the finding using the rubric below, and record into results.json (verified / false-positive / dependency) so progress survives compaction.

3. Comment + label (+ close)

Write each comment to /tmp/vsa/comments/<n>.md, then per issue:

gh issue comment <n> --repo <owner/repo> --body-file /tmp/vsa/comments/<n>.md
# verified:
gh issue edit <n> --repo <owner/repo> --add-label true-positive
# false positive:
gh issue edit <n> --repo <owner/repo> --add-label false-positive
# pure dependency bump:
gh issue edit <n> --repo <owner/repo> --add-label dependency
gh issue close <n> --repo <owner/repo> --reason "not planned"

Loop over a batch; report any failures.

Verdict rubric

The commit diff is ground truth — the issue body is an LLM guess and is often wrong about the mechanism, severity, or even the vuln class.

Verified — a genuine security fix (true-positive)

• Commit message cites a CVE-ID, GHSA advisory, HackerOne report, an internal vuln tracker (GL-Vuln:), or a [security] tag.
• Hand-written code change that removes a real, demonstrably exploitable flaw (memory safety, auth/authz check added, injection/CRLF/XSS sink fixed, DoS guard), ideally with a regression test. A guard that closes a demonstrated bypass still counts even when low-severity — but a change that only addresses a theoretical concern does not; see "Preventive hardening" under False positive.
• A vendored-dependency update that backports/cherry-picks specific fix code for a vulnerability (not just a version number) — even though it touches deps/, it carries the actual patch, so keep it open.
• Still verify even if the issue mislabeled the class — say so and correct it (e.g. "this is request smuggling, not SSRF"). Note overstated severity.

Pure dependency bump (dependency, then close)

• The whole diff is package manifests / lockfiles / version changelogs — no fix code in the project itself. The fix, if any, lives entirely upstream.
• Applies whether the issue is otherwise "verified" (real upstream advisory) or a false positive — either way it is dependency noise, label dependency + close.
• Includes vendored-dependency version bumps (re-vendoring a bundled dep to a new release). But see the Verified exception: a vendored security backport that patches specific code stays open.

False positive — not a substantiated vulnerability (false-positive)

• Build/dev tooling: indirect or dev dependency edits under tools/, hack/, devDependencies, or a maintainer-only script — no runtime/attacker surface.
• Feature, not a fix: the "vulnerable" code and its validation/guard are introduced in the same commit (no prior released version was vulnerable); or the commit just wires up / enables an in-development feature.
• Mischaracterized: the issue's described mechanism contradicts the diff (e.g. claims "no cert validation" but pre-patch used standard TLS; claims prototype pollution but __proto__ was already handled).
• Correctness bug with no attacker: internal concurrency/overflow/data- integrity bugs (build engines, ORM dedup, migration races) with no security boundary or attacker-controlled input.
• Preventive / defense-in-depth hardening: the change hardens against a theoretical or downstream weakness with no demonstrated exploit in a released version — e.g. giving an object a null prototype to address prototype-pollution concerns, or a guard the commit frames as precautionary. Tell-tale: the commit message hedges ("harden", "defense-in-depth", "concerns") and cites no CVE / advisory / PoC showing the prior version was actually exploitable. (Contrast: a guard that closes a demonstrated bypass is Verified, even if low-severity.)
• Docs-only change — unless it reflects a real default-hardening shipped elsewhere.

When unsure, state the caveat explicitly (dependency bump → flaw is upstream; dev-only reach; severity overstated) rather than forcing a binary call.

Comment format

Start with a bold verdict line, reference the short commit SHA, reason from the diff, then a one-line Verdict:. Keep it substantive but tight:

## Verification analysis — **Verified** (genuine <class> fix)

Reviewed commit `<sha>`. <what the diff actually does and why it is/ isn't a
real fix; cite advisory refs, tests, caveats>.

Verdict: verified — <one line>.

If an issue already has a CVE in its commit message but no cve: label, mention it (it would graduate the finding from "verified" to "confirmed").