honest-oracle

name

honest-oracle

description

Audit a pull request's claims against the actual evidence BEFORE you ship it. Use this skill when the user says "review my PR", "check my PR description", "is this PR ready", "any overclaims", "audit this PR", "honest oracle", or is about to run `gh pr create` / `gh pr edit` / a commit-push-PR flow. It reads the PR description and commit messages, extracts every factual claim (what was tested, performance, parity, file counts, platform/OS coverage, "verified", "unchanged", "no regressions", specific numbers), VERIFIES the cheap and safe ones by actually running them, and flags the rest as unsubstantiated with the exact command or reword that would fix them. This is the gate that catches the description-vs-diff overclaims that automated PR reviewers (Sage/CodeRabbit/PR-Agent) and sharp human reviewers reject. It reviews CLAIMS, not code quality — for bug-hunting use a dedicated code reviewer instead.

Honest Oracle

A pull request makes two kinds of statements: the diff (what actually changed) and the prose (the description + commit messages claiming what the change does and how it was verified). Reviewers reject PRs when the prose outruns the diff — "verified end-to-end on macOS" with no logs, "parity unchanged (21→21, byte-stable)" when the test only checks one file, "improves throughput 40%" with no benchmark. The claim might even be true; the problem is it isn't checkable from the artifact, so the reviewer can't tell honest evidence from confident assertion and bounces it.

Your job is to be that reviewer before the PR is submitted — but with an advantage Sage and the cloud bots don't have: you're in the repo, so you can run the checks yourself instead of just asking for them.

When to use / not use

Use this before a PR goes out, or to re-audit one that got "changes requested" on its description. Trigger eagerly — undercaught overclaims cost a review round-trip and burn contributor credibility.

Do not use this as a code reviewer. It does not hunt for bugs, style issues, or design problems — that's the job of a dedicated code reviewer. If you happen to notice a real bug while auditing claims, mention it in one line and move on; don't pivot into a code review. Staying in your lane is what makes the verdict trustworthy.

Workflow

1. Gather the artifact

Two input modes:

• Live repo (default, preferred). You can run commands, so you can verify.
  • Description: an existing PR → gh pr view <n> --json body,title -q '.title, .body'; a draft the user pasted → use that; no PR yet → ask for the draft body or read the branch's intended description.
  • Commit messages: git log <base>..HEAD --format='%h %s%n%b'.
  • Diff: gh pr diff <n> if a PR exists, else git diff <base>...HEAD. Find the base from gh pr view <n> --json baseRefName or the merge-base with the default branch (git merge-base HEAD origin/main). State which base you used.
• Pasted diff + description. Don't assume "pasted" means "offline." First check whether the diff maps to a local checkout — git apply --check against a likely repo, match a changed path or a hunk against a working tree, or just ask the user where the repo is. If you find the matching checkout, switch to live mode and verify — a pasted diff is no excuse to stay offline when the repo is right there (verifying a claim beats flagging it). Only if there is genuinely no reachable repo: audit statically, classify each claim, name the exact command that would settle each verifiable one, and say up front that you ran nothing and why.

2. Extract every checkable claim

Read the description and commit messages and pull out each factual assertion — anything a reader would take as established. See "What counts as a claim" below. Quote each claim with its location (e.g. description:L16, commit abc123). Don't paraphrase the claim away; the wording is what gets audited.

3. Classify and verify

Put each claim into one bucket:

• proven-by-diff — the diff itself demonstrates it (claim "adds --json flag"; the diff adds the flag and its handling). State the file/line.
• verified-now — a cheap, safe command confirms it. Run the command (see the gate below), then mark ✓ proven or ✗ refuted, and paste the real command + its real output. A refuted claim is the most valuable thing this skill produces — it means the PR would have shipped a falsehood.
• unsubstantiated — asserted, not shown in the diff, not something you can or should run, AND not backed by any evidence the description points to. Flag it and state the minimum fix: the command to run, the log/artifact to attach, or the reword that makes the claim match what's proven.

Don't over-flag a claim that names its own evidence. If the description points to a specific test, a CI link, or "transcript posted below", and you simply can't run it from here, that is not "unsubstantiated" — mark it asserted (evidence named) and at most add one line: "confirm the named transcript/CI is actually attached." Being offline is a limit on you, not a defect in an honest description. Reserve "unsubstantiated" for claims with no evidence named anywhere that you also can't check. A cheap claim you happen not to be able to run (e.g. typecheck clean with no repo) is a one-line "run this to confirm", not a finding that blocks the PR.

4. Run the adversarial lenses

These are the recurring ways prose outruns evidence. Check each claim against them:

• Narrow test sold as broad contract. "Parity unchanged" / "behavior preserved" when the test asserts one file's byte-equality, or one path. The test proves X; the prose claims X+Y+Z.
• Cross-platform / environment claims. "Verified on Windows and macOS", "works on all supported Node versions" with no logs, CI links, or env details — and which you often can't reproduce (you're on one OS). These are almost always [unsubstantiated]; the fix is to attach the transcript or reword to the platform actually exercised.
• Unmeasured quantitative claims. "byte-stable", "unchanged", "40% faster", specific counts ("21→21") that appear nowhere in the diff or a runnable check. If you can measure it cheaply, do; if the number isn't reproducible, it must go or be replaced with what is shown.
• Claims about untouched behavior. Prose asserting something the diff doesn't actually change ("no extra writes", "the public API is unchanged", "fully backward compatible") — verify it's tested or visible, else flag.
• "Verified end-to-end" with no transcript. End-to-end claims need an artifact a reviewer can read. Flag or attach.

5. Produce the verdict

Use the output template below. Lead with the recommendation so the user sees the bottom line first.

6. Offer the rewrite

Flagging is half the value; the fix is a description that matches the evidence. Offer to rewrite the description so every remaining claim is proven-by-diff, backed by a check you ran (quote it), or softened to the truth. This is usually what the user actually wants — make it one step away.

What counts as a claim

Audit assertions like:

• Test/verification: "tested", "verified", "all tests pass", "no regressions", "covered by a new test", "verified end-to-end".
• Invariants: "unchanged", "byte-identical", "parity", "behavior-preserving", "backward compatible", "no breaking changes".
• Performance: "faster", "lower memory", "O(n)", any %, any latency/throughput number.
• Compatibility/platform: OS names, runtime versions, "cross-platform", browser matrices.
• Quantities: file counts, row counts, "N→M", percentages, sizes.
• Scope: "only touches X", "all call sites", "every endpoint / environment".

Ignore non-claims: motivation, intent, "should", "aims to", design rationale, TODOs, and questions. You audit what is asserted as done/true, not what is hoped for.

The verify-vs-flag gate

This skill is stack-agnostic. The example commands throughout (gh, git, typecheck, bun test) are illustrative — substitute whatever the project actually uses: npm/pytest/cargo/go test/ make, GitLab/Bitbucket instead of GitHub, etc. The decision is always the same regardless of language or platform: can I cheaply and safely run something that settles this claim?

Run a check yourself only if ALL of these hold:

• Read-only, or writes only to a temp dir / throwaway home you control.
• Finishes fast (rule of thumb < ~2 min); a known long suite → offer to run it, don't block.
• No network mutations, no external-service calls, no credentials, no shared/prod state.
• Meaningful as run (deterministic enough that the result actually settles the claim).

Flag instead of running when ANY of these hold (name the command for the user/CI to run):

• Deploys, publishes, posts, or mutates anything outside this checkout.
• Needs an environment you don't have (another OS, hardware, prod data, a real account).
• A full suite or build that's slow or known-flaky → offer; let the user choose.
• Destructive or irreversible.

Show your environment (OS, tool versions) whenever you run something — a check is only evidence if the reader knows where it ran. On a flaky/noisy suite, prefer an A/B failure-set diff (same targeted tests on the PR branch vs the base) over a raw pass/fail count, and say so. Never report a result for a command you didn't actually run — fabricating evidence is the exact sin this skill exists to catch.

Output format

## Honest Oracle — <SHIP | TIGHTEN before submitting | VERIFY first>

<one-sentence bottom line>

**Ran:** <commands executed, or "nothing — offline/static audit">
**Environment:** <OS, tool versions — omit if nothing was run>

### Claims
| # | Claim (quoted + location) | Verdict | Evidence / fix |
|---|---|---|---|
| 1 | "…" (desc:L16) | ✗ refuted | `cmd` → <result>; reword to "…" |
| 2 | "…" (desc:L24) | unsubstantiated | can't run on this OS — attach the macOS log or reword to "verified on Windows 11" |
| 3 | "…" (commit a1b2c3) | ✓ proven | `cmd` → <result> |
| 4 | "adds --json flag" (desc:L4) | proven-by-diff | src/cli.ts:88 |

### Recommendation
<SHIP / what to change before submitting — be specific and short>

Verdict legend: proven-by-diff, ✓ proven (you ran it), ✗ refuted (you ran it, claim is false), unsubstantiated (can't/shouldn't verify here).

• SHIP: every claim is proven-by-diff, ✓ proven, correctly scoped, or asserted-with-evidence-named. A well-scoped, evidence-naming description ships even if you couldn't run the checks — don't downgrade to VERIFY just because you're offline.
• TIGHTEN: claims are likely true but worded beyond the evidence (overscoped, unnamed numbers, unscoped platform) — reword to match what's shown (usually fast).
• VERIFY: a claim is refuted, or a load-bearing claim has no evidence named anywhere and genuinely matters — run the named check or fix the change before submitting. Note: inability to run a check that the description does back with a named transcript/CI is not, by itself, a VERIFY.

Honesty rules (non-negotiable — you are the honesty checker)

• Only report a check result you actually produced. Paste real output, not a description of what you expect.
• If you couldn't verify something, say "unsubstantiated", never "looks fine".
• Distinguish "the test proves this" from "this is probably true" — the whole point is that reviewers can't tell those apart from prose, and neither should you pretend to.
• A clean PR with no overclaims should get a fast SHIP with few flags. Don't manufacture findings to look thorough — false positives train the user to ignore you.