pr-trajectory-audit

name	pr-trajectory-audit
description	Two connected workflows that audit a repo's PRs against its OWN documented AI layer (rules, skills, workflow command definitions) -- not generic code-quality judgment. Workflow A (audit, run via `/pr-trajectory-audit`) reads the repo's own rules/workflow docs, mines PR history for evidence of compliance or violation against them, writes a rubric, installs the live-review GitHub Action, and tells the user the remaining one-time GitHub setup steps. Workflow B (live review) runs automatically on every new PR via CI: deterministic checks first, then Claude reads the PR's actual diff/body against the rubric and posts a review citing the specific documented rule involved, then a human reads that review as part of normal PR review. Findings feed system evolution -- fixing the AI layer, not the one PR. Use when the user wants to "audit my PR history", "check if my agent follows our workflow", "build a trajectory rubric", "review this PR for AI-layer compliance", "pr-trajectory-audit", or is setting up automatic trajectory review on pull requests. This is NOT a general code review or security scan, and NOT a judgment of whether a fix is good code -- it evaluates the PROCESS: did the agent's trajectory actually follow the workflow, rules, and conventions defined for it.
argument-hint	[owner/repo]

PR Trajectory Audit

Evaluate a coding agent's (or team's) work the same way you'd evaluate any agent run: the PR is the trajectory, and the trajectory gets checked against the AI layer that was supposed to govern it -- the rules, skills, and workflow command definitions the repo already has, not an inferred sense of "good code." The question is never "is this fix good" -- it's "did the agent's process, as evidenced by this PR, actually follow what we told it to do, and did it do the self-validation it was supposed to do before claiming done." A correct fix produced by a process that skipped a required step is still a finding here.

Three tiers, one pipeline, not three separate tools:

1. Deterministic (Tier 1) -- cheap, code-only checks: did CI actually run and pass on this PR (gh pr checks), does a narrowly-titled fix touch files it has no business touching, does a very similar PR already exist nearby in time. Zero LLM calls, milliseconds, always runs first, and every one of these is evidence toward a specific documented rule (see references/failure-patterns.md), not a standalone verdict.
2. LLM-as-judge (Tier 2) -- Claude reads the actual PR (diff, body, commits), the Tier 1 evidence, and the repo's own documented AI-layer files, and judges the PR's trajectory against references/failure-patterns.md -- a rubric of specific rule violations and coverage gaps, mined from this repo's own history, each one citing the actual rule text. This is the eval. It runs automatically, on every new PR, via CI.
3. Human (Tier 3) -- the PR author and reviewers read Claude's posted review as part of the PR they were already going to review. No separate tooling.

Every finding feeds system evolution, not just this one PR. A rule violation means the AI layer's enforcement is too weak (fix: tighten it -- a doc step becomes a hook, a soft convention becomes a required check). A coverage gap means the AI layer is missing a rule entirely (fix: add one). Either way, the loop isn't closed by reading the finding -- it's closed when a file in the AI layer changes, same as the course's own outer loop.

When to use

• Workflow A is what runs when the user types /pr-trajectory-audit [owner/repo], or on an equivalent direct request ("audit my PR history," "build/refresh the trajectory rubric"). Runs steps 1-8 below in order, including installing the live review at the end -- this is the one command that takes a repo from nothing to a working live review.
• Workflow B normally triggers when invoked by assets/trajectory-review.yml (the CI action) on a specific PR -- the prompt will say which repo and PR number. It can also be triggered directly by a user asking "review PR N for trajectory issues" -- in that interactive case only, confirm with the user before posting anything to GitHub (step 6). The CI action is pre-authorized to post by the person who installed it; a direct chat request is not the same thing, and autonomously commenting on someone else's repo from an interactive session is not a default you get to assume.

All commands below assume you're running from the target repo's root, with this skill installed at .claude/skills/pr-trajectory-audit/ inside it (the normal install location -- see the top-level README's "How to use it").

Workflow A -- Audit (read the AI layer, then mine history for compliance evidence)

Run this periodically (after a batch of merges, or every few weeks) -- not on every PR. It's error analysis, but the data isn't just PR text: it's PR text checked against rules that already exist.

1. Resolve the target repo from the arguments, or detect it via gh repo view --json nameWithOwner -q .nameWithOwner if none was given.
2. Read the repo's own AI layer FIRST, before mining anything. Find and read its documented rules/workflow definitions. A real repo often has more than one of these at once -- read all that exist, don't stop at the first one you find: .archon/commands/defaults/*.md (Archon-style workflow procedures), CLAUDE.md/AGENTS.md (general engineering rules -- these frequently carry real, checkable conventions like a required PR-body template or a zero-warnings lint policy that a workflow-procedures-only read would miss entirely), .claude/commands//.claude/skills/*/SKILL.md, .cursorrules, or whatever else the repo actually uses. These layer on top of each other, not instead of each other -- a rule from any of them is fair game for a Tier 2 verdict.
   • How to actually get these files on disk, for a repo you haven't cloned: either git clone --depth 1 <repo-url> <scratch-dir> and read the tree directly (fastest for a whole directory like .archon/commands/defaults/), or gh api repos/<owner/repo>/contents/<path> per file if you only need a handful. For a whole directory of unknown file count, clone -- don't loop gh api one file at a time.
   • Write down a short list of the specific, checkable steps these define -- a validation step that must run before done, an investigate-before-implement gate, a staging/commit convention, whatever is actually there. Keep this list as your own working notes (it doesn't have a prescribed file -- you'll use it directly in step 4 and fold the ones that produce real findings into step 6's failure-patterns.md). This list is what step 4 checks PRs against. Skip this step and every later finding is just inferred opinion wearing a citation it doesn't have.
3. Mine the PR history:

   uv run .claude/skills/pr-trajectory-audit/scripts/mine_prs.py mine <owner/repo> --limit <N, default 1000> --top 10
   

   Ranks PRs by how many later PRs referenced them (useful for spotting a rule that got violated more than once) and surfaces duplicate-title clusters (useful for spotting a coverage gap). Why two regex passes instead of one, and the tradeoffs: references/mining-methodology.md. Note: numbers in the "most-referenced" list can be GitHub issues, not PRs -- they share one number sequence. If gh pr view <n> 404s, try .

Workflow B -- Live review (runs on ONE PR, automatically, via CI)

This is what Claude does when trajectory-review.yml invokes it on a new PR.

1. Run Tier 1 evidence-gathering:

   uv run .claude/skills/pr-trajectory-audit/scripts/mine_prs.py checks <owner/repo> <pr_number> --json
   

   Prints CI-status, scope-blast-radius, and duplicate-vs-recent findings as JSON. This is evidence toward specific rules, not a verdict -- read it, don't just relay it.
2. Read references/failure-patterns.md -- the rubric of specific rule violations and coverage gaps, each citing the repo's own documented AI layer. If it has no real findings in it yet (Workflow A has never been run), say so plainly in the review and stop -- don't invent findings to fill the gap. Also check the rubric actually belongs to this repo before applying it: if it cites rule files (e.g. .archon/commands/defaults/*.md) that don't exist in this repo's tree, it's leftover content from copying the skill folder from another repo/the workshop demo and was never regenerated by Workflow A here -- treat this exactly like an empty rubric (say so plainly, don't cite it) rather than judging this PR against rules that were never this repo's own.
3. Read the PR itself: gh pr diff <pr_number> --repo <owner/repo> for the actual diff, and the PR body already available from step 1's gh pr view call inside the script. Read the diff -- Tier 1 can tell you CI didn't run or file paths look suspicious; only reading the actual content tells you whether a documented step was genuinely followed.
4. If duplicate_vs_recent fired in step 1, read the flagged PR(s)' comments and closing reason before treating it as a coordination gap. Title similarity alone can't tell a genuine independent duplicate apart from the same author's own resubmission or rebase -- that distinction is usually sitting in a comment and is easy to miss if you only read the diff and body. Getting this wrong produces a real false positive, not just a hedge.
5. Judge the PR's trajectory against each rubric entry, explicitly, citing the specific rule. For each rule-violation entry in the rubric: does this PR show the same documented step being skipped or shallowed (e.g., is there CI/test evidence backing a "validation complete" claim, or just the claim)? For each coverage-gap entry: does this PR match the recurring pattern (e.g., does step 4's comment check confirm this is genuinely independent duplicate work)? Apply references/judging-rubric.md for how to render a verdict that cites the actual rule text and actual evidence, not "looks off."
6. Post ONE PR review comment combining the Tier 1 evidence and the Tier 2 judgment -- don't post twice. For every finding, name the specific documented rule or the coverage-gap pattern it's checking, and cite the evidence (file paths, CI status, PR numbers, quoted diff or comment text). If nothing in the rubric applies, say so plainly and name what was checked -- a silent pass looks identical to "didn't run" from the outside.

Gotchas

• gh must be authenticated against the target repo (gh auth status) for both workflows.
• Installing trajectory-review.yml needs the Claude Code GitHub App installed on the target repo (https://github.com/apps/claude) -- a separate, one-time step from setting the CLAUDE_CODE_OAUTH_TOKEN secret, only doable interactively by the repo owner (not something a CLI/API call can do). Skipping it fails with a specific error, Claude Code is not installed on this repository, only at the Claude step -- confirmed via an actual live-webhook test where checkout and every earlier step in the workflow succeeded first. This tripped up this skill's own testing before Friday's workshop -- expect it to trip up an audience member too if it isn't called out explicitly.
• This skill ships with references/failure-patterns.md pre-populated from Archon, not blank. Copying pr-trajectory-audit/ into a different repo (the README's documented "how to use it") brings Archon's findings along with it. Confirmed via a real fresh-context test: a naive first run on a brand-new repo correctly found nothing to mine, but nothing forced a check on whether the existing file still applied, and it was left citing .archon/commands/... files that don't exist in the new repo. Workflow A step 5 above and Workflow B step 2 both now check for this explicitly and reset/ignore a mismatched rubric -- but if you're adapting this skill by hand rather than running the full workflow, don't skip that check yourself.
• Fork PRs on a public repo will not get a posted review. GitHub downgrades GITHUB_TOKEN to read-only for pull_request-triggered runs on PRs opened from a fork, regardless of the workflow's permissions: block, and may not expose repo secrets to that run either -- so trajectory-review.yml can silently no-op (or fail to post) on exactly the PRs a popular open-source repo gets the most of. This shows up only as a red X in the Actions tab, which most outside contributors never check. There is no way to safely bypass this from a pull_request trigger without taking on real security risk (pull_request_target runs with write access against untrusted fork code) -- the honest mitigation is a maintainer-triggered fallback: add an issue_comment trigger gated on a trigger phrase from a trusted association (OWNER/MEMBER/COLLABORATOR), the same pattern this org's other -style workflows already use, so a maintainer can manually ask for the review on a fork PR when it matters.

Resources

• ../sample-report-archon.md (one level up, at the workshop repo's root, NOT copied into this skill folder) -- a full worked example from auditing coleam00/Archon. Useful to read for the shape a finished report takes, but it doesn't ship when you copy just this pr-trajectory-audit/ folder into another repo -- that's expected, it's a demo artifact, not part of the skill.
• references/mining-methodology.md -- Workflow A's two-pass regex rationale and precision/recall tradeoffs.
• references/failure-patterns.md -- the rubric. Workflow A writes it; Workflow B reads it on every PR. The actual connective tissue between the two workflows.
• references/judging-rubric.md -- how to render a Tier 2 verdict that's evidence-cited and checkable, used by both workflows.
• scripts/mine_prs.py -- run it (mine / scope / checks subcommands, --help for flags), don't read its source into context.
• assets/report-template.md -- optional human-readable audit summary shape (Workflow A, step 7).
• assets/trajectory-review.yml -- the GitHub Action that invokes Workflow B on every new PR. Copy to .github/workflows/ in the target repo; needs the CLAUDE_CODE_OAUTH_TOKEN secret (same one used by any other anthropics/claude-code-action workflow in the org).