pr-review

name

pr-review

description

AI-assisted pull request review workflow for Azure DevOps Git repositories. Use when the user asks to review a pull request, perform a PR review, analyse PR changes, or run the PR review pipeline. Provides a complete scripted workflow to fetch PR metadata, prepare a local branch folder, export diffs, generate review reports and checklists, and post review comments back to Azure DevOps. Requires Python 3.9+ and Git.

AI Pull Request Review Agent SOP (Azure DevOps + Python)

Purpose-built standard operating procedures and minimal workspace structure for an AI-assisted PR review workflow targeting Azure DevOps Git repositories using Python as the cross-platform scripting language.

For script invocation guidance, see references/tool-instructions.md.

1. Objectives

   • Provide a reproducible, minimal workspace for offline PR review artifacts.
   • Fetch and normalize diffs and changed files against a base branch.
   • Drive a consistent, checklist-led review that is easy to audit.
   • Keep authentication safe and avoid leaking PATs or secrets into logs.
   • Always confirm the existence of pr-review.json by first calling list_files on the workspace directory before using read_file or asking follow-up questions. This prevents assumptions about its presence.

2. Workspace layout

   The repository is organized into the following structure:

   .
   ├── pr-review.json                      Read-only user configuration (NEVER modified)
   ├── .ai/
   │   └── skills/
   │       └── pr-review/
   │           ├── SKILL.md                         This SOP file
   │           ├── scripts/                         Python workflow scripts
   │           │   ├── review_config.py             Helper module for merged configuration
   │           │   ├── invoke_build.py              Cross-platform build wrapper (dotnet msbuild / MSBuild)
   │           │   ├── Setup_Util_TrustedRootCertificates_Save.ps1  Windows corporate cert helper
   │           │   ├── s01_reset_workspace.py       Clean previous review artifacts
   │           │   ├── s02_get_azure_devops_info.py Fetch PR and work item metadata
   │           │   ├── s03_extract_attachments.py   Download images from PR/work-item comments + descriptions
   │           │   ├── s04_fetch_repository.py      Prepare BranchFolder and fetch branches when PullBranch is true
   │           │   ├── s05_reset_templates.py       Create review templates with actual PR data
   │           │   ├── s06_export_diff_artifacts.py Export per-file diffs and changed files
   │           │   ├── s07_consolidate_diffs_and_content.py Consolidate patches and snapshots
   │           │   ├── s08_rasterize_images.py      Render ```mermaid blocks to PNG (offline, no upload)
   │           │   ├── s09_upsert_review_attachments.py Upload diagrams + local images to PR attachments
   │           │   ├── s10_upsert_review_comment.py Post review comment to Pull Request
   │           │   ├── s11_upsert_pr_description.py Upsert PR description (guarded; refuses to overwrite real content)
   │           │   └── s12_upsert_suggestion_threads.py Post suggestion threads to Pull Request
   │           ├── assets/                          Templates (SSOT)
   │           │   ├── review.template.md           Template for review.md
   │           │   ├── checklist.template.md        Template for checklist.md
   │           │   └── config.template.json         Template for pr-review.json
   │           └── references/
   │               └── tool-instructions.md         Python invocation rules
   └── {WorkFolder}/                    Configurable via WorkFolder in pr-review.json
       ├── review.md                    Consolidated findings and verdict (generated)
       ├── checklist.md                 Checklist and scoring rubric (generated)
       ├── pr-description.md            Brief PR description (generated; posted only when original PR description is empty)
       ├── diagrams/                    Optional generated images (when GenerateImages: true)
       ├── attachments/                 Screenshots from PR + work items (written by s02b; see manifest.json)
       ├── context.json                 Dynamic Azure DevOps data (written by scripts)
       ├── meta.json                    Git repository metadata (written by scripts)
       ├── changes/                     Working tree copies of changed files
       ├── diffs/                       Per-file unified diff patches (path.patch)
       ├── base/                        Optional base versions of changed files
       └── branch/                      Default local PR branch folder when BranchFolder is not `.`
   

3. PR configuration and file structure

   The configuration system uses three separate files to maintain clean separation between user input, fetched data, and Git metadata:

   pr-review.json - Read-only user configuration (root of workspace)

   • This file is NEVER modified by scripts
   • Can be version controlled without dynamic data
   • Should contain as little as possible. Scripts derive Azure DevOps values from the current Git checkout first, then ask the user only for values that cannot be derived.
   • Default derived values:
     • BaseUrl, OrganizationName, ProjectName, and RepoName are parsed from git remote get-url origin for Azure DevOps HTTPS or SSH remotes.
     • BranchName is derived from the current Git branch.
     • TargetBranchName is derived from origin/HEAD and falls back to master.
     • PullRequestId is optional. If omitted, s02_get_azure_devops_info.py finds the active PR whose source branch is the current branch. If none or multiple active PRs are found, the workflow stops with a clear message and asks for PullRequestId.
     • WorkItemIds is optional. If omitted, s02_get_azure_devops_info.py reads linked work items from the PR.
     • AzureApiVersion defaults to 7.1.
     • WorkFolder defaults to .tmp/pr-review.
     • PullBranch defaults to false and BranchFolder defaults to . so reviewing the current branch is the default workflow.
   • Optional override fields:
     • BaseUrl, OrganizationName, ProjectName, RepoName - only set these when the Git remote cannot be parsed.
     • PullRequestId - only set this when multiple/no active PRs are found for the current branch.
     • WorkItemIds - only set this when PR work item links are absent or incomplete.
     • BranchName, TargetBranchName - only set these when Git branch/default-branch detection is wrong.

4. Prerequisites

   • Python 3.9 or newer
   • Git 2.35 or newer with Git Credential Manager enabled
   • .NET SDK (for building solutions via invoke_build.py)
   • Access to Azure DevOps repositories
   • Install dependencies: pip install -r .ai/skills/pr-review/scripts/requirements.txt
   • Current Python dependencies include requests and requests-ntlm because the Windows Integrated Authentication fallback uses NTLM helpers when no PAT or Azure CLI token is available

   Authentication methods (scripts try in this order):

   1. Personal Access Token (PAT) - Set AZDO_PAT environment variable with a PAT that has Code (Read) scope. Works on all platforms.
   2. Azure CLI - Uses az account get-access-token if Azure CLI is installed and authenticated. Cross-platform.
   3. Windows Integrated Authentication - Final fallback for domain-joined Windows machines.

5. Quick start using scripts

   Use the Python scripts in the .ai/skills/pr-review/scripts/ directory to prepare the workspace and export diffs and changed files. Scripts are prefixed with execution order (s01, s02, etc.). Run them in the following order:

   Run each script as a separate command and wait for it to finish before invoking the next script. Do not chain multiple scripts together on a single command line.

   • python .ai/skills/pr-review/scripts/s01_reset_workspace.py - Clean previous review artifacts
   • python .ai/skills/pr-review/scripts/s02_get_azure_devops_info.py - Fetch PR and work item metadata
   • python .ai/skills/pr-review/scripts/s03_extract_attachments.py - Download attached images from PR + linked work items into {WorkFolder}/attachments/
   • python .ai/skills/pr-review/scripts/s04_fetch_repository.py - Prepare the local branch folder; clones/fetches only when PullBranch is true
   • python .ai/skills/pr-review/scripts/s05_reset_templates.py - Create review templates with actual data
   • python .ai/skills/pr-review/scripts/s06_export_diff_artifacts.py - Export diffs and changed files
   • python .ai/skills/pr-review/scripts/s07_consolidate_diffs_and_content.py - Consolidate patches and generate before-and-after content snapshots
   • AI writes {WorkFolder}/review.md with findings
   • python .ai/skills/pr-review/scripts/s08_rasterize_images.py - Render ```mermaid blocks in review.md to PNGs (offline, always safe)
   • python .ai/skills/pr-review/scripts/s09_upsert_review_attachments.py - Upload diagrams + local images to PR; rewrite review.md (skipped when SKIP_POST_COMMENT=1)
   • python .ai/skills/pr-review/scripts/s10_upsert_review_comment.py - Upsert review comment to Pull Request

   Attachments

   If {WorkFolder}/attachments/manifest.json exists with totalCount > 0, the PR (or a linked work item) carries screenshots. For each entry: open imagePath (multimodal), read contextPath (why it was attached). Treat any ai-vision-summary.md inside a workitem-*/ folder as authoritative unless the image contradicts it. Cite findings with the attachment path (e.g. Source: attachments/workitem-238853/img-001-image.png). A screenshot that documents a bug the diff doesn't fix is a Blocker.

6. Review flow

   • Think through upcoming steps deliberately and verify instructions (e.g., consult .github/copilot-instructions.md) before executing commands.

   • Prepare workspace using the quick start scripts. The s01_reset_workspace.py script cleans artifacts, then s02_get_azure_devops_info.py fetches PR data, then s03_extract_attachments.py downloads PR + work-item attachments into {WorkFolder}/attachments/, then s04_fetch_repository.py prepares BranchFolder and writes Git metadata, then s05_reset_templates.py creates {WorkFolder}/review.md and {WorkFolder}/checklist.md from their respective template files with actual PR data.

   • When PullBranch is false, BranchFolder is treated as the already checked-out PR branch and the head ref is HEAD; set BranchFolder to . to review the current workspace branch.

   • If present, read .github/copilot-instructions.md from the target repository (BranchFolder) and incorporate any guidance it contains into the review process, such as which projects to test.

   • Load {WorkFolder}/all-pre-content.txt – full "before" state of all changed files

   • Load {WorkFolder}/all-post-content.txt – full "after" state of all changed files

   • Use {WorkFolder}/all-diffs.txt – concise summary of changed lines and statuses

   • Skim {WorkFolder}/diffs/changed-files.tsv to understand scope.

   • Read per-file patches under {WorkFolder}/diffs and the corresponding working files under {WorkFolder}/changes.

   • Restore/build the solution as a best-effort sanity check, not a gate:

     • Pick the build target deterministically:
       • When both .slnx (modern XML format) and .sln (legacy INI format) exist with the same stem, always use the .slnx — it is the source of truth in repositories that have started migrating. invoke_build.py enforces this automatically when given a .sln with a sibling .slnx.
       • When only one of .slnx / .sln exists at the repo root, use it.
       • When no solution exists, build the most relevant changed project (.csproj//).

7. Review checklist

   The review checklist template is maintained in assets/checklist.template.md. This is the single source of truth for the checklist structure.

   The s05_reset_templates.py script creates {WorkFolder}/checklist.md from the template, replacing placeholders with actual values from the configuration and fetched data.

   Template placeholders:

   • {PR_LINK} - Pull request URL
   • {BASE_BRANCH} - Base branch name
   • {FEATURE_BRANCH} - Feature branch name
   • {REPO_NAME} - Repository name
   • {PROJECT_NAME} - Project name
   • {WORK_ITEM_LINK} - Link(s) to associated work item(s)

   The checklist includes sections for:

   • Preparation - workspace setup verification
   • Scope - change description and dependencies
   • Code quality - readability, maintainability, structure
   • Correctness - logic, edge cases, error handling
   • Security - credentials, validation, authorization
   • Performance - efficiency, algorithms, data access
   • Testing - unit tests, integration tests, coverage
   • Operations - migrations, configs, observability
   • Documentation - README, changelog, comments
   • Scoring rubric (0-5 per dimension)
   • Decision and follow-ups

8. Review report

   The review report template is maintained in assets/review.template.md. This is the single source of truth for the report structure.

   s05_reset_templates.py creates {WorkFolder}/review.md from the template, replacing static placeholders with values from configuration and fetched data. AI-judged placeholders are filled in by the agent during review.

   Template placeholders (static, replaced by s05_reset_templates.py):

   • {PR_LINK} — Pull request URL
   • {REPO_NAME} — Repository name
   • {PROJECT_NAME} — Project name
   • {BASE_BRANCH} — Base branch name
   • {FEATURE_BRANCH} — Feature branch name
   • {WORK_ITEM_LINK} — Link(s) to associated work item(s)

   Template placeholders (AI-judged, filled in by the agent):

   • {RISK_BADGE} — Low ⇒ 🟢, Medium ⇒ 🟡, High ⇒ 🔴
   • {RISK_LEVEL} — Low / Medium / High
   • {RISK_REASON} — short one-line justification (≤ ~12 words)
   • {CONFIDENCE_BADGE} — High ⇒ 🟢, Medium ⇒ 🟡, Low ⇒ 🔴 (inverted vs. Risk so 🟢 always means "good")
   • {CONFIDENCE_LEVEL} — High / Medium / Low
   • {CONFIDENCE_REASON} — short one-line justification (≤ ~12 words)
   • {DECISION_SYMBOL} — ✅ or ❌
   • {DECISION_LABEL} — Approve / Approve with comments / Request changes

   Visible block (above the collapsible audit):

   • Risk header line: {RISK_BADGE} **Risk: {RISK_LEVEL}** — {RISK_REASON}
   • Confidence header line: {CONFIDENCE_BADGE} **Confidence: {CONFIDENCE_LEVEL}** — {CONFIDENCE_REASON}
   • Decision (symbol + label + one-line rationale)
   • Related — one bullet per linked reference, using each platform's native auto-link syntax (Azure DevOps work item , Azure DevOps pull request , GitHub issue/PR ). No markdown wrapping, no state, no comment count, no placeholder — those add noise and the platform already shows that data. Optionally append only when the title is in and genuinely helps the scanner. External references that the platform does not auto-link use . Empty list → .

8a. PR description (auto-generated brief)

`s05_reset_templates.py` also generates `{WorkFolder}/pr-description.md` from `assets/pr-description.template.md`. The AI overwrites it during review with `## Changes` + `## Why` content.

**Unfilled sentinel:** the template opens with `<!-- pr-description:unfilled — agent: remove this line ... -->`. The agent MUST delete that sentinel line when writing real content. The n8n workflow's `Build Review Comment` node skips the PR-description update entirely if the sentinel is still present, so an unfilled scaffold cannot reach the PR.

**Do NOT add a `Related` section.** Azure DevOps shows linked work items, and GitHub shows linked issues/PRs, in the native PR UI already — duplicating them in the description is noise. Only mention references that are *not* already auto-linked by the platform (external wiki page, ticket in another system, public docs link).

The n8n workflow `Refactoring - Review Pull Request` reads this file and updates the actual PR description **only when** (a) the sentinel has been removed AND (b) the original PR description is effectively empty (literally empty, whitespace-only, or one of the documented placeholder patterns). Author content is never overwritten.

9. Script interfaces specification

review_config.py - Helper module for configuration management

Functions:

• get_review_config() - Load and merge pr-review.json + {WorkFolder}/context.json
• get_git_repository_root() - Find Git repository root for a path
• assert_standalone_git_repository() - Ensure path is a standalone Git repo root
• get_workspace_root() - Locate workspace root by walking up to pr-review.json
• get_work_folder() - Resolve artifact output directory from config
• get_branch_folder() - Resolve the local PR branch repository folder from BranchFolder or legacy BranchPath
• should_pull_branch() - Normalize the PullBranch setting to a boolean
• get_auth_headers() - Build Azure DevOps authentication headers (PAT / Azure CLI / fallback)

s02_get_azure_devops_info.py - Fetch PR and work item metadata

Purpose: Query Azure DevOps REST API and populate {WorkFolder}/context.json with dynamic data. Uses settings from pr-review.json.

s03_extract_attachments.py - Download attached images from PR + linked work items

Purpose: Scan PR and linked work items for image references and download each into {WorkFolder}/attachments/ with a .context.md and .meta.json sidecar. The reviewing AI reads attachments/manifest.json. Deterministic; no AI calls.

s04_fetch_repository.py - Prepare the local branch folder and fetch branches when configured

Purpose: Clone or update BranchFolder and fetch base/feature refs when PullBranch is true; otherwise use BranchFolder as an existing local checkout and compare base against HEAD. Uses settings from pr-review.json and {WorkFolder}/context.json.

s05_reset_templates.py - Create review templates with actual PR data

Purpose: Create review.md and checklist.md from templates with actual PR data. Uses settings from pr-review.json and {WorkFolder}/context.json.

s06_export_diff_artifacts.py - Export diffs and changed files

Purpose: Export per-file diffs, patches, and changed files for base..feature comparison. Uses settings from pr-review.json and {WorkFolder}/context.json.

s07_consolidate_diffs_and_content.py - Consolidate patches and snapshots

Purpose: Concatenate per-file diffs and pre/post snapshots into single artifacts for the reviewing AI, with chunked output for large PRs.

s01_reset_workspace.py - Clean previous review artifacts

Purpose: Remove generated artifacts from previous review to prepare for new review.

Parameters:

• --keep-repo - Keep repository checkout in BranchFolder when it is under {WorkFolder} (faster for same repo)

s08_rasterize_images.py - Render ```mermaid blocks in review.md to PNGs

Purpose: Detect ```mermaid fenced blocks in review.md and render each to {WorkFolder}/diagrams/diagram-NNN-<sha>.png using mmdc (mermaid-cli). Writes mermaid-manifest.json linking each block to its PNG. Offline — no network or PR API calls — so safe to run unconditionally even when SKIP_POST_COMMENT=1. Idempotent on content sha256. No-op (with warning) when mmdc is not installed.

Parameters:

• --scale N - mmdc render scale (default: 2)
• --background-color COLOR - mmdc background color (default: white)
• --mmdc PATH - Explicit path to mmdc binary

s09_upsert_review_attachments.py - Upload diagrams + local images to PR; rewrite review.md

Purpose: For Azure DevOps PRs, upload each PNG produced by s08 to the PR's attachments endpoint and substitute ```mermaid fences in review.md with ![](attachment-url) (collapsed source kept in <details>). Also uploads any ![alt](local-path) references the AI embedded (e.g. ai-image-generation output) and substitutes their URLs. Skipped on GitHub (which renders mermaid natively).

This script POSTs to the PR's attachments endpoint and must not run when SKIP_POST_COMMENT=1 — entrypoint.sh gates it the same way as s10/s11.

Parameters:

• --dry-run - Print actions without uploading or rewriting

s10_upsert_review_comment.py - Upsert review comment to Pull Request

Purpose: Post the content of {WorkFolder}/review.md as a comment thread on the PR. Uses settings from pr-review.json and {WorkFolder}/context.json.

Parameters:

• --dry-run - Print payload without posting

s11_upsert_pr_description.py - Upsert PR description from pr-description.md

Purpose: Update the PR description from {WorkFolder}/pr-description.md ONLY when all four guards pass — file is non-empty, no <!-- pr-description:unfilled --> sentinel, no scaffold-signature phrases (Briefly list what changed, One-line motivation, {WORK_ITEM_LINK}), AND the PR's current description is literally empty. Refuses to overwrite real author content (institutional memory: the n8n equivalent once wiped a real description on PR 32147 when its guards were too loose).

Parameters:

• --dry-run - Evaluate guards and print decision without PATCHing

s12_upsert_suggestion_threads.py - Upsert suggestion threads to Pull Request

Purpose: Post suggestions as dedicated inline PR comment threads. Uses settings from pr-review.json and {WorkFolder}/context.json.

Parameters:

• --remove-posted - Remove previously posted suggestion threads
• --dry-run - Print payloads without posting

invoke_build.py - Build solutions or projects

Purpose: Cross-platform build wrapper. Uses dotnet msbuild (all platforms) or full MSBuild via vswhere (Windows with Visual Studio).

Parameters:

• All arguments are passed through to the build tool (solution/project file path and any MSBuild switches).
• The first argument is normalized to an absolute solution/project path before invocation.

Behaviours worth knowing:

• Always prefer .slnx over .sln when both exist with the same stem; the wrapper auto-substitutes.
• On non-Windows hosts, the wrapper walks the solution/project graph and rewrites recorded paths in-place for any ProjectReference or solution-listed project whose casing doesn't match the on-disk file (e.g. Web.Core/Web.Core.csproj -> Web.Core/web.core.csproj). Rewrites are line-based string substitution against the matched span; no XML/INI reflow, BOMs preserved. The rewrites live in the disposable clone only and never reach a commit.

Examples:

python .ai/skills/pr-review/scripts/invoke_build.py {solution|project} /v:minimal /clp:Summary
python .ai/skills/pr-review/scripts/invoke_build.py MyApp.slnx /p:Configuration=Release /t:Rebuild

10. Python guidance

    • Run scripts with python script.py from the repository root.
    • All scripts use UTF-8 without BOM for text output.
    • All scripts accept --help for usage information.
    • Use pathlib.Path for cross-platform path handling.
    • Avoid writing secrets to disk or to logs.
    • Platform-specific: Setup_Util_TrustedRootCertificates_Save.ps1 is automatically invoked on Windows by API scripts for corporate certificate environments using a direct powershell -File invocation.

11. Mermaid overview

    flowchart TD
      A[Start] --> B[Setup workspace dirs]
      B --> C{PullBranch?}
      C -->|true| D[Clone/update BranchFolder and fetch base and feature]
      C -->|false| E[Use current BranchFolder HEAD]
      D --> F[List changed files]
      E --> F
      F --> G[Export diffs]
      F --> H[Export changed files]
      G --> I[Review diffs and files]
      H --> I
      I --> J[Complete checklist]
      J --> K[Record verdict]
    

12. Quality bar for approvals

    • No known correctness issues.
    • Security and secrets posture unchanged or improved.
    • Risk appropriate tests present or clearly ticketed with timeline.
    • Operational concerns documented and migration risks addressed.
    • Code clarity acceptable or improved.

13. Known edge cases

    • Large binary or generated files should be excluded from export; skip via --max-file-bytes.
    • Line-ending normalization may affect patch readability; set core.autocrlf consistently.
    • Submodules and LFS objects require additional steps not covered here.

14. Maintenance

    • Keep this instructions file aligned with the scripts contract.
    • Update defaults when the target repository or branches change.
    • Consider adding a small validation script to lint the exported artifacts.