review-docs-pr

# Review Documentation PR This skill reviews documentation pull requests for the Warp docs repository. ## Usage Use this skill when reviewing documentation changes in PRs. The skill will: - Check for potential broken links - Verify style guide compliance - Review content quality - Review AEO/source-data fit for docs changes that target search or answer-engine visibility - Check Astro Starlight structure integrity When the PR carries the `warpy-factory` label, this is the **independent v1 agent-doc quality review pass** dispatched by `.github/workflows/agent-docs-review.yml` for the current head SHA (see "Agent-marked PR review (v1 contract)" below). ## Agent-marked PR review (v1 contract) For a PR carrying the `warpy-factory` label, in addition to the standard review focus areas, this pass is the human-review-blocking gate defined in `.agents/references/doc-quality-policy.md`: 1. **Re-validate the declared risk.** Parse the PR's `## Documentation risk` section (`.agents/skills/doc_quality_policy/policy.py`'s `parse_documentation_risk_section`). Walk the diff against the low-risk allowlist yourself. A declared `low` risk that actually touches any engineering-review trigger (a new/changed feature page, or any of the technical claim categories in the allowlist) is a **risk misclassification** — flag it `⚠️ [IMPORTANT]` and reflect it in the verdict (see "Severity Labels" below); it is not a soft suggestion. 2. **Verify technical claims against source** when the PR is (or should be) `engineering-review-required`: confirm the cited `Source files consulted` actually support the claims. Use `answer_question` when source access is available. 3. **Check the compression contract.** Run `.agents/skills/doc_quality_policy/check_compression_contract.py` against each changed content-type page. An unjustified violation (no reasoning in the PR body for the overage) is `⚠️ [IMPORTANT]`; a justified one is `💡 [SUGGESTION]` at most. 4. **Check VERIFY accounting.** Run the structural mode of `.agents/skills/doc_quality_policy/check_pr_contract.py --body <file> <changed public documentation files>` against the current head. Pass only changed `.md` or `.mdx` files under `src/content/docs/`, matching the checker's autodiscovery scope. Do not pass `.agents/` policy prose, workflow files, tests, or other non-public documentation files. Do not invoke `--enforce-engineering-gate` during the independent review; a pending human approval is not itself a contract violation. Any reported contract violation (missing section, unlisted marker, listed marker at `low` risk) is `🚨 [CRITICAL]`. 5. **Check Gate 0 for server-backed claims.** Apply the server-flag check in `.agents/references/docs-worthiness-criteria.md` → Gate 0 to every new or changed claim about a warp-server field, endpoint, default, error, or behavior, including Factory file fields. The PR must record evidence that the feature is live in production: a stable changelog entry, the released OpenAPI spec, a `server_rollout.py` check at a named `warp-server` revision, or an engineer's confirmation. A published schema, a parser that accepts the field, a merged PR, or staging behavior does not count. When a `warp-server` checkout is available, run `.agents/skills/doc_quality_policy/server_rollout.py check` with the claim's user-facing names yourself. A claim whose related flag is off in production is `🚨 [CRITICAL]`; a server-backed claim with no rollout evidence is `⚠️ [IMPORTANT]`. Reviews are public: describe the problem as "a server flag is off in production" without naming the flag or quoting the script output. ## Review Instructions Review this documentation PR for the docs repository. Focus on: 1. **Broken links**: Check for potential broken internal links (relative paths, cross-space links). You can run the broken link checker at `python3 .agents/skills/check_for_broken_links/check_links.py --internal-only` if helpful. 2. **Style guide compliance**: Reference `AGENTS.md` for documentation standards (voice, formatting, terminology). 3. **Content quality**: Check for clarity, accuracy, proper frontmatter, and appropriate use of headers and lists. For setup instructions, check each requirement: * The task is stated before fields or example names. * Each selection explains what the agent, environment, or credential is used for, not only its name. * Readers can find the supported path to edit or select an existing resource instead of creating a duplicate. * Configuration, access setup, skill or prompt instructions, and testing are separate steps or groups of related steps. * Safety limits appear beside the relevant action and inside prompts that can be copied separately. * Tests use explained sample inputs, require credentials only when needed, follow applied configuration changes, and check observed behavior rather than only run completion. * Proposed sentence edits still make sense when read with the whole paragraph and surrounding steps. 4. **Code snippets**: Verify that examples, commands, and configuration work as documented. When procedures connect, read their examples together and check: * Which event starts each task, which agent receives it, and what information passes to the next stage. * Whether several rules can match the same input, whether that is intended, and how the conditions prevent unwanted matches. * Whether adding a new condition preserves existing filters and exclusions. * What happens when an input matches none of the specialized rules. * Whether changing a rule affects new tasks, existing conversations, or both; verify any claim that existing work moves to another agent. * Where each stage stops, which decisions require a person, and whether tests cover matching, overlapping, unmatched, and continued work. Use [Check connected procedures](../../../AGENTS.md#check-connected-procedures) for the full checks. If a technical detail is uncertain, use `answer_question` to verify against the docs or search the source code. 5. **Astro Starlight structure**: Verify `src/sidebar.ts` updates if files were added, moved, or renamed, and that redirects are added to the `redirects` array in `vercel.json` when needed. 6. **Product name variables**: Check whether any product names with a corresponding entry in `src/data/vars.ts` are hardcoded as literal strings instead of using `{VARS.KEY}` (prose) or `{{TOKEN}}` (frontmatter). Key strings to watch for: "Oz CLI", "Oz web app", "oz.warp.dev", "Oz dashboard", "Oz run". Flag as `⚠️ [IMPORTANT]` if a new file adds these without using the variable system. For existing files, flag as `💡 [SUGGESTION]`. 7. **Tone and AI-isms**: Flag marketing buzzwords (seamless, powerful, robust, comprehensive, leverage, streamline), meta-openers ("This page covers/explains/walks through..."), restated cause-and-effect ("This process ensures..."), recap lines, consecutive callouts or more than one callout per section, and internal-architecture detail the reader can't act on (orchestrators, control planes, lifecycle states). Reference the "Voice & tone" section of `AGENTS.md`. Use `💡 [SUGGESTION]` for isolated instances; use `⚠️ [IMPORTANT]` when the pattern is pervasive in new content. 8. **Length and reader understanding**: Apply AGENTS.md → Procedural → Reader-understanding test before proposing a cut. Name repetition removed and any choice, consequence, or success signal lost. Cut padding before splitting; word count alone is not an editorial verdict. Preserve necessary facts by linking or relocating them. 9. **Unreleased features**: For any PR, agent-marked or not, apply item 5 of the v1 contract above to server-backed claims. A launch PR is not exempt; Gate 0 applies to launches too. 10. **AEO/source-data fit**: Provide actionable, constructive feedback. Focus on documentation quality issues, not code bugs. Include an editorial verdict in the review rationale: reader task, concept ownership across siblings, unnecessary scaffolds or repetition removed, unsupported assurances, and useful understanding preserved. Identify each unconfirmed claim and its narrow correction. Removing a page or changing its purpose requires human approval; a transition guide is not interchangeable with a chooser. Use `.agents/rules/oz-style-guidelines.md` for framing and the canonical glossary for terminology. Historical Oz references, commands, and package names are not automatic rename findings. ### Severity Labels (Required) Every comment body MUST begin with one of: - `🚨 [CRITICAL]` — Broken links, incorrect commands/code, factually wrong information that could confuse users, a feature whose server flag is off in production, or a v1 contract violation (missing/invalid `## Documentation risk`, an unlisted `VERIFY` marker) - `⚠️ [IMPORTANT]` — Style guide violations, missing redirects, structural issues, a documentation-risk misclassification, or an unjustified compression-contract violation - `💡 [SUGGESTION]` — Improvements to clarity, wording, or structure - `🧹 [NIT]` — Typos, minor formatting (ONLY if providing a suggestion block) **Blocking rule for agent-marked PRs.** Any `🚨 [CRITICAL]` or `⚠️ [IMPORTANT]` finding — including a risk misclassification — means the verdict is **Request changes**. Suggestions and nits never block. ### Using answer_question for verification If you encounter: - Unclear or potentially incorrect technical information - Commands or code examples you want to verify - Feature descriptions that seem outdated or inaccurate - Questions about how something actually works Use the `answer_question` skill to search the documentation and source code for authoritative information before making your review comment. ### AEO review guidance For AEO-driven docs PRs, review the diff through these questions: - **Source rationale** - Does the brief or PR rationale name the exact query or recommendation and what was measured? For each proposed topic, does it distinguish direct supporting data from the author's recommendation? Data about a broad topic does not prove demand for every narrower topic. If evidence is missing, ask for it or narrow the claim; do not invent demand. - **Vocabulary translation** - Does the draft preserve high-intent search terms where useful while translating awkward source-data language into natural docs language? - **Reader value** - Does the page help a developer complete or understand its intended task? Read the opening and description without the title or navigation: both must identify the relevant product and make sense on their own. Do not require the product name in every heading or add unrelated topics to include more search terms. - **Terminology and UI surfaces** - Are product names, settings paths, panel names, and navigation labels consistent with `AGENTS.md`, `.agents/references/terminology.md`, and the actual UI? - **Scannability** - Are dense procedures broken into numbered steps, bullets, or concise subsections with expected outcomes? - **Duplication risk** - Should the content update an existing page, link to an existing page, or be merged with related content instead of living as a new standalone doc? When the issue is a judgment call, prefer a `💡 [SUGGESTION]` comment that explains the tradeoff rather than blocking the PR. ## Output Format Create a `review.json` file with the following structure: ```json { "summary": "High-level overview of the changes and verdict", "comments": [ { "path": "path/to/file.md", "line": 42, "side": "RIGHT", "body": "Your feedback here" } ] } ``` ### Summary Requirements The summary should: - Use at most three concise sentences describing the PR and its assessment. - Include issue counts: `Found: X critical, Y important, Z suggestions, N nits`. - End with the recommendation: `Approve`, `Approve with nits`, or `Request changes`. The publishing step renders the summary under a `## Review summary` heading. ### Comment Format Each comment should: - Reference a specific line in a changed file - Be actionable and constructive - Use `side: "RIGHT"` for new/added lines (the new content after changes) - use this in most cases - Use `side: "LEFT"` only when commenting on deleted/old lines (the content before changes) - Focus on the diff, not unchanged code - Keep comment spans ≤10 lines (i.e., `line - start_line <= 10`) for easier review - Use single-line comments for specific issues (typos, broken links); use multi-line spans when the issue or suggestion requires multiple lines of context #### Using GitHub Suggestion Syntax When you have a specific fix or improvement, use GitHub's suggestion syntax in your comment body. This allows the author to apply your suggestion with one click. Use suggestions for: - Fixing typos or grammar - Correcting commands or code snippets - Improving wording or phrasing - Fixing broken links - Correcting terminology to match the style guide If the changes look good with no major issues, indicate approval in the summary. For minor issues, suggest improvements but still recommend approval. Only request changes if there are significant problems that need to be fixed before merge. ## Validation After creating `review.json`: - Validate JSON with `jq . review.json` - if this fails, fix the JSON syntax and try again - Verify all paths exist in the PR diff and match the changed files - Check that line numbers are within the changed files and reference lines that were actually modified - Ensure comment spans don't exceed 10 lines ## Signal logging After creating and validating `review.json` (immediately after the Validation section above), emit a summary record for the `improve-drafting-skills` outer loop. Do this before any step that submits or hands off the review — the marker must appear in the Oz run output regardless of how the review is ultimately published. Apply only when reviewing an agent-authored PR (branch created by a drafting skill, or commit author is `oz-agent@warp.dev`). 1. Count comments in `review.json` by severity label (`🚨 [CRITICAL]`, `⚠️ [IMPORTANT]`, `💡 [SUGGESTION]`, `🧹 [NIT]`). 2. Identify the top 3 issue categories by frequency (use the `check` name if available from style lint output, or infer a short category from the comment body). 3. Determine the skill used from the PR branch name or PR description if available. 4. Include the following structured marker in your **text response** (write it as part of your agent message, not via a shell `echo` command). This ensures it appears as a `TextContentBlock` in the conversation, where `oz run get --conversation` can reliably retrieve it: ``` [SIGNAL:pr-review] {"date":"YYYY-MM-DD","pr":"NNN","branch":"branch-name","head_sha":"abc1234","skill_used":"draft_feature_doc","reviewer_login":"GITHUB_LOGIN","verdict":"Request changes","critical":N,"important":N,"suggestions":N,"nits":N,"top_categories":["category (N)","category (N)","category (N)"],"actionable_findings":["`path/to/file.mdx:42` — [IMPORTANT] Explain the problem. Requested change: state the concrete resolution."]} ``` Set `head_sha` to the exact commit SHA this review evaluated (the head SHA `.github/workflows/agent-docs-review.yml` passed in, or `gh pr view NNN --json headRefOid --jq .headRefOid` when reviewing interactively). A push of a new commit makes any earlier signal for this PR stale; the collector in `improve-drafting-skills` keys its `review_outcome` lookup on this field matching the PR's current head. Include one `actionable_findings` entry for every critical, important, suggestion, or nit. Each entry must identify the changed file and line (or quote the affected text), explain the issue, and state the requested resolution. The GitHub Actions publisher renders these entries in the review body, so category-only findings are invalid. The `improve-drafting-skills` outer loop reads this signal from the conversation via `oz run get --conversation`, scanning assistant `TextContentBlock` messages for the marker. No git operations are required. ## Publishing a GitHub review After creating `review.json`, publishing the signal, and completing validation, create one GitHub PR review pinned to the evaluated head SHA. The review body must include the same `[SIGNAL:pr-review]` JSON record used in the text response. When `.github/workflows/agent-docs-review.yml` starts this skill, do not publish the review yourself. Emit the signal with `reviewer_login` set to `github-actions[bot]`; the GitHub Actions runner publishes the review with its short-lived token after the cloud agent returns. 1. Determine the authenticated reviewer: ```bash REVIEWER_LOGIN=$(gh api user --jq .login) ``` Every verdict — `Approve`, `Approve with nits`, and `Request changes` — publishes as a non-blocking `COMMENT` GitHub review event. The required `Agent docs review` status check still fails on critical or important findings. Engineering review requests remain advisory. 2. Write the signal JSON object to `/tmp/review-signal.json`, set its `reviewer_login` to `$REVIEWER_LOGIN`, and render that same object as the `[SIGNAL:pr-review]` line in the final response. Then construct the pinned review request from `review.json`: ```bash HEAD_SHA="<evaluated PR head SHA>" VERDICT="<Approve|Approve with nits|Request changes>" export HEAD_SHA VERDICT python3 - <<'PY' import json import os from pathlib import Path review = json.loads(Path("review.json").read_text()) signal = json.loads(Path("/tmp/review-signal.json").read_text()) event = { "Approve": "COMMENT", "Approve with nits": "COMMENT", "Request changes": "COMMENT", }[os.environ["VERDICT"]] findings = [] for comment in review["comments"]: location = f"`{comment['path']}:{comment['line']}`" finding = comment["body"].splitlines()[0] findings.append(f"- {location} — {finding}") findings_text = "\n".join(findings) or "- No inline findings." payload = { "commit_id": os.environ["HEAD_SHA"], "event": event, "body": ( f"## Review summary\n{review['summary'].strip()}\n\n" f"## Findings\n{findings_text}\n\n" f"## Verdict\n{os.environ['VERDICT']}\n\n" f"<!-- [SIGNAL:pr-review] {json.dumps(signal, sort_keys=True)} -->" ), "comments": review["comments"], } Path("/tmp/review-request.json").write_text(json.dumps(payload)) PY ``` 3. Submit the review: ```bash gh api --method POST "repos/OWNER/REPO/pulls/PR_NUMBER/reviews" \ --input /tmp/review-request.json ``` Replace `OWNER/REPO`, `PR_NUMBER`, and `commit_id` with the pull request being reviewed and its exact head SHA. The workflow verifies that a current-head review from `reviewer_login` contains the exact signal fields. Do not leave the signal only in the agent response or an issue comment.