setup-checks

# Set Up AI Checks You are helping a developer set up **AI Checks** — markdown files that define code quality standards. Each check runs as a full AI agent on every PR — not just reading the diff, but able to read files, run commands, and use a browser. If it finds something, it fails the PR with a suggested fix. Otherwise, it passes silently. **Complete all steps below in order.** --- ## Step 1: Orient Welcome the user. Keep it brief — one or two sentences: > We're going to set up AI checks for your repo. I'll explore your codebase, write checks tailored to your project, and get you set up to run them. Then create a todo list for the session: 1. Explore codebase & write checks 2. Try a check 3. Set up CI (optional) 4. Commit & PR Mark task 1 in progress. ## Step 2: Git Setup 1. Verify this is a git repo: `git rev-parse --git-dir 2>/dev/null` — stop if not. 2. Check for uncommitted changes: `git status --porcelain` If dirty, ask: > You have uncommitted changes. I'd like to stash them and create a new branch. OK? If approved: ```bash git rev-parse --abbrev-ref HEAD # save original branch git stash push -m "Stashing before checks setup" git checkout -b add-checks ``` Remember the original branch and whether you stashed — you'll restore later. ## Step 3: Explore & Write Checks **Speed target: under 60 seconds. Stay shallow — surface patterns, don't read entire files.** Run these in parallel where possible: ### Codebase scan - Project structure, languages, frameworks, build system - Linters, formatters, type checkers already in use - Testing framework & patterns - CI/CD workflows - Contributing guidelines (CONTRIBUTING.md, CLAUDE.md, AGENTS.md) ### PR review history (via `gh` CLI, skip if `gh` is not available) ```bash gh pr list --state merged --limit 30 --json number,title # For 3-5 recent PRs with review comments: gh api repos/{owner}/{repo}/pulls/{number}/comments --jq '.[] | {body: .body, path: .path}' ``` Look for: what reviewers commonly flag, what linters don't catch, what areas get the most review attention. ### Propose checks Share 3-5 bullet findings, then propose exactly **3 checks** via a multiselect question (all selected by default). **Cardinal rule: never create a check for something already enforced by a deterministic tool** (linter, type checker, CI step). Checks are for things that require judgment — slop, drifting conventions, security anti-patterns, poor taste. Each proposed check must be specific to this project and clearly valuable. ### Write check files Create `.checks/` and write one `.md` file per check: ```markdown --- name: Human-Readable Check Name description: One-sentence description. --- # Check Name ## Context WHY this check exists. Reference specific project patterns, configs, or review comments you found. ## What to Check ### 1. Category Concrete instructions with specific file paths/patterns and GOOD/BAD code examples from this codebase. ### 2. Category ... ## Key Files List the files most relevant to this check. ## Exclusions Cases where this check does NOT apply. ``` Keep file names lowercase with hyphens (e.g., `error-handling.md`, `api-conventions.md`). ### Update agent config Add a section to the project's agent configuration file so checks become part of the workflow. Choose the right file based on what exists or what tool the user is using: - **CLAUDE.md** — if using Claude Code - **AGENTS.md** — if using a generic agent setup - **COPILOT.md** — if using GitHub Copilot - If none of these exist, create the most appropriate one. Append a section like this: ```markdown ## Checks Before opening a pull request, run all checks in `.checks/` against your current diff and fix any failures before requesting review. ``` Mark task 1 complete. ## Step 4: Try a Check Mark task 2 in progress. Let the user see a check in action. Ask them which check they'd like to try: > Which check would you like to try? I'll run it against the codebase right now. Present the list of checks as options. Once they pick one: 1. Read the selected check file from `.checks/` 2. Scan the codebase against the check's criteria — read the relevant source files, not just a diff 3. Evaluate what you find: - For each finding: state severity (Error / Warning / Info), reference the specific file and line, explain what's wrong and how to fix it - If everything looks good, report PASS - If there are findings, report FAIL and list them 4. Present the result — if it fails, offer to fix the issues Mark task 2 complete. ## Step 5: CI Setup (Optional) Mark task 3 in progress. Ask the user: > Would you like to also run these checks automatically on every PR via GitHub Actions? I'll add the workflow file to the PR — you just need to add an API key as a repository secret. If they say yes, create `.github/workflows/checks.yml`: ```yaml name: AI Checks on: pull_request: types: [opened, synchronize, reopened] concurrency: group: ai-checks-${{ github.event.pull_request.number }} cancel-in-progress: true jobs: checks: runs-on: ubuntu-latest permissions: contents: read pull-requests: write checks: write steps: - uses: actions/checkout@v4 with: fetch-depth: 0 - uses: actions/cache@v4 with: path: ~/.npm key: claude-code-${{ runner.os }} - name: Install agent CLI run: npm install -g @anthropic-ai/claude-code - name: Get PR diff run: | gh pr diff ${{ github.event.pull_request.number }} > /tmp/pr.diff head -c 100000 /tmp/pr.diff > /tmp/pr-truncated.diff env: GH_TOKEN: ${{ github.token }} - name: Run all checks env: ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }} GH_TOKEN: ${{ github.token }} run: | REPORT_FILE=$(mktemp) echo "## AI Check Results" >> "$REPORT_FILE" echo "" >> "$REPORT_FILE" echo "| Check | Result |" >> "$REPORT_FILE" echo "|---|---|" >> "$REPORT_FILE" for check_file in .checks/*.md; do [ -f "$check_file" ] || continue name=$(basename "$check_file" .md) # Create check run in progress CHECK_RUN_ID=$(gh api repos/${{ github.repository }}/check-runs \ -f name="check/$name" \ -f head_sha=${{ github.event.pull_request.head.sha }} \ -f status=in_progress \ --jq '.id') # Run agent with the check prompt and PR diff OUTPUT=$(claude -p "$(cat <<PROMPT You are running a quality check on a pull request. ## Check $(cat "$check_file") ## PR Diff $(cat /tmp/pr-truncated.diff) ## Instructions Evaluate the changed files against this check. Only review changed lines. Do not flag pre-existing issues in unchanged code. Output valid JSON: {"verdict":"PASS" or "FAIL","reason":"...","suggestions":[{"file":"...","line":0,"fix":"..."}]} PROMPT )" --output-format json 2>&1) || true # Parse verdict — claude --output-format json wraps the response in an envelope # with the actual content in .result as a string (possibly markdown-fenced) INNER=$(echo "$OUTPUT" | jq -r '.result // empty' 2>/dev/null | sed 's/^```json//;s/^```//;s/```$//' | jq -r '.' 2>/dev/null || echo "$OUTPUT") VERDICT=$(echo "$INNER" | jq -r '.verdict // "ERROR"' 2>/dev/null || echo "ERROR") REASON=$(echo "$INNER" | jq -r '.reason // "No reason given"' 2>/dev/null || echo "Parse failure") # Extract suggestions for detail text SUGGESTIONS=$(echo "$INNER" | jq -r '.suggestions // [] | .[] | "- **\(.file):\(.line)** — \(.fix)"' 2>/dev/null || echo "") # Build detail markdown DETAIL="## $name\n\n**Verdict:** $VERDICT\n\n### Reason\n\n$REASON" if [ -n "$SUGGESTIONS" ]; then DETAIL="$DETAIL\n\n### Suggestions\n\n$SUGGESTIONS" fi # Complete the check run CONCLUSION=$([ "$VERDICT" = "PASS" ] && echo "success" || echo "failure") gh api repos/${{ github.repository }}/check-runs/$CHECK_RUN_ID \ -X PATCH \ -f status=completed \ -f conclusion="$CONCLUSION" \ -f "output[title]=$(echo "$REASON" | head -c 140)" \ -f "output[summary]=$REASON" \ -f "output[text]=$(printf "$DETAIL")" > /dev/null # Append to report ICON=$([ "$VERDICT" = "PASS" ] && echo "✅" || echo "❌") echo "| $ICON $name | $REASON |" >> "$REPORT_FILE" # Save full output for debugging echo "$OUTPUT" > "/tmp/check-$name.json" done # Post single consolidated comment gh pr comment ${{ github.event.pull_request.number }} --body "$(cat "$REPORT_FILE")" - name: Upload check outputs if: always() uses: actions/upload-artifact@v4 with: name: check-results path: /tmp/check-*.json retention-days: 7 ``` Tell the user they can add their API key as a repository secret by running this command in their terminal (do NOT run it yourself — it prompts for the secret value securely): > To activate the workflow after merging, run this in your terminal: > > ``` > echo "YOUR_API_KEY" | gh secret set ANTHROPIC_API_KEY --repo <owner>/<repo> > ``` > > Replace `YOUR_API_KEY` with your actual Anthropic API key. Fill in the actual `<owner>/<repo>` for them. Mark task 3 complete. ## Step 6: Commit & PR Mark task 4 in progress. Commit everything and open a PR: ```bash git add .checks/ git add .github/workflows/checks.yml 2>/dev/null # if CI was set up git add CLAUDE.md AGENTS.md COPILOT.md 2>/dev/null # whichever was modified git commit -m "Add checks for automated PR review Checks: - [list each check name]" git push origin HEAD gh pr create --title "Add checks for automated PR review" --body "Adds AI-powered code review checks that run on every PR. ## Checks added [list each check with one-line description] ## How it works Each check is a markdown file in \`.checks/\`. They run as full agents on every PR — reading files, running commands, and providing suggested fixes when they find issues." ``` If you stashed changes earlier, restore them: ```bash git checkout <original-branch> git stash pop ``` Mark task 4 complete. Share the PR URL.