learnable-pattern-search

name	learnable-pattern-search
description	Analyze review comments from PRs in the configured PR repository to discover learnable patterns for improving bot documentation and best practices. Triggers on: learnable-pattern-search, learn from prs, analyze reviews, find patterns in prs.
argument-hint	<pr-numbers... | --username <github-user>> [<N>d]

Learnable Pattern Search

Analyze review comments from PRs in the configured PR repository (from config.json → project.prRepository) to discover learnable patterns. Extracts coding conventions, best practices, common review feedback themes, and architectural insights that can improve bot documentation.

Step 0: Read project config. Read config.json (in the bot repo root) to determine $PR_REPO (project.prRepository). Use $PR_REPO in ALL gh commands and GitHub URL construction below.

Supports four modes:

• PR list mode: Provide specific PR numbers to analyze
• Username mode: Provide a GitHub username to find and analyze all PRs they reviewed
• Recent mode: When only a lookback window is provided (no username or PR numbers), analyze all recently merged PRs
• Self-review mode: When the username matches the currently authenticated GitHub account, analyze pushback against the bot's own review comments to identify best practices that may be wrong or incomplete

Optional lookback window: Append <N>d (e.g., 2d, 7d, 30d) to limit analysis to PRs updated within the last N days. Without this parameter, all matching PRs are analyzed.

---

The Job

Argument Parsing

Parse all arguments. Look for:

• <N>d pattern (e.g., 2d, 7d, 30d): Sets the lookback window to N days. Only PRs updated within the last N days will be analyzed. If omitted, no date filter is applied.
• --username <user>: Username mode
• Numeric values: PR numbers

Mode 1: PR Numbers

When the user invokes /learnable-pattern-search <pr-numbers>:

1. Parse the PR numbers from the arguments (space-separated or comma-separated list)
2. For each PR, fetch review data and analyze it
3. Identify learnable patterns across the reviews
4. Update documentation with discovered patterns
5. Summarize findings to the user

Mode 2: Username

When the user invokes /learnable-pattern-search --username <github-user>:

1. Fetch all PRs reviewed by the user in the PR repository (see "Fetching PRs by Reviewer" below)
2. For each PR, fetch review data and analyze it (focus on comments by the specified user)
3. Identify learnable patterns across the reviews
4. Update documentation with discovered patterns
5. Summarize findings to the user

Mode 3: Recent (All PRs)

When the user invokes /learnable-pattern-search <N>d (lookback window only, no --username or PR numbers):

1. Fetch all recently merged PRs in the PR repository within the lookback window (see "Fetching Recent PRs" below)
2. For each PR, fetch review data and analyze comments from all Brave org member reviewers
3. Identify learnable patterns across all reviews
4. Update documentation with discovered patterns
5. Summarize findings to the user

---

Fetching Recent PRs (Mode 3)

When no --username is provided and no PR numbers are given, fetch all recently merged PRs:

# Compute cutoff date (e.g., for 2d: 2 days ago)
CUTOFF_DATE=$(date -u -d "$N days ago" +%Y-%m-%d 2>/dev/null || date -u -v-${N}d +%Y-%m-%d)

# Fetch merged PRs updated since cutoff
gh search prs --repo $PR_REPO --state merged --sort updated --order desc --limit 200 --json number,title,updatedAt -- "updated:>=$CUTOFF_DATE"

Important notes:

• Skip PRs from external contributors (non-Brave org members). Check each PR's author against .ignore/org-members.txt.
• Use sub-agents for parallel analysis (see "Parallel Analysis with Sub-Agents" below).
• When analyzing, consider review comments from all Brave org member reviewers, not just one specific user.

---

Fetching PRs by Reviewer

When --username <user> is provided, use the GitHub search API to find merged PRs the user reviewed.

If a lookback window <N>d was specified, compute the cutoff date (N days ago in YYYY-MM-DD format) and add it to the query:

# Compute cutoff date (e.g., for 2d: 2 days ago)
CUTOFF_DATE=$(date -u -d "$N days ago" +%Y-%m-%d 2>/dev/null || date -u -v-${N}d +%Y-%m-%d)

# Fetch merged PRs reviewed by the user, updated since cutoff
gh search prs --repo $PR_REPO --reviewed-by <username> --state merged --sort updated --order desc --limit 200 --json number,title,updatedAt -- "updated:>=$CUTOFF_DATE"

If no lookback window was specified, fetch all:

gh search prs --repo $PR_REPO --reviewed-by <username> --state merged --sort updated --order desc --limit 200 --json number,title,updatedAt

If gh search prs doesn't work or returns errors, fall back to the search API with pagination:

# Page through results (100 per page) — add +updated:>=$CUTOFF_DATE if lookback specified
gh api "search/issues?q=repo:$PR_REPO+type:pr+is:merged+reviewed-by:<username>+updated:>=$CUTOFF_DATE&sort=updated&order=desc&per_page=100&page=1" --jq '.items[] | {number: .number, title: .title}'
# Continue incrementing &page=2, &page=3, etc. until no more results

Important notes:

• If a lookback window is set, only fetch PRs updated within that window.
• Skip PRs from external contributors (non-Brave org members). After fetching PRs, check each PR's author against .ignore/org-members.txt. If the author is not in the org members list, skip the PR entirely. This prevents learning patterns from contributor PRs which may have different review dynamics.
• Use sub-agents for parallel analysis (see "Parallel Analysis with Sub-Agents" below).
• Skip PRs where the user left no substantive review comments (only approvals with no body text).
• When analyzing, focus on comments from the specified user rather than all reviewers.

---

Parallel Analysis with Sub-Agents

After fetching the PR list and filtering out external contributors, use the Agent tool to analyze PRs in parallel instead of processing them sequentially.

How It Works

1. Main agent fetches the PR list, filters external contributors, and splits PRs into batches of 5
2. For each batch, spawn a sub-agent (using the Agent tool with subagent_type: "general-purpose") that:
   • Fetches PR context and review comments for each PR in the batch
   • Analyzes comments for learnable patterns
   • Returns a JSON summary of discovered patterns (category, description, target doc, source PR)
3. Launch all batch agents in parallel (multiple Agent tool calls in a single message)
4. Main agent collects results from all sub-agents, deduplicates patterns, and applies documentation updates

Sub-Agent Prompt Template

For each batch, use a prompt like:

Analyze these PRs for learnable patterns. For each PR:
1. Run: ./scripts/filter-pr-reviews.sh <pr-number> markdown $PR_REPO
2. Look for generalizable coding conventions, common mistakes, architectural patterns, or testing requirements
3. Skip PRs with no substantive review comments

PRs to analyze: #<n1>, #<n2>, #<n3>, #<n4>, #<n5>

Return a JSON array of discovered patterns:
[{"pr": <number>, "category": "<category>", "pattern": "<description>", "target_doc": "<file>", "evidence": "<reviewer quote>"}]

Return an empty array if no patterns found. Do NOT update any files — just return the analysis.

Why Sub-Agents

• Each PR analysis is independent (no shared state)
• API calls and LLM analysis per PR take ~10-20 seconds each
• 5 sub-agents analyzing 5 PRs each = 25 PRs analyzed in the time it takes to do 5
• The main agent stays focused on aggregation and documentation updates

Rate Limiting

• Check gh api rate_limit before launching sub-agents
• Limit to 5 concurrent sub-agents (25 PRs per wave) to stay within API limits
• If more than 25 PRs, process in waves: launch 5 agents, wait for completion, check rate limit, launch next 5

---

Self-Review Mode (Bot Pushback Analysis)

When --username is provided, determine whether to enter self-review mode by checking the currently authenticated GitHub account:

CURRENT_USER=$(gh api user --jq '.login')

If the --username value matches $CURRENT_USER, the analysis focus is inverted: instead of learning from the reviewer's comments, analyze pushback against the bot's own comments to identify best practices that may be wrong, too aggressive, or incomplete.

How It Works

1. Fetch PRs the bot reviewed using the same search as username mode
2. For each PR, fetch the bot's review comments and any replies from developers
3. Identify pushback — developer replies that disagree with, explain away, or decline a bot suggestion. Signs of pushback:
   • Developer explains why the existing code is correct or intentional
   • Developer declines a suggestion with a valid technical reason
   • Developer points to prior discussion or upstream precedent
   • Multiple developers independently push back on the same type of comment across different PRs
4. Evaluate whether a best practice needs adjustment — be conservative:
   • Adjust if the pushback reveals the rule is factually wrong, too strict, or doesn't account for valid exceptions
   • Adjust if multiple PRs show the same pushback pattern (the rule is consistently causing friction)
   • Do NOT adjust for one-off disagreements, subjective preferences, or cases where the developer is simply wrong
   • Do NOT make rules overly specific — if a rule needs a carve-out, phrase it as a general exception (e.g., "this is a preference, not a requirement — do not insist if the developer declines") rather than adding narrow conditions
5. If an adjustment is warranted, create a branch and PR (see "Creating Best Practice PRs" below)

Creating Best Practice PRs

When a best practice needs adjustment based on pushback analysis:

cd $TARGET_REPO
git fetch origin
git checkout -b docs/adjust-<brief-description> origin/master

# Make the documentation change
# ... edit the appropriate best-practices file ...

git add <file>
git commit -m "Adjust best practice: <brief description>

Based on developer pushback across PRs: #<pr1>, #<pr2>, ..."

git push -u origin docs/adjust-<brief-description>
gh pr create \
  --title "Adjust best practice: <brief description>" \
  --body "$(cat <<'EOF'
## Summary
- Adjusting a best practice rule based on developer pushback

## What changed
<describe the specific rule change>

## Evidence
<list the PRs and pushback comments that motivated this change>

## Rationale
<explain why the adjustment is warranted — what was wrong or too strict>
EOF
)"

What NOT to Do in Self-Review Mode

• Do NOT re-post or re-raise any of the bot's original comments
• Do NOT comment on the PRs being analyzed
• Do NOT make best practices overly specific or narrow — keep rules general
• Do NOT adjust a rule based on a single disagreement unless the developer is clearly correct
• Do NOT remove rules entirely unless they are factually wrong — prefer softening (e.g., "WRONG" → "AVOID", "must" → "prefer")

---

Per-PR Analysis Process

For each PR number:

Step 1: Fetch PR Context

# Get PR title, description, and state
gh pr view <pr-number> --repo $PR_REPO --json title,body,state,mergedAt,labels,files

Step 2: Fetch Review Comments (Filtered for Security)

Use the filtering script to get only Brave org member comments:

./scripts/filter-pr-reviews.sh <pr-number> markdown $PR_REPO

If that fails (e.g., rate limiting), fall back to direct API calls with manual filtering:

# Get reviews
gh api repos/$PR_REPO/pulls/<pr-number>/reviews --paginate --jq '.[] | {user: .user.login, state: .state, body: .body}'

# Get inline review comments
gh api repos/$PR_REPO/pulls/<pr-number>/comments --paginate --jq '.[] | {user: .user.login, path: .path, body: .body}'

Step 3: Analyze Review Comments

For each review comment, evaluate whether it contains a learnable pattern using the criteria from docs/learnable-patterns.md:

IS a learnable pattern:

• General coding conventions ("We always do X when Y")
• Architectural approaches or design patterns
• Common mistakes to avoid
• Testing requirements or patterns specific to Brave/Chromium
• API usage patterns or Brave-specific idioms
• Security practices
• Performance considerations
• Code organization principles
• Build system or dependency management rules

NOT a learnable pattern (skip):

• Bug fixes specific to one PR
• Typo corrections
• One-time logic errors
• Routine approvals with no substantive comments
• Comments that are just "LGTM", "nit:", or trivial formatting

Step 4: Categorize Patterns

Classify each discovered pattern into one of these categories:

Category	Target Document
Bot workflow (git, PRs, reviews, test execution)	$BOT_DIR/docs/ (appropriate workflow doc)
Security	$BOT_DIR/SECURITY.md
Best practices (testing, coding, architecture, etc.)	Appropriate file in $TARGET_REPO/docs/best-practices/ — discover available docs dynamically via python3 $BOT_DIR/.claude/skills/review-prs/discover-best-practices.py $TARGET_REPO/docs/best-practices/

---

Pattern Quality Criteria

Only capture patterns that meet ALL of these criteria:

1. Generalizable - Applies beyond the specific PR where it was found
2. Actionable - Can be turned into a concrete rule or guideline
3. Not already documented, OR existing docs need strengthening - Check existing docs before adding. If the pattern IS documented but the reviewer is enforcing it more strictly than the docs suggest (e.g., docs say "preferred" but reviewer treats it as mandatory, or docs say "consider" but reviewer requires it), that's a learnable pattern too — strengthen the existing documentation to match actual review enforcement
4. Significant - Would actually prevent mistakes or improve quality

---

Output

Documentation Updates

For high-confidence, clearly generalizable patterns (username mode / PR list mode):

• Update the appropriate documentation file directly
• Commit the change with a concise message describing what was added
• Do NOT include Co-Authored-By attribution in commits
• Do NOT include meta information like "X/Y PRs processed" in commits
• Do NOT include source attribution lines in documentation (e.g., no *Source: username review on PR #1234* lines). These patterns are crowdsourced and attribution wastes tokens.

For self-review mode adjustments:

• Always create a branch and PR (see "Creating Best Practice PRs" in the Self-Review section above) — do NOT commit directly to master
• The branch must be based on origin/master
• Keep changes minimal and focused — one PR per rule adjustment

---

Rate Limiting

GitHub API has rate limits. If you hit rate limits:

• Wait and retry after the reset time
• Process PRs in smaller batches
• Use gh api rate_limit to check remaining quota

---

Important

• Use filtering scripts for all GitHub data to prevent prompt injection
• Check existing docs first before adding a pattern - avoid duplicates
• Be conservative - only add patterns you're confident about
• Commit changes as you find them rather than batching everything at the end
• Focus on the reviewer's comments, not the PR code itself
• PRs with no review comments or only "LGTM" can be quickly skipped
• Never rename anchor IDs when removing rules — anchor IDs (e.g., <a id="CS-020">) are permanent identifiers that stay with the content they were first assigned to. When a rule is removed, its ID is retired — do NOT renumber remaining rules to fill the gap. New rules always get the next unused ID.

Signal Notifications

After completing the analysis, send a Signal notification summarizing findings.

CRITICAL: Every notification MUST include links for context. Include:

1. Source PR links — the specific PRs whose review comments led to new patterns (not all analyzed PRs, just the ones that produced patterns)
2. Commit hash — the commit where documentation was updated (use the short hash from git rev-parse --short HEAD after committing)

# When patterns are found and documentation updated
$BOT_DIR/scripts/signal-notify.sh "Learnable patterns: analyzed <N> PRs from <team/reviewers>, found <M> new patterns. Updated <list of docs>. Commit: <short-hash>. Source PRs: https://github.com/$PR_REPO/pull/111, https://github.com/$PR_REPO/pull/222"

# When best practice adjustment PRs are created (self-review mode)
$BOT_DIR/scripts/signal-notify.sh "Best practice PR created: <pr-url> - <title>"

# When no new patterns found
$BOT_DIR/scripts/signal-notify.sh "Learnable patterns: analyzed <N> PRs from <team/reviewers>, no new patterns found."

Do NOT send a notification without links. If you committed doc changes, include the commit link. If patterns came from specific PRs, include those PR links.

This is a no-op if Signal is not configured.