Fetch documentation from a repository and distill it into a practical reference
doc with concrete recipes and patterns. The output goes to references/ in the
repo root.
Use this lookup table to resolve short names to repos. If $ARGUMENTS doesn't
match a known name, treat it as a GitHub URL or owner/repo.
The generated reference doc includes frontmatter with per-source indexing
metadata. Each source tracks its own date so partial re-indexing is possible
and so it's clear which sources have actually been fetched:
During indexing, if an issue title matches any pattern (case-insensitive),
skip it. If you encounter a new category of noise issues during distillation,
add the pattern to exclude-issue-patterns for future runs.
Every repo has different labels. Instead of hardcoding which labels matter,
discover and assess them dynamically.
Do NOT hardcode label names. Read the actual label names and descriptions,
then use judgment to classify them. Every repo is different — git-branchless
has "has workaround" and "answered", another repo might have "solved" or
"recipe".
-
Resolve the repo from $ARGUMENTS using the lookup table above, or
parse as owner/repo or full URL.
-
Check for incremental update as described above. If up to date, skip.
-
Fetch the wiki (if it exists):
tmp_dir=$(mktemp -d "${TMPDIR:-/tmp}/claude-repo-index.XXXXXX")
git clone --depth 1 "https://github.com/${owner}/${repo}.wiki.git" "$tmp_dir/wiki" 2>/dev/null || true
-
Discover and fetch documentation:
Documentation lives in different places across repos. Use a multi-pass
discovery approach and track what was found in doc-sources frontmatter.
4a. Fetch README (always)
gh api "repos/${owner}/${repo}/readme" --jq '.content | @base64d' > "$tmp_dir/README.md"
4b. Scan repo tree for doc-like files
Fetch the repo tree and filter for documentation files. Don't hardcode
directory names — search broadly:
First, get the default branch:
default_branch=$(gh api "repos/${owner}/${repo}" --jq '.default_branch')
gh api "repos/${owner}/${repo}/git/trees/${default_branch}?recursive=1" \
--jq '.tree[] | select(.type=="blob") | .path' \
| grep -iE '\.(md|adoc|rst|txt)$' \
| grep -iE '^(docs?/|documentation/|man/|guide|readme|contributing|changelog|faq|usage|tutorial)' \
> "$tmp_dir/doc-paths.txt"
Also catch top-level doc files that aren't README:
gh api "repos/${owner}/${repo}/git/trees/${default_branch}?recursive=1" \
--jq '.tree[] | select(.type=="blob") | .path' \
| grep -iE '^[^/]*\.(adoc|rst)$' \
>> "$tmp_dir/doc-paths.txt"
Fetch each discovered file:
while read -r doc_path; do
encoded=$(printf '%s' "$doc_path" | jq -sRr @uri)
gh api "repos/${owner}/${repo}/contents/${encoded}" --jq '.content | @base64d' \
> "$tmp_dir/repo-$(echo "$doc_path" | tr '/' '-')" 2>/dev/null || true
done < "$tmp_dir/doc-paths.txt"
4c. Follow external documentation links
Scan the README and any fetched docs for links to external documentation
sites (readthedocs, GitHub Pages, gitbook, wiki URLs, etc.):
grep -ohE 'https?://[^ )">]+' "$tmp_dir"/*.md "$tmp_dir"/*.adoc 2>/dev/null \
| grep -iE '(readthedocs|gitbook|github\.io|wiki|docs\.)' \
| sort -u > "$tmp_dir/external-links.txt"
For each external link, fetch the page content if it's reachable and
appears to be documentation (HTML or markdown). Use WebFetch for
HTML pages. Record each in doc-sources with type: external.
4d. Reconcile with previous doc-sources
If the existing frontmatter has doc-sources, compare:
- New paths not in previous run → fetch and flag as new discovery
- Previous paths missing from tree → mark
reachable: false, keep
the entry but note it's gone (the content may still be relevant if
the file was moved or renamed)
- Previous external URLs → re-check reachability, update status
- Prune entries marked
reachable: false for 3+ consecutive runs
The doc-sources list persists across runs as the "memory" of what
documentation exists for this repo. Each entry includes a relevance
note explaining why it matters — this helps future runs prioritize.
-
Discover and fetch issues/discussions using multiple search signals.
The goal is a thorough sample of usage-relevant content — not just the top
10. Fetch greedily, deduplicate, and only prompt the user if volume is
unmanageable.
5a. Count total issues to determine strategy
Use the search API to get accurate issue counts (the REST issues endpoint
and open_issues_count both include pull requests):
total_open=$(gh api "search/issues?q=repo:${owner}/${repo}+type:issue+state:open&per_page=1" \
--jq '.total_count')
total_closed=$(gh api "search/issues?q=repo:${owner}/${repo}+type:issue+state:closed&per_page=1" \
--jq '.total_count')
total_issues=$((total_open + total_closed))
- < 500 total (open + closed): Fetch all, paginating fully. Filter out
excludes during distillation.
- 500–2000: Fetch all from value-label and keyword searches (below), plus
top 100 most-reacted. This gives broad coverage without full enumeration.
- > 2000: Prompt the user with the counts and ask how to proceed. Suggest
a strategy like: "There are ~3400 issues. I can fetch all labeled
'answered'/'has workaround' (~180), keyword matches (~250), and top 100
most-reacted. Or I can paginate through everything (~35 API calls). Which
do you prefer?"
5b. Label-based fetches
For each label in the cached value-labels list, fetch all matching issues:
encoded_label=$(printf '%s' "$label_name" | jq -sRr @uri)
gh api "repos/${owner}/${repo}/issues?labels=${encoded_label}&state=all&per_page=100${since_param}" \
--paginate --jq '.[] | select(has("pull_request") | not) | "## Issue #\(.number): \(.title)\n\(.body)\n"'
5c. Keyword searches
Search issues/discussions for usage-pattern keywords. These surface "how do
I...?" and "can I...?" questions that reveal practical workflows:
Important: gh search issues returns a JSON array per call. Pipe each
call through jq individually to emit JSONL — do NOT append raw arrays
with >> (creates invalid JSON when concatenated).
keywords=("how" "can I" "workflow" "workaround" "example" "recipe" "pattern")
for kw in "${keywords[@]}"; do
gh search issues "${kw}" --repo "${owner}/${repo}" --sort reactions \
--json number,title,body,labels,reactions --limit 50 2>/dev/null \
| jq -c '.[] | {number, title, body: (.body // "" | .[0:2000]), labels: [.labels[].name]}' \
>> "$tmp_dir/keyword-issues.jsonl" 2>/dev/null || true
done
For repos with Discussions enabled, also search discussions:
for kw in "${keywords[@]}"; do
gh api graphql -f query='
query {
search(query: "repo:'"${owner}/${repo}"' \"'"${kw}"'\" is:discussion",
type: DISCUSSION, first: 50) {
nodes {
... on Discussion {
number title body url
answer { body }
labels(first:10) { nodes { name } }
}
}
}
}' --jq '.data.search.nodes[] | @json' >> "$tmp_dir/keyword-discussions.jsonl" 2>/dev/null || true
done
5d. Reaction-sorted fallback
Also fetch top most-reacted issues as a catch-all for popular content that
keyword and label searches might miss:
gh search issues --repo "${owner}/${repo}" --sort reactions --order desc --state all \
--limit 100 --json number,title,body,labels,reactions \
--jq '.[] | {number, title, body: (.body // "" | .[0:2000]), labels: [.labels[].name], reactions: .reactions.total_count}'
5e. Deduplicate and filter
Merge all fetched issues/discussions by number. Remove duplicates. Apply
exclude-issue-patterns to titles (case-insensitive). The final set is what
gets distilled into the reference doc.
For each issue/discussion, capture: number, title, body, labels, and
reaction count. Issues with answers (from Discussions) or resolution
comments from maintainers are especially valuable — note the resolution.
-
Extract Local Notes from the existing reference doc (if it exists).
Parse all lines from the one that starts with <!-- BEGIN LOCAL NOTES through
the one that starts with <!-- END LOCAL NOTES (inclusive of those full marker
lines, which may include trailing text and the closing -->). Store this block
verbatim — it must be spliced back into the regenerated doc unchanged.
-
Read all fetched content and distill into a reference doc with this
structure:
---
repo: owner/repo
repo-head: <sha>
repo-indexed: <date>
wiki-head: <sha or null>
wiki-indexed: <date or null>
issues-indexed: <date or null>
discussions-indexed: <date or null>
labels-indexed: <date>
label-head: <sha256 of sorted label names>
doc-sources:
- path: "<relative path in repo>"
type: repo-file
relevance: "<why this file matters>"
- url: "<external URL>"
type: external
relevance: "<why this link matters>"
reachable: true
exclude-issue-patterns:
- "renovate"
- "dependabot"
- "bump version"
- "release v"
value-labels:
- name: "<label>"
reason: "<why this label surfaces useful content>"
issue-stats:
total-fetched: <n>
from-labels: <n>
from-keywords: <n>
from-reactions: <n>
after-dedup: <n>
---
# <Tool Name> Reference
Distilled from <repo URL>, updated <date>.
## Overview
<1-2 paragraph summary of what the tool does and why>
## Installation & Setup
<How to install, configure, prerequisites>
## Core Concepts
<Key mental models needed to use the tool effectively>
## Command Reference
<Commands with practical examples, grouped by workflow>
## Recipes
<Concrete step-by-step patterns for common tasks, written as numbered
procedures. These should be copy-pasteable workflows, not abstract
descriptions. Focus on:
- The happy path for each common operation
- How to recover from mistakes
- Integration with other tools (git-branchless + git-absorb, etc.)>
## Anti-Patterns
<Common mistakes and what to do instead>
## Integration
<How this tool works with the other stacked workflow tools>
<!-- BEGIN LOCAL NOTES — preserved across regeneration -->
## Local Notes
Hard-won lessons, workarounds, and patterns discovered through actual usage.
This section is never overwritten by index-repo-docs. Add entries here when
you solve a pain point or discover undocumented behavior.
<!-- END LOCAL NOTES -->
-
Write the draft to a temp file — never directly to the reference doc.
-
Present a change summary for user review. Do NOT write the final doc
until the user approves. Show, in this order:
- Additions — new sections, recipes, gotchas, anti-patterns
- Changes — modified sections (briefly describe what changed)
- Removals — any sections, recipes, or gotchas from the old doc that
are absent in the draft (flag these prominently — removals need
justification)
- Stats — line count old vs new, recipe count old vs new, issue-ref
(
#NNN) count old vs new
If there are removals, explain why each one was dropped. The user may
reject the draft or ask for revisions. Only proceed to step 10 after
explicit approval.
-
Write the approved doc to the reference file path.
-
Clean up:
rm -rf "$tmp_dir"
-
Report what was generated (full/incremental/skipped), how many source
files were read, and the output path.
