Analyze internal link structure by crawling a domain. Identifies orphan pages, underlinked pages (configurable heuristic, default fewer than 3 inbound links), and broken internal links. Suggests anchor text for top 5 underlinked pages. Reuses existing fetch/parse scripts. Optional Ahrefs enrichment. Use when user says "internal links", "link structure", "orphan pages", "internal linking", "link graph", "anchor text audit".
Analyze internal link structure by crawling a domain. Identifies orphan pages, underlinked pages (configurable heuristic, default fewer than 3 inbound links), and broken internal links. Suggests anchor text for top 5 underlinked pages. Reuses existing fetch/parse scripts. Optional Ahrefs enrichment. Use when user says "internal links", "link structure", "orphan pages", "internal linking", "link graph", "anchor text audit".
allowed-tools
["Read","Bash","Glob","WebFetch","ToolSearch"]
Internal Link Audit
Crawls a domain to build an internal link graph. Identifies orphan pages, underlinked pages,
and broken internal links. Suggests anchor text improvements.
Inputs
domain: Target domain URL (e.g., https://example.com). Include protocol.
max_pages (optional): Max pages to crawl. Default: 100. Maximum: 200.
Execution
Step 1: Crawl Site
Use scripts/fetch_page.py and scripts/parse_html.py to crawl the site:
# Fetch homepage
python3 scripts/fetch_page.py <domain>
# Parse HTML to extract links
python3 scripts/parse_html.py <html_content>
Follow internal links only (same domain hostname). Normalize URLs:
Remove trailing slashes (treat /about and /about/ as same)
Remove URL fragments (#section)
Preserve query strings only if they appear to be content (not tracking: strip utm_*, ref=, source=)
Respect robots.txt: fetch <domain>/robots.txt first, skip disallowed paths.
Cap crawl at max_pages. Track crawl queue (BFS order from homepage).
Step 2: Build Link Graph
For each crawled page, record:
Source URL
Target URL (all internal links found)
Anchor text for each link
Build adjacency structure:
outbound[url] = list of (target, anchor_text)
inbound[url] = list of (source, anchor_text)
Step 3: Calculate Per-Page Metrics
For each crawled page:
Inbound internal links count (links from other crawled pages pointing here)
Outbound internal links count (links from this page to other pages)
Link depth from homepage (BFS level at which this page was first discovered)
Step 4: Identify Issues
Orphan pages (hard finding): Pages discovered via sitemap (<domain>/sitemap.xml) or linked from other pages but with zero inbound internal links. Fetch sitemap first to identify all known URLs.
Underlinked pages (heuristic flag): Pages with fewer inbound internal links than a configurable threshold (default: 3). This threshold is a heuristic for prioritizing internal-linking work — Google defines no numeric minimum, so it is not a Google requirement.
