doc-scraper

namespace	aiwg
name	doc-scraper
description	Scrape documentation websites into organized reference files. Use when converting docs sites to searchable references or building Claude skills.
tools	Read, Write, Bash, WebFetch
platforms	["all"]

Documentation Scraper Skill

Purpose

Single responsibility: Convert documentation websites into organized, categorized reference files suitable for Claude skills or offline archives. (BP-4)

Grounding Checkpoint (Archetype 1 Mitigation)

Before executing, VERIFY:

• Target URL is accessible (test with curl -I)
• Documentation structure is identifiable (inspect page for content selectors)
• Output directory is writable
• Rate limiting requirements are known (check robots.txt)

DO NOT proceed without verification. Inspect before scraping.

Uncertainty Escalation (Archetype 2 Mitigation)

ASK USER instead of guessing when:

• Content selector is ambiguous (multiple <article> or <main> elements)
• URL patterns unclear (can't determine include/exclude rules)
• Category mapping uncertain (content doesn't fit predefined categories)
• Rate limiting unknown (no robots.txt, unclear ToS)

NEVER substitute missing configuration with assumptions.

Context Scope (Archetype 3 Mitigation)

Context Type	Included	Excluded
RELEVANT	Target URL, selectors, output path	Unrelated documentation
PERIPHERAL	Similar site examples for selector hints	Historical scrape data
DISTRACTOR	Other projects, unrelated URLs	Previous failed attempts

Workflow Steps

Step 1: Verify Target (Grounding)

# Test URL accessibility
curl -I <target-url>

# Check robots.txt
curl <base-url>/robots.txt

# Inspect page structure (use browser dev tools or fetch sample)

Step 2: Create Configuration

Generate scraper config based on inspection:

{
  "name": "skill-name",
  "description": "When to use this skill",
  "base_url": "https://docs.example.com/",
  "selectors": {
    "main_content": "article",
    "title": "h1",
    "code_blocks": "pre code"
  },
  "url_patterns": {
    "include": ["/docs", "/guide", "/api"],
    "exclude": ["/blog", "/changelog", "/releases"]
  },
  "categories": {
    "getting_started":   
       
       
  
   
   

Step 3: Execute Scraping

Option A: With skill-seekers (if installed)

# Verify skill-seekers is available
pip show skill-seekers

# Run scraper
skill-seekers scrape --config config.json

# For large docs, use async mode
skill-seekers scrape --config config.json --async --workers 8

Option B: Manual scraping guidance

1. Use sitemap.xml or crawl starting URL
2. Extract content using configured selectors
3. Categorize pages based on URL patterns and keywords
4. Save to organized directory structure

Step 4: Validate Output

# Check output structure
ls -la output/<skill-name>/

# Verify content quality
head -50 output/<skill-name>/references/index.md

# Count extracted pages
find output/<skill-name>_data/pages -name "*.json" | wc -l

Recovery Protocol (Archetype 4 Mitigation)

On error:

1. PAUSE - Stop scraping, preserve already-fetched pages
2. DIAGNOSE - Check error type:
   • Connection error → Verify URL, check network
   • Selector not found → Re-inspect page structure
   • Rate limited → Increase delay, reduce workers
   • Memory/disk → Reduce batch size, clear temp files
3. ADAPT - Adjust configuration based on diagnosis
4. RETRY - Resume from checkpoint (max 3 attempts)
5. ESCALATE - Ask user for guidance

Checkpoint Support

State saved to: .aiwg/working/checkpoints/doc-scraper/

Resume interrupted scrape:

skill-seekers scrape --config config.json --resume

Clear checkpoint and start fresh:

skill-seekers scrape --config config.json --fresh

Output Structure

output/<skill-name>/
├── SKILL.md              # Main skill description
├── references/           # Categorized documentation
│   ├── index.md          # Category index
│   ├── getting_started.md
│   ├── api_reference.md
│   └── guides.md
├── scripts/              # (empty, for user additions)
└── assets/               # (empty, for user additions)

output/<skill-name>_data/
├── pages/                # Raw scraped JSON (one per page)
└── summary.json          # Scrape statistics

Configuration Templates

Minimal Config

{
  "name": "myframework",
  "base_url": "https://docs.example.com/",
  "max_pages": 100
}

Full Config

{
  "name": "myframework",
  "description": "MyFramework documentation for building web apps",
  "base_url": "https://docs.example.com/",
  "selectors": {
    "main_content": "article, main, div[role='main']",
    "title": "h1, .title",
    "code_blocks": "pre code, .highlight code",
    "navigation": "nav, .sidebar"
  },
  "url_patterns": {
    "include": ["/docs/", "/api/", "/guide/"],
    "exclude": ["/blog/", "/changelog/", "/v1/", "/v2/"
  
   
        
       
        
        
       
  
   
   
   
     
     
  

Troubleshooting

Issue	Diagnosis	Solution
No content extracted	Selector mismatch	Inspect page, update main_content selector
Wrong pages scraped	URL pattern issue	Check include/exclude patterns
Rate limited	Too aggressive	Increase rate_limit to 1.0+ seconds
Memory issues	Too many pages	Add max_pages limit, enable checkpoints
Categories wrong	Keyword mismatch	Update category keywords in config

References

• Skill Seekers: https://github.com/jmagly/Skill_Seekers
• REF-001: Production-Grade Agentic Workflows (BP-1, BP-4, BP-9)
• REF-002: LLM Failure Modes (Archetype 1-4 mitigations)