web-application-spider-crawler

name	Web Application Spider & Crawler
description	Intelligent web crawler for discovering endpoints, forms, links, and API routes. Extracts JavaScript endpoints and supports depth-limited crawling with categorized output.

Web Application Spider & Crawler

Intelligent web crawler for discovering attack surface, endpoints, forms, and API routes.

Features

• Smart Crawling: Breadth-first traversal with depth limiting
• Link Extraction: HTML <a>, <form>, <script>, <link> tags
• JavaScript Analysis: Static extraction of API endpoints from JS files
• Form Discovery: Extracts forms with fields, methods, and actions
• Scope Control: Same-domain, same-host, or unrestricted crawling
• Categorization: Pages, APIs, forms, assets (CSS, JS, images)
• Deduplication: URL normalization to avoid redundant crawling
• JSON Output: Structured output for downstream tools

Architecture

web-spider/
├── SKILL.md                # This file - AgentSkills.io spec
├── README.md              # User documentation
├── scripts/
│   ├── spider.sh          # Main crawler with link extraction
│   ├── js-endpoints.sh    # JavaScript endpoint extractor
│   ├── extract-forms.sh   # HTML form parser
│   └── url-normalize.sh   # URL deduplication helper
├── references/
│   ├── URL_PATTERNS.md    # Common endpoint patterns
│   └── FORM_EXTRACTION.md # Form parsing guide
└── tests/
    └── test-suite.sh      # Integration tests

Quick Start

# Basic crawl
./scripts/spider.sh https://example.com

# Deep crawl with JS analysis
./scripts/spider.sh https://example.com \
  --max-depth 5 \
  --extract-js

# Authenticated crawl
./scripts/spider.sh https://app.example.com \
  --cookie "PHPSESSID=abc123" \
  --max-pages 200

Output Format

{
  "target": "https://example.com",
  "start_time": "2026-03-16T09:00:00Z",
  "end_time": "2026-03-16T09:02:15Z",
  "crawl_config": {
    "max_depth": 3,
    "max_pages": 100,
    "scope": "same-domain"
  },
  "discovered": {
    "pages": [
      {
        "url": "https://example.com/about",
        "depth": 1,
        "status_code": 200,
        "title": "About Us"
      }
    ],
    "api_endpoints": [
      
         
         
         
         
      
    
     
      
         
         
         
         
             
             
        
      
    
     
      
         
         
      
    
  
   
     
     
     
     
     
     
  

Usage Examples

1. Discover All Endpoints

./scripts/spider.sh https://example.com \
  --max-depth 4 \
  --max-pages 200 \
  --output endpoints.json

2. Extract API Endpoints from JavaScript

# Single file
./scripts/js-endpoints.sh https://app.example.com/static/app.js

# All JS files from a crawl
cat crawl-results.json | \
  jq -r '.discovered.static_assets[] | select(.type=="javascript") | .url' | \
  while read js_url; do
    ./scripts/js-endpoints.sh "$js_url"
  done

3. Find All Forms

./scripts/spider.sh https://example.com | \
  jq '.discovered.forms[]'

4. Integration with POST Parameter Tester

# Crawl and test all POST forms
./scripts/spider.sh https://example.com --output spider.json

cat spider.json | jq -r '.discovered.forms[] | 
  select(.method=="POST") | 
  "\(.url) \(.fields | map(.name + "=test") | join("&"))"' | \
  while read url params; do
    /skills/pentest/post-parameter-tester/scripts/test-post.sh "$url" "$params"
  done

Scope Modes

• same-domain: Crawl all subdomains

  • https://example.com → includes api.example.com, www.example.com

• same-host: Exact hostname only

  • https://app.example.com → excludes api.example.com

• all: Follow all links (use with caution!)

  • May crawl external sites, CDNs, social media

JavaScript Endpoint Extraction

The js-endpoints.sh script detects:

• fetch() API: fetch('/api/users')
• axios: axios.get('/api/data')
• XMLHttpRequest: xhr.open('GET', '/api/endpoint')
• jQuery AJAX: $.ajax({url: '/api/resource'})
• REST patterns: /api/v1/users, /v2/products/{id}
• GraphQL: /graphql, query { users { ... } }

Performance Considerations

• Rate Limiting: 1 second delay between requests (default)
• Max Pages: Default 100 pages to avoid excessive crawling
• Timeout: 10 second timeout per request
• Memory: Stores all URLs in memory - limit with --max-pages

Limitations

• No JS Rendering: Static HTML only (can't execute JavaScript)
  • Use Puppeteer/Playwright for SPAs
• No Authentication: Cookie-based auth only (no OAuth flows)
• No Form Submission: Discovery only, not interaction
• Static Analysis: May miss dynamic/runtime-generated endpoints

Roadmap

• Headless browser support (Puppeteer) for JS rendering
• GraphQL schema introspection
• OpenAPI/Swagger endpoint import
• Screenshot capture of discovered pages
• Broken link detection
• Sitemap.xml parsing
• robots.txt analysis
• Concurrent/parallel crawling
• Resume capability for large crawls
• Authentication flow support (OAuth, SAML)

Integration Examples

With Directory Enumeration

# Crawl discovered pages, then enumerate directories
./scripts/spider.sh https://example.com --output spider.json

cat spider.json | jq -r '.discovered.pages[].url' | \
  sed 's|/[^/]*$||' | sort -u | \
  while read dir; do
    gobuster dir -u "$dir" -w /usr/share/wordlists/dirb/common.txt
  done

With Vulnerability Scanner

# Discover endpoints, then scan with nuclei
./scripts/spider.sh https://example.com | \
  jq -r '.discovered.pages[].url' > urls.txt

nuclei -l urls.txt -t ~/nuclei-templates/

Contributing

Improvements welcome! Add endpoint patterns to references/URL_PATTERNS.md.

---

Author: Sir Agravain, Knight of Pentesting
Created: 2026-03-16
Version: 1.0.0