web-crawling

name	web-crawling
version	2.0
last_updated	2026-03-15
description	crawl4ai-powered browser crawler for pentest recon. Deep crawl with form discovery, API endpoint capture, and JS file extraction. Use when: 'crawl this site', 'browser crawl', 'capture attack surface', 'map all endpoints', 'authenticated crawl'.

Web Crawling — TRACER 🌐

Trigger Phrases

• "crawl this site"
• "browser crawl target.com"
• "map the attack surface"
• "capture all requests"
• "authenticated crawl"
• "discover endpoints"
• "run TRACER"

Inputs

• target (required): Any format — the crawler auto-normalizes:
  • target.com → https://target.com
  • http://target.com → upgraded to https:// if available, fallback to http://
  • https://target.com/path → used as-is
  • Pass the FULL target string exactly as given by the user/task — do NOT manually strip scheme or path
• taskId (required): Task identifier for output file naming
• credentials (optional): --login-url URL --username USER --password PASS

Workflow

Step 1: Run crawl4ai_crawler.py

python3 <agents-root>/tracer/skills/web-crawling/scripts/crawl4ai_crawler.py \
  -u "${target}" \
  -d 4 \
  --max-pages 200 \
  -o /root/.openclaw/intel/crawl-${taskId}/

Output directory: /root/.openclaw/intel/crawl-${taskId}/

Step 2: Read output files

After crawl completes, read these files from the output dir:

• urls.txt — all discovered URLs (GET endpoints)
• forms.json — all HTML forms with fields (POST endpoints + params)
• api_endpoints.txt — API/AJAX endpoints discovered
• js_files.txt — JavaScript files (for secrets, API routes)
• crawl_results.json — full crawl data

Step 3: Build unified pentest endpoints file

Save to: /root/.openclaw/intel/pentest-endpoints-${taskId}.json

Format:

{
  "taskId": "...",
  "target": "...",
  "crawledAt": "...",
  "totalUrls": N,
  "source": "crawl4ai",
  "endpoints": [
    {"method": "GET", "path": "/page.php", "params": ["id", "cat"]},
    {"method": "POST", "path": "/login.php", "params": ["uname", "pass"]}
  ],
  "forms": [...],
  "apiEndpoints": [...],
  "jsFiles": [...]
}

Step 4: Log summary

Log: PHASE 0.5: Real crawl complete — X URLs, Y endpoints — saved to /root/.openclaw/intel/pentest-endpoints-${taskId}.json

Log highlights:

• Forms found: N (list action URLs)
• POST params discovered: [list]
• API endpoints: N
• JS files: N

Output Consumed By

Agent	File	Purpose
DRILL	forms.json + pentest-endpoints-{taskId}.json	SQLi: test every GET param AND POST form param
VIPER	forms.json + pentest-endpoints-{taskId}.json	XSS: test all input fields, HPP params
GATEWAY	api_endpoints.txt	API security testing
SCOUT	js_files.txt	JS analysis for secrets, routes
VAULT	pentest-endpoints-{taskId}.json	LFI: look for file= page= path= params
SENTRY	urls.txt + pentest-endpoints-{taskId}.json	Headers check on ALL discovered pages

Notes

• Fallback: if crawl4ai fails, use gau + katana + ffuf
• Always check for secured/newuser.php, phpinfo.php in crawl output
• If phpinfo.php found: flag for SENTRY (session.use_only_cookies check)
• If showimage.php or ?file= params found: flag for VAULT