hakrawler

name	hakrawler
description	Auth/lab ref: Fast Go web crawler for discovering URLs, endpoints, and JavaScript files. For crawling web applications to build a URL inventory before fuzzing or during OSINT on web infrastructure.
license	MIT
compatibility	Go; Linux/macOS/Windows.
metadata	{"author":"AeonDave","version":"1.1"}

Hakrawler

Fast Go web crawler — discover URLs, JS files, forms, and endpoints.

Quick Start

go install github.com/hakluke/hakrawler@latest

# Crawl a domain
echo https://target.com | hakrawler

# Depth 3, include subdomains
echo https://target.com | hakrawler -d 3 -subs

# Output as JSON
echo https://target.com | hakrawler -json

# Pipe multiple domains
cat domains.txt | hakrawler -d 2

Core Flags

Flag	Purpose
-d N	Depth (default: 1)
-subs	Include subdomains
-u	Unique URLs only
-insecure	Skip TLS verification
-t N	Threads
-timeout N	Timeout per request (s)
-H "K:V"	Custom header
-json	JSON output
-scope REGEX	Limit to URL pattern
-plain	Print plain text (no color)
-proxy <url>	HTTP/SOCKS5 proxy
-cookie <str>	Cookie string
-dr	Disable following redirects
-w N	Wait N ms between requests

Common Workflows

Build URL inventory for fuzzing:

echo https://target.com | hakrawler -d 3 -u | tee urls.txt
# Feed to ffuf
ffuf -w urls.txt:URL -u URL -mc 200

Discover JS files:

echo https://target.com | hakrawler -d 2 | grep "\.js$"

Combine with httpx for live check:

cat domains.txt | hakrawler | httpx -silent -mc 200

Scope-limited crawl (stay in scope):

echo https://target.com | hakrawler -d 3 -scope ".*\.target\.com.*" -u

Extract API endpoints:

echo https://target.com | hakrawler -d 3 -u | \
  grep -E "(/api/|/v[0-9]+/|\.json|\.xml)" | sort -u

Form action discovery:

echo https://target.com | hakrawler -json | \
  jq -r 'select(.type=="form") | .source'

Multi-target via live hosts:

subfinder -d target.com -silent | \
  httpx -silent | \
  hakrawler -d 2 -u | sort -u > all_urls.txt

Resources

File	When to load
references/crawl-tips.md	Scope filtering, JS analysis, pipeline patterns, JS secret extraction