AeonDave Hakrawler
Fast Go web crawler — discover URLs, JS files, forms, and endpoints.
Quick Start
go install github.com/hakluke/hakrawler@latest
# Crawl a domain
echo https://target.com | hakrawler
# Depth 3, include subdomains
echo https://target.com | hakrawler -d 3 -subs
# Output as JSON
echo https://target.com | hakrawler -json
# Pipe multiple domains
cat domains.txt | hakrawler -d 2
Core Flags
| Flag | Purpose |
|---|---|
-d N |
Depth (default: 1) |
-subs |
Include subdomains |
-u |
Unique URLs only |
-insecure |
Skip TLS verification |
-t N |
Threads |
-timeout N |
Timeout per request (s) |
-H "K:V" |
Custom header |
-json |
JSON output |
-scope REGEX |
Limit to URL pattern |
-plain |
Print plain text (no color) |
-proxy <url> |
HTTP/SOCKS5 proxy |
-cookie <str> |
Cookie string |
-dr |
Disable following redirects |
-w N |
Wait N ms between requests |
Common Workflows
Build URL inventory for fuzzing:
echo https://target.com | hakrawler -d 3 -u | tee urls.txt
# Feed to ffuf
ffuf -w urls.txt:URL -u URL -mc 200
Discover JS files:
echo https://target.com | hakrawler -d 2 | grep "\.js$"
Combine with httpx for live check:
cat domains.txt | hakrawler | httpx -silent -mc 200
Scope-limited crawl (stay in scope):
echo https://target.com | hakrawler -d 3 -scope ".*\.target\.com.*" -u
Extract API endpoints:
echo https://target.com | hakrawler -d 3 -u | \
grep -E "(/api/|/v[0-9]+/|\.json|\.xml)" | sort -u
Form action discovery:
echo https://target.com | hakrawler -json | \
jq -r 'select(.type=="form") | .source'
Multi-target via live hosts:
subfinder -d target.com -silent | \
httpx -silent | \
hakrawler -d 2 -u | sort -u > all_urls.txt
Resources
| File | When to load |
|---|---|
references/crawl-tips.md |
Scope filtering, JS analysis, pipeline patterns, JS secret extraction |