gau (getallurls)
Passive URL mining from Wayback Machine, Common Crawl, URLScan.io, and AlienVault OTX — no requests to target.
Quick Start
# Fetch all known URLs for a domain
gau target.com
# With subdomains
gau --subs target.com
# Pipe into other tools
echo target.com | gau | httpx -silent
Core Flags
| Flag |
Description |
--subs |
Include subdomains |
--providers <list> |
Sources: wayback,commoncrawl,urlscan,otx |
--blacklist <ext> |
Exclude extensions (e.g., png,jpg,gif,css,woff) |
--fc <codes> |
Filter by status codes (from URLScan data) |
--ft <types> |
Filter by content type |
--mc <codes> |
Match status codes |
--threads <n> |
Concurrent threads |
--retries <n> |
Retry count |
--timeout <n> |
Timeout per request |
--verbose |
Verbose output |
--json |
JSON output (includes status, content-type, length) |
-o <file> |
Output file |
--config <file> |
Config file path |
Common Workflows
# All URLs for domain + subs, skip media files
gau --subs --blacklist png,jpg,gif,svg,ico,css,woff,woff2,ttf target.com
# Specific providers only
gau --providers wayback,urlscan target.com
# JSON output for analysis
gau --json target.com > urls.json
# Filter parameters (find injectable endpoints)
gau target.com | grep "?"
# Unique parameters across all URLs
gau target.com | grep "?" | \
sed 's/=.*/=/' | sort -u
# Old endpoints → check if still alive
gau target.com | httpx -silent -mc 200
# Mine API endpoints
gau target.com | grep -E "(/api/|/v[0-9]+/)" | sort -u
# Find backup/config files (historical)
gau target.com | grep -E "\.(bak|old|conf|cfg|env|log|sql|tar|zip|gz)$"
# Feed to nuclei for historical endpoint scan
gau target.com | httpx -silent | nuclei -tags exposure,misconfig -severity medium,high
Parameter Mining
# Extract all parameter names
gau target.com | grep "?" | \
python3 -c "
import sys, urllib.parse
params = set()
for url in sys.stdin:
parsed = urllib.parse.urlparse(url.strip())
for k in urllib.parse.parse_qs(parsed.query).keys():
params.add(k)
print('\n'.join(sorted(params)))
" > params.txt
# Feed to ffuf for parameter fuzzing
ffuf -u "https://target.com/page?FUZZ=test" -w params.txt -ac
Config File (~/.gau.toml)
providers = ["wayback", "commoncrawl", "urlscan", "otx"]
threads = 1
verbose = false
retries = 3
timeout = 45
blacklist = ["png", "jpg", "gif", "svg", "ico", "css", "woff", "woff2"]
Resources
| File |
When to load |
references/sources.md |
Provider details, URL parsing patterns, combination with other tools |
AeonDave