# Hakrawler

> Auth/lab ref: Fast Go web crawler for discovering URLs, endpoints, and JavaScript files. For crawling web applications to build a URL inventory before fuzzing or during OSINT on web infrastructure.

- Skill: `aeondave/hakrawler` (Agent Skill, multi-file: 2 files)
- Install (CLI): `npx skillmds@latest add aeondave/hakrawler`
- Raw SKILL.md: https://api.skillmd.com/api/skills/aeondave/hakrawler/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- License: MIT
- Author: AeonDave (https://skillmd.com/u/aeondave)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/aeondave/hakrawler

---


# Hakrawler

Fast Go web crawler — discover URLs, JS files, forms, and endpoints.

## Quick Start

```bash
go install github.com/hakluke/hakrawler@latest

# Crawl a domain
echo https://target.com | hakrawler

# Depth 3, include subdomains
echo https://target.com | hakrawler -d 3 -subs

# Output as JSON
echo https://target.com | hakrawler -json

# Pipe multiple domains
cat domains.txt | hakrawler -d 2
```

## Core Flags

| Flag | Purpose |
|------|---------|
| `-d N` | Depth (default: 1) |
| `-subs` | Include subdomains |
| `-u` | Unique URLs only |
| `-insecure` | Skip TLS verification |
| `-t N` | Threads |
| `-timeout N` | Timeout per request (s) |
| `-H "K:V"` | Custom header |
| `-json` | JSON output |
| `-scope REGEX` | Limit to URL pattern |
| `-plain` | Print plain text (no color) |
| `-proxy <url>` | HTTP/SOCKS5 proxy |
| `-cookie <str>` | Cookie string |
| `-dr` | Disable following redirects |
| `-w N` | Wait N ms between requests |

## Common Workflows

**Build URL inventory for fuzzing:**
```bash
echo https://target.com | hakrawler -d 3 -u | tee urls.txt
# Feed to ffuf
ffuf -w urls.txt:URL -u URL -mc 200
```

**Discover JS files:**
```bash
echo https://target.com | hakrawler -d 2 | grep "\.js$"
```

**Combine with httpx for live check:**
```bash
cat domains.txt | hakrawler | httpx -silent -mc 200
```

**Scope-limited crawl (stay in scope):**
```bash
echo https://target.com | hakrawler -d 3 -scope ".*\.target\.com.*" -u
```

**Extract API endpoints:**
```bash
echo https://target.com | hakrawler -d 3 -u | \
  grep -E "(/api/|/v[0-9]+/|\.json|\.xml)" | sort -u
```

**Form action discovery:**
```bash
echo https://target.com | hakrawler -json | \
  jq -r 'select(.type=="form") | .source'
```

**Multi-target via live hosts:**
```bash
subfinder -d target.com -silent | \
  httpx -silent | \
  hakrawler -d 2 -u | sort -u > all_urls.txt
```

## Resources

| File | When to load |
|------|--------------|
| `references/crawl-tips.md` | Scope filtering, JS analysis, pipeline patterns, JS secret extraction |