Crawl

Web Crawling Skill

Overview

This skill enables deep web crawling using hakrawler and gospider to discover subdomains, extract endpoints, analyze JavaScript files, and map web application attack surfaces. The skill provides a unified Python wrapper that combines both tools for comprehensive URL discovery and categorization. All bash command outputs (stdout/stderr) are automatically logged for debugging and analysis.

When to Use This Skill

Use this skill when users request:

• Crawling or spidering a URL or domain
• Subdomain discovery and enumeration
• Endpoint and API discovery
• JavaScript file extraction and analysis
• Web application reconnaissance or mapping
• Bug bounty scoping and enumeration
• Attack surface discovery

Trigger keywords: crawl, spider, enumerate, discover subdomains, find endpoints, map application, recon, reconnaissance

Quick Start

The primary tool is scripts/crawl.py, which provides a unified interface to both hakrawler and gospider:

# Basic crawl - creates crawl_example.com_TIMESTAMP/
python scripts/crawl.py https://example.com

# Deep crawl with subdomain discovery
python scripts/crawl.py https://example.com --depth 3 --subdomains

# Crawl with JS analysis
python scripts/crawl.py https://example.com --js-analysis

# Use only one tool
python scripts/crawl.py https://example.com --tool hakrawler

# Custom output directory and timeout
python scripts/crawl.py https://example.com --output-dir ./results --timeout 600

Note: The script automatically creates timestamped output directories following the pattern crawl_{DOMAIN}_{timestamp}/ unless a custom output directory is specified.

Tool Installation

If the crawling tools are not installed, use the installation helper:

bash scripts/install_tools.sh

Or install manually:

go install github.com/hakluke/hakrawler@latest
go install github.com/jaeles-project/gospider@latest
export PATH=$PATH:~/go/bin

Verify with: python scripts/crawl.py --install

Workflow

1. Initial Setup

Before crawling, ensure tools are installed:

# Check if tools are installed
hakrawler -h && gospider -h

# If not, run the installer
bash scripts/install_tools.sh

2. Execute Crawl

Run the crawl script with appropriate options based on user requirements:

# For subdomain discovery
python scripts/crawl.py https://example.com --subdomains --depth 2

# For comprehensive endpoint discovery
python scripts/crawl.py https://example.com --depth 3 --js-analysis

# For quick reconnaissance
python scripts/crawl.py https://example.com --tool hakrawler --depth 1

3. Process Results

The script automatically categorizes and saves results to an organized directory structure:

Output structure:

crawl_{DOMAIN}_{timestamp}/
├── results/              # Categorized URL lists
│   ├── urls.txt         # All discovered URLs
│   ├── subdomains.txt   # Unique subdomains
│   ├── js_files.txt     # JavaScript files
│   ├── api_endpoints.txt # API endpoints
│   ├── parameters.txt   # URL parameters
│   └── extensions.json  # URLs grouped by file extension
├── logs/                # Bash command logs (stdout/stderr)
│   ├── hakrawler.log   # hakrawler execution log
│   └── gospider.log    # gospider execution log
├── summary.json         # Comprehensive JSON summary
└── REPORT.md           # Human-readable markdown report

4. Analyze Results

After crawling, analyze the results based on the user's goals:

# Read summary
cat crawl_example.com_*/summary.json

# Review markdown report
cat crawl_example.com_*/REPORT.md

# Review subdomains
cat crawl_example.com_*/results/subdomains.txt

# Check for interesting endpoints
grep -i "admin\|api\|login\|upload" crawl_example.com_*/results/urls.txt

# Find JavaScript files for further analysis
cat crawl_example.com_*/results/js_files.txt

# Check bash command logs if needed
cat crawl_example.com_*/logs/hakrawler.log
cat crawl_example.com_*/logs/gospider.log

Use Cases

Subdomain Discovery

When users want to discover subdomains:

python scripts/crawl.py https://example.com --subdomains --depth 2

Then read the results/subdomains.txt file from the output directory to provide the list of discovered subdomains, or review the REPORT.md for a formatted overview.

Endpoint Extraction

When users need to extract all endpoints:

python scripts/crawl.py https://example.com --depth 3 --js-analysis

The script will identify and categorize:

• API endpoints (paths containing /api/, /v1/, /graphql, etc.)
• Regular web pages
• Static resources
• Parameters for further testing

JavaScript Analysis

When users want to find JavaScript files for secret extraction or endpoint discovery:

python scripts/crawl.py https://example.com --js-analysis

The results/js_files.txt output can be passed to other tools like:

• jsluice (use the jsluice skill for deep JS analysis)
• SecretFinder
• LinkFinder

Example integration with jsluice skill:

# After crawling, feed JS files to jsluice
python3 ~/.claude/skills/jsluice/scripts/batch_analyze.py example.com --urls-file crawl_example.com_*/results/js_files.txt

Bug Bounty Reconnaissance

For comprehensive bug bounty reconnaissance:

# Initial deep crawl with all options
python scripts/crawl.py https://target.com --subdomains --js-analysis --depth 3 --timeout 600

Then analyze the results to identify:

1. Interesting subdomains for further testing
2. API endpoints for authentication/authorization testing
3. JavaScript files for secret extraction
4. Parameters for injection testing

Tool Comparison

hakrawler

• Strengths: Fast, simple, good for quick enumeration
• Best for: Initial reconnaissance, quick scans
• Limitations: Less comprehensive than gospider

gospider

• Strengths: More thorough, JSON output, better JS analysis
• Best for: Deep crawling, comprehensive mapping
• Limitations: Slower than hakrawler

Default behavior: Use both tools (--tool both) for comprehensive coverage.

Advanced Usage

Integration with Other Skills

This skill integrates well with other reconnaissance skills:

1. After crawling, use jsluice skill for JavaScript endpoint extraction:

   # Feed JS files to jsluice for deep analysis
   python3 ~/.claude/skills/jsluice/scripts/batch_analyze.py example.com \
     --urls-file crawl_example.com_*/results/js_files.txt
   

2. Feed subdomains to subdomain takeover tools

   cat crawl_example.com_*/results/subdomains.txt | subdomain-takeover-tool
   

3. Pass API endpoints to authentication testing tools

   cat crawl_example.com_*/results/api_endpoints.txt | api-testing-tool
   

4. Use parameters list for injection testing

   cat crawl_example.com_*/results/parameters.txt
   

Custom Filtering

After crawling, apply custom filtering based on user needs:

# Find specific file types
grep "\.pdf$\|\.xlsx$\|\.docx$" crawl_example.com_*/results/urls.txt

# Find admin panels
grep -i "admin\|dashboard\|panel\|management" crawl_example.com_*/results/urls.txt

# Find upload functionality
grep -i "upload\|file" crawl_example.com_*/results/urls.txt

# Filter by extension using the extensions.json file
jq '.js' crawl_example.com_*/results/extensions.json

Troubleshooting

Tools Not Found

If crawl.py reports missing tools:

1. Run scripts/install_tools.sh
2. Verify ~/go/bin is in PATH: echo $PATH
3. Add to PATH if needed: export PATH=$PATH:~/go/bin

Timeout Issues

For large targets that timeout:

• Increase timeout: --timeout 900
• Reduce depth: --depth 1
• Use one tool: --tool hakrawler
• Crawl subdomains separately

Permission Denied

Make scripts executable:

chmod +x scripts/crawl.py scripts/install_tools.sh

Best Practices

1. Start shallow, go deep: Begin with depth 1-2, then increase if needed
2. Use both tools: Default --tool both provides best coverage
3. Enable subdomain discovery for comprehensive reconnaissance
4. Review reports first: Check REPORT.md for human-readable overview, then summary.json for detailed statistics
5. Categorize findings: Use the categorized output files in results/ to prioritize testing
6. Check bash logs: If tools fail or produce unexpected results, review logs/hakrawler.log and logs/gospider.log for stdout/stderr
7. Respect scope: Only crawl targets within authorized scope
8. Monitor resources: Large crawls can be resource-intensive, use timeouts appropriately
9. Combine with other tools: Chain crawl results into specialized tools for deeper analysis (especially jsluice for JS files)
10. Archive outputs: Each crawl creates a timestamped directory - keep these for historical tracking and comparison

Resources

scripts/

• crawl.py - Main Python wrapper for hakrawler and gospider with:
  • Unified interface for both tools
  • Automatic bash command logging (stdout/stderr)
  • Result categorization and organization
  • Timestamped output directories
  • JSON summary and markdown report generation
• install_tools.sh - Bash script to install hakrawler and gospider via Go

references/

• tool_comparison.md - Detailed comparison of hakrawler vs gospider features, performance, and use cases

Output Directory Reference

Every crawl creates a structured output directory:

crawl_{DOMAIN}_{timestamp}/
├── results/              # Categorized results
│   ├── urls.txt         # All URLs found
│   ├── subdomains.txt   # Unique subdomains
│   ├── js_files.txt     # JavaScript files
│   ├── api_endpoints.txt # Detected API endpoints
│   ├── parameters.txt   # URL parameters
│   └── extensions.json  # URLs grouped by file extension
├── logs/                # Bash execution logs
│   ├── hakrawler.log   # Command, stdout, stderr, return code
│   └── gospider.log    # Command, stdout, stderr, return code
├── summary.json         # Complete crawl statistics and metadata
└── REPORT.md           # Human-readable markdown report

Key features:

• All bash commands are logged with full stdout/stderr for debugging
• Results are categorized for easy filtering and analysis
• Summary includes tool execution logs for transparency
• Markdown reports provide immediate insights
• Timestamped directories enable historical tracking