# Crawl

> Deep web crawling using hakrawler and gospider for subdomain discovery, endpoint extraction, and JavaScript analysis. Use this skill when users request to crawl URLs, discover subdomains, extract endpoints, map web applications, or perform reconnaissance tasks. All outputs (categorized results, bash logs, summaries) are saved to timestamped folders with format crawl_{DOMAIN}_{timestamp}.

- Skill: `securitytalent/crawl` (Agent Skill, multi-file: 4 files)
- Install (CLI): `npx skillmds@latest add securitytalent/crawl`
- Raw SKILL.md: https://api.skillmd.com/api/skills/securitytalent/crawl/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Product & Planning
- Author: SecurityTalent (https://skillmd.com/u/securitytalent)
- Updated: 2026-09-22
- Page: https://skillmd.com/skills/securitytalent/crawl

---


# Web Crawling Skill

## Overview

This skill enables deep web crawling using hakrawler and gospider to discover subdomains, extract endpoints, analyze JavaScript files, and map web application attack surfaces. The skill provides a unified Python wrapper that combines both tools for comprehensive URL discovery and categorization. All bash command outputs (stdout/stderr) are automatically logged for debugging and analysis.

## When to Use This Skill

Use this skill when users request:
- Crawling or spidering a URL or domain
- Subdomain discovery and enumeration
- Endpoint and API discovery
- JavaScript file extraction and analysis
- Web application reconnaissance or mapping
- Bug bounty scoping and enumeration
- Attack surface discovery

**Trigger keywords:** crawl, spider, enumerate, discover subdomains, find endpoints, map application, recon, reconnaissance

## Quick Start

The primary tool is `scripts/crawl.py`, which provides a unified interface to both hakrawler and gospider:

```bash
# Basic crawl - creates crawl_example.com_TIMESTAMP/
python scripts/crawl.py https://example.com

# Deep crawl with subdomain discovery
python scripts/crawl.py https://example.com --depth 3 --subdomains

# Crawl with JS analysis
python scripts/crawl.py https://example.com --js-analysis

# Use only one tool
python scripts/crawl.py https://example.com --tool hakrawler

# Custom output directory and timeout
python scripts/crawl.py https://example.com --output-dir ./results --timeout 600
```

**Note:** The script automatically creates timestamped output directories following the pattern `crawl_{DOMAIN}_{timestamp}/` unless a custom output directory is specified.

## Tool Installation

If the crawling tools are not installed, use the installation helper:

```bash
bash scripts/install_tools.sh
```

Or install manually:
```bash
go install github.com/hakluke/hakrawler@latest
go install github.com/jaeles-project/gospider@latest
export PATH=$PATH:~/go/bin
```

Verify with: `python scripts/crawl.py --install`

## Workflow

### 1. Initial Setup

Before crawling, ensure tools are installed:

```bash
# Check if tools are installed
hakrawler -h && gospider -h

# If not, run the installer
bash scripts/install_tools.sh
```

### 2. Execute Crawl

Run the crawl script with appropriate options based on user requirements:

```python
# For subdomain discovery
python scripts/crawl.py https://example.com --subdomains --depth 2

# For comprehensive endpoint discovery
python scripts/crawl.py https://example.com --depth 3 --js-analysis

# For quick reconnaissance
python scripts/crawl.py https://example.com --tool hakrawler --depth 1
```

### 3. Process Results

The script automatically categorizes and saves results to an organized directory structure:

**Output structure:**
```
crawl_{DOMAIN}_{timestamp}/
├── results/              # Categorized URL lists
│   ├── urls.txt         # All discovered URLs
│   ├── subdomains.txt   # Unique subdomains
│   ├── js_files.txt     # JavaScript files
│   ├── api_endpoints.txt # API endpoints
│   ├── parameters.txt   # URL parameters
│   └── extensions.json  # URLs grouped by file extension
├── logs/                # Bash command logs (stdout/stderr)
│   ├── hakrawler.log   # hakrawler execution log
│   └── gospider.log    # gospider execution log
├── summary.json         # Comprehensive JSON summary
└── REPORT.md           # Human-readable markdown report
```

### 4. Analyze Results

After crawling, analyze the results based on the user's goals:

```bash
# Read summary
cat crawl_example.com_*/summary.json

# Review markdown report
cat crawl_example.com_*/REPORT.md

# Review subdomains
cat crawl_example.com_*/results/subdomains.txt

# Check for interesting endpoints
grep -i "admin\|api\|login\|upload" crawl_example.com_*/results/urls.txt

# Find JavaScript files for further analysis
cat crawl_example.com_*/results/js_files.txt

# Check bash command logs if needed
cat crawl_example.com_*/logs/hakrawler.log
cat crawl_example.com_*/logs/gospider.log
```

## Use Cases

### Subdomain Discovery

When users want to discover subdomains:

```bash
python scripts/crawl.py https://example.com --subdomains --depth 2
```

Then read the `results/subdomains.txt` file from the output directory to provide the list of discovered subdomains, or review the REPORT.md for a formatted overview.

### Endpoint Extraction

When users need to extract all endpoints:

```bash
python scripts/crawl.py https://example.com --depth 3 --js-analysis
```

The script will identify and categorize:
- API endpoints (paths containing /api/, /v1/, /graphql, etc.)
- Regular web pages
- Static resources
- Parameters for further testing

### JavaScript Analysis

When users want to find JavaScript files for secret extraction or endpoint discovery:

```bash
python scripts/crawl.py https://example.com --js-analysis
```

The `results/js_files.txt` output can be passed to other tools like:
- jsluice (use the jsluice skill for deep JS analysis)
- SecretFinder
- LinkFinder

Example integration with jsluice skill:
```bash
# After crawling, feed JS files to jsluice
python3 ~/.claude/skills/jsluice/scripts/batch_analyze.py example.com --urls-file crawl_example.com_*/results/js_files.txt
```

### Bug Bounty Reconnaissance

For comprehensive bug bounty reconnaissance:

```bash
# Initial deep crawl with all options
python scripts/crawl.py https://target.com --subdomains --js-analysis --depth 3 --timeout 600
```

Then analyze the results to identify:
1. Interesting subdomains for further testing
2. API endpoints for authentication/authorization testing
3. JavaScript files for secret extraction
4. Parameters for injection testing

## Tool Comparison

### hakrawler
- **Strengths:** Fast, simple, good for quick enumeration
- **Best for:** Initial reconnaissance, quick scans
- **Limitations:** Less comprehensive than gospider

### gospider
- **Strengths:** More thorough, JSON output, better JS analysis
- **Best for:** Deep crawling, comprehensive mapping
- **Limitations:** Slower than hakrawler

**Default behavior:** Use both tools (`--tool both`) for comprehensive coverage.

## Advanced Usage

### Integration with Other Skills

This skill integrates well with other reconnaissance skills:

1. **After crawling, use jsluice skill** for JavaScript endpoint extraction:
   ```bash
   # Feed JS files to jsluice for deep analysis
   python3 ~/.claude/skills/jsluice/scripts/batch_analyze.py example.com \
     --urls-file crawl_example.com_*/results/js_files.txt
   ```

2. **Feed subdomains to subdomain takeover tools**
   ```bash
   cat crawl_example.com_*/results/subdomains.txt | subdomain-takeover-tool
   ```

3. **Pass API endpoints to authentication testing tools**
   ```bash
   cat crawl_example.com_*/results/api_endpoints.txt | api-testing-tool
   ```

4. **Use parameters list for injection testing**
   ```bash
   cat crawl_example.com_*/results/parameters.txt
   ```

### Custom Filtering

After crawling, apply custom filtering based on user needs:

```bash
# Find specific file types
grep "\.pdf$\|\.xlsx$\|\.docx$" crawl_example.com_*/results/urls.txt

# Find admin panels
grep -i "admin\|dashboard\|panel\|management" crawl_example.com_*/results/urls.txt

# Find upload functionality
grep -i "upload\|file" crawl_example.com_*/results/urls.txt

# Filter by extension using the extensions.json file
jq '.js' crawl_example.com_*/results/extensions.json
```

## Troubleshooting

### Tools Not Found

If crawl.py reports missing tools:
1. Run `scripts/install_tools.sh`
2. Verify `~/go/bin` is in PATH: `echo $PATH`
3. Add to PATH if needed: `export PATH=$PATH:~/go/bin`

### Timeout Issues

For large targets that timeout:
- Increase timeout: `--timeout 900`
- Reduce depth: `--depth 1`
- Use one tool: `--tool hakrawler`
- Crawl subdomains separately

### Permission Denied

Make scripts executable:
```bash
chmod +x scripts/crawl.py scripts/install_tools.sh
```

## Best Practices

1. **Start shallow, go deep:** Begin with depth 1-2, then increase if needed
2. **Use both tools:** Default `--tool both` provides best coverage
3. **Enable subdomain discovery** for comprehensive reconnaissance
4. **Review reports first:** Check `REPORT.md` for human-readable overview, then `summary.json` for detailed statistics
5. **Categorize findings:** Use the categorized output files in `results/` to prioritize testing
6. **Check bash logs:** If tools fail or produce unexpected results, review `logs/hakrawler.log` and `logs/gospider.log` for stdout/stderr
7. **Respect scope:** Only crawl targets within authorized scope
8. **Monitor resources:** Large crawls can be resource-intensive, use timeouts appropriately
9. **Combine with other tools:** Chain crawl results into specialized tools for deeper analysis (especially jsluice for JS files)
10. **Archive outputs:** Each crawl creates a timestamped directory - keep these for historical tracking and comparison

## Resources

### scripts/

- `crawl.py` - Main Python wrapper for hakrawler and gospider with:
  - Unified interface for both tools
  - Automatic bash command logging (stdout/stderr)
  - Result categorization and organization
  - Timestamped output directories
  - JSON summary and markdown report generation
- `install_tools.sh` - Bash script to install hakrawler and gospider via Go

### references/

- `tool_comparison.md` - Detailed comparison of hakrawler vs gospider features, performance, and use cases

## Output Directory Reference

Every crawl creates a structured output directory:

```
crawl_{DOMAIN}_{timestamp}/
├── results/              # Categorized results
│   ├── urls.txt         # All URLs found
│   ├── subdomains.txt   # Unique subdomains
│   ├── js_files.txt     # JavaScript files
│   ├── api_endpoints.txt # Detected API endpoints
│   ├── parameters.txt   # URL parameters
│   └── extensions.json  # URLs grouped by file extension
├── logs/                # Bash execution logs
│   ├── hakrawler.log   # Command, stdout, stderr, return code
│   └── gospider.log    # Command, stdout, stderr, return code
├── summary.json         # Complete crawl statistics and metadata
└── REPORT.md           # Human-readable markdown report
```

**Key features:**
- All bash commands are logged with full stdout/stderr for debugging
- Results are categorized for easy filtering and analysis
- Summary includes tool execution logs for transparency
- Markdown reports provide immediate insights
- Timestamped directories enable historical tracking