Spiderfoot

SpiderFoot

Automated OSINT platform — 200+ modules, correlates all data types around a target.

Quick Start

git clone https://github.com/smicallef/spiderfoot
cd spiderfoot
pip install -r requirements.txt

# CLI scan on domain
python3 sf.py -s target.com -t INTERNET_NAME -o json -q

# Web UI (interactive)
python3 sf.py -l 127.0.0.1:5001
# → http://127.0.0.1:5001

Target Types (-t)

CLI Usage

# Domain scan — all passive modules
python3 sf.py -s target.com -t INTERNET_NAME -o json -q > results.json

# Email scan
python3 sf.py -s ceo@target.com -t EMAILADDR -o json -q

# Specific modules only
python3 sf.py -s target.com -t INTERNET_NAME -m sfp_dnsresolve,sfp_sublist3r,sfp_crtsh -o json -q

# List all modules
python3 sf.py -M

# List modules by type
python3 sf.py -M | grep -i "passive"

Key CLI Flags

Module Categories

# List all categories
python3 sf.py -M | awk '{print $1}' | sort -u

# Useful module groups
sfp_dnsresolve      # DNS resolution
sfp_crtsh           # Certificate transparency
sfp_sublist3r       # Subdomain enumeration
sfp_shodan          # Shodan integration (API key)
sfp_virustotal      # VirusTotal (API key)
sfp_hunter          # Hunter.io emails (API key)
sfp_emailharvest    # Email harvesting
sfp_haveibeenpwned  # Breach check (API key)
sfp_linkedin        # LinkedIn profiles
sfp_twitter         # Twitter profiles
sfp_github          # GitHub user/org data
sfp_pastebin        # Pastebin leaks
sfp_darkweb         # Dark web mentions (Tor)
sfp_threatintel     # Threat intelligence feeds
sfp_whois           # WHOIS data

Web UI Workflow

python3 sf.py -l 127.0.0.1:5001

1. Browse to http://127.0.0.1:5001
2. New Scan → enter target + type
3. Select scan profile: Passive, Investigate, Footprint, All
4. Run → watch live graph of discovered entities
5. Browse results by data type: emails, IPs, subdomains, leaked credentials, social profiles
6. Export: JSON/CSV for downstream use

Scan Profiles

API Key Configuration

# spiderfoot.db stores settings after first run
# Or configure via web UI: Settings → API Keys

High-value API keys:

Hunter.io       → email harvest
Shodan          → port/service data
VirusTotal      → malware/domain intel
HaveIBeenPwned  → breach lookup
SecurityTrails  → DNS history
IntelX          → dark web / leaks

Parse JSON Output

import json

with open("results.json") as f:
    results = json.load(f)

# Group by data type
from collections import defaultdict
by_type = defaultdict(list)
for item in results:
    by_type[item["type"]].append(item["data"])

# Print emails found
for email in set(by_type.get("EMAILADDR", [])):
    print(email)

# Print subdomains
for sub in set(by_type.get("INTERNET_NAME", [])):
    print(sub)

Resources