Performing OSINT with SpiderFoot
Overview
SpiderFoot is an open-source OSINT automation tool with 200+ modules that integrates with data sources for threat intelligence and attack surface mapping. This skill uses the SpiderFoot REST API and CLI (sf.py/spiderfoot-cli) to create and manage scans, select modules by use case (footprint, investigate, passive), parse structured results for domains, IPs, email addresses, leaked credentials, and DNS records, and generate target intelligence profiles.
When to Use
- When conducting security assessments that involve performing osint with spiderfoot
- When following incident response procedures for related security events
- When performing scheduled security testing or auditing activities
- When validating security controls through hands-on testing
Detection Gaps & Validation
- Coverage = enabled modules + API keys: SpiderFoot's 200+ modules are mostly inert without keys - lacking VirusTotal, Shodan, and HaveIBeenPwned keys, the
investigate/footprint use cases silently skip the richest sources and return a thin profile that looks "clean." Audit which modules actually ran (and which errored on a missing key) before concluding a target has no exposure.
- Passive vs active tradeoff: the
passive use case avoids touching the target but misses live ports/banners; footprint/investigate actively probe and can tip off the target or trip a WAF. Choose the scan type deliberately - a passive-only scan is not evidence the attack surface is small.
- Stale and recycled data: OSINT sources lag reality - dead subdomains, expired breach credentials, and recycled combolists produce findings that no longer apply. Check first/last-seen dates.
- Name-collision false positives: email/name/username pivots match unrelated people and orgs; a "leaked credential" hit may belong to a different entity sharing the keyword. Confirm the identifier truly maps to the target (matching domain, corroborating record) before reporting.
- How to confirm: treat SpiderFoot output as leads - validate high-impact findings (an exposed service, a leaked credential) directly against the authoritative source before acting.
Prerequisites
- SpiderFoot 4.0+ installed or SpiderFoot HX cloud account
- Python 3.8+ with requests library
- SpiderFoot server running on default port 5001
- Optional: API keys for VirusTotal, Shodan, HaveIBeenPwned modules
Steps
- Connect to SpiderFoot REST API or use CLI interface
- Create a new scan with target specification (domain, IP, email, name)
- Select scan modules by use case (all, footprint, investigate, passive)
- Monitor scan progress via API polling
- Retrieve and parse scan results by data element type
- Extract key findings: subdomains, IPs, emails, leaked credentials
- Generate structured OSINT intelligence report
Expected Output
JSON report containing OSINT findings organized by data type (domains, IPs, emails, credentials, DNS records), module source attribution, and target profile summary with risk indicators.
1---2name: performing-osint-with-spiderfoot3description: Automate OSINT collection using SpiderFoot REST API and CLI for target profiling, module-based reconnaissance, and structured result analysis across 200+ data sources4license: Apache-2.05---67# Performing OSINT with SpiderFoot89## Overview1011SpiderFoot is an open-source OSINT automation tool with 200+ modules that integrates with data sources for threat intelligence and attack surface mapping. This skill uses the SpiderFoot REST API and CLI (sf.py/spiderfoot-cli) to create and manage scans, select modules by use case (footprint, investigate, passive), parse structured results for domains, IPs, email addresses, leaked credentials, and DNS records, and generate target intelligence profiles.121314## When to Use1516- When conducting security assessments that involve performing osint with spiderfoot17- When following incident response procedures for related security events18- When performing scheduled security testing or auditing activities19- When validating security controls through hands-on testing2021## Detection Gaps & Validation2223- **Coverage = enabled modules + API keys:** SpiderFoot's 200+ modules are mostly inert without keys - lacking VirusTotal, Shodan, and HaveIBeenPwned keys, the `investigate`/`footprint` use cases silently skip the richest sources and return a thin profile that looks "clean." Audit which modules actually ran (and which errored on a missing key) before concluding a target has no exposure.24- **Passive vs active tradeoff:** the `passive` use case avoids touching the target but misses live ports/banners; `footprint`/`investigate` actively probe and can tip off the target or trip a WAF. Choose the scan type deliberately - a passive-only scan is not evidence the attack surface is small.25- **Stale and recycled data:** OSINT sources lag reality - dead subdomains, expired breach credentials, and recycled combolists produce findings that no longer apply. Check first/last-seen dates.26- **Name-collision false positives:** email/name/username pivots match unrelated people and orgs; a "leaked credential" hit may belong to a different entity sharing the keyword. Confirm the identifier truly maps to the target (matching domain, corroborating record) before reporting.27- **How to confirm:** treat SpiderFoot output as leads - validate high-impact findings (an exposed service, a leaked credential) directly against the authoritative source before acting.2829## Prerequisites3031- SpiderFoot 4.0+ installed or SpiderFoot HX cloud account32- Python 3.8+ with requests library33- SpiderFoot server running on default port 500134- Optional: API keys for VirusTotal, Shodan, HaveIBeenPwned modules3536## Steps37381. Connect to SpiderFoot REST API or use CLI interface392. Create a new scan with target specification (domain, IP, email, name)403. Select scan modules by use case (all, footprint, investigate, passive)414. Monitor scan progress via API polling425. Retrieve and parse scan results by data element type436. Extract key findings: subdomains, IPs, emails, leaked credentials447. Generate structured OSINT intelligence report4546## Expected Output4748JSON report containing OSINT findings organized by data type (domains, IPs, emails, credentials, DNS records), module source attribution, and target profile summary with risk indicators.
xalgord