Collecting Open-Source Intelligence
When to Use
Use this skill when:
- Investigating external infrastructure associated with a phishing campaign targeting your organization
- Enriching threat actor profiles with publicly observable indicators (WHOIS, ASN data, SSL certificates)
- Conducting authorized attack surface discovery to understand your organization's external exposure
Do not use this skill for active scanning against targets without explicit written authorization — OSINT collection must remain passive (no packets sent to target systems) unless scope permits active recon.
Prerequisites
- Maltego CE or commercial license for graph-based link analysis
- Shodan API key (https://shodan.io) for internet-wide device/service discovery
- OSINT Framework familiarity (https://osintframework.com) for tool selection
- SpiderFoot HX or open-source SpiderFoot for automated OSINT correlation
Workflow
Step 1: Define Collection Requirements
Establish the intelligence requirement (IR) before collecting. Document:
- Target: threat actor group, malicious domain, IP range, or organization
- Priority Intelligence Requirements (PIRs): What specific questions need answering?
- Legal authority: Passive OSINT is legal; active probing requires authorization
- Data handling: TLP classification for collected intelligence
Step 2: Passive DNS and WHOIS Investigation
# Passive DNS via SecurityTrails API
curl "https://api.securitytrails.com/v1/domain/evil-domain.com/dns/a" \
-H "apikey: YOUR_KEY"
# WHOIS history via ARIN / RIPE
whois -h whois.arin.net evil-domain.com
# Certificate transparency logs (no API key required)
curl "https://crt.sh/?q=%.evil-domain.com&output=json" | jq '.[].name_value'
Certificate transparency logs reveal all subdomains for a target domain, often exposing staging, VPN, or internal infrastructure inadvertently made public.
Step 3: Shodan Infrastructure Mapping
import shodan
api = shodan.Shodan("YOUR_SHODAN_API_KEY")
# Search for specific C2 framework signatures (Cobalt Strike beacon)
results = api.search('product:"Cobalt Strike" port:443')
for r in results['matches']:
print(r['ip_str'], r['port'], r['org'], r.get('ssl', {}).get('cert', {}).get('subject', ''))
# Find infrastructure associated with a known threat actor's ASN
results = api.search('asn:AS12345 http.title:"Redirector"')
Correlate Shodan results with passive DNS to build infrastructure clusters.
Step 4: Maltego Graph Analysis
In Maltego, use these built-in transforms for threat actor infrastructure mapping:
- Start with a known malicious domain (Entity: Domain)
- Run "To IP Address [DNS]" → identifies hosting IPs
- Run "To Shared Hosting" → identifies co-hosted domains (potentially same threat actor)
- Run "To DNS Name [Reverse DNS]" → identifies PTR records
- Run "To Whois" → identifies registrant email/organization
- Pivot on registrant email → "To Domains [Registrant Email]" → expands to all domains registered with same email
Maltego Maltego Cyber threat intelligence transforms (VirusTotal, Shodan, PassiveTotal, URLScan) extend graph coverage.
Step 5: Dark Web and Paste Site Monitoring
Use SpiderFoot HX or manual searches for:
- Paste sites (Pastebin, Ghostbin): search for leaked credentials, IOCs, malware configs
- Dark web forums: via Tor browser with appropriate operational security
- GitHub/GitLab: search for exposed credentials or organization-specific strings
# SpiderFoot CLI for automated OSINT
python sf.py -s evil-domain.com -m sfp_shodan,sfp_virustotal,sfp_passivetotal \
-o TF -R result.json
Key Concepts
| Term |
Definition |
| Passive OSINT |
Intelligence collection that does not send any packets to target systems — uses public databases, search engines, cached data |
| PIR |
Priority Intelligence Requirement — specific question the intelligence collection must answer, preventing unfocused data gathering |
| Certificate Transparency |
Public log of all SSL/TLS certificates issued by CAs, enabling discovery of subdomains via crt.sh |
| Pivoting |
Using one data point (IP, email, registrant name) to discover related infrastructure or accounts |
| ASN |
Autonomous System Number — block of IP addresses under a single routing policy; useful for clustering threat actor infrastructure |
| Co-hosted Domains |
Multiple domains resolving to the same IP, potentially indicating shared attacker infrastructure |
Tools & Systems
- Maltego: Graph-based link analysis platform with 50+ data source transforms for IP, domain, email, and social media analysis
- Shodan: Internet-wide scanner database with 1B+ indexed devices; supports banner, port, SSL certificate, and vulnerability searches
- SpiderFoot: Automated OSINT tool with 200+ modules covering DNS, WHOIS, dark web, breach data, and social media
- Recon-ng: Python-based OSINT framework with modular design for domain, email, and social media reconnaissance
- crt.sh: Free certificate transparency search engine for subdomain and certificate discovery
- OSINT Framework (osintframework.com): Curated directory of OSINT tools organized by intelligence category
Common Pitfalls
- Leaving digital footprints: Visiting a threat actor's website or Shodan-queried IP can alert the adversary. Use Tor or VPN with a dedicated OSINT VM.
- Confirmation bias in graph analysis: Maltego graphs can create false connections. Verify each pivot independently before treating as confirmed.
- Outdated data: WHOIS privacy services and bulletproof hosting rotate frequently. Always check data timestamps — 6-month-old passive DNS may no longer be valid.
- Attribution overconfidence: Infrastructure overlap does not guarantee same threat actor. False flag operations deliberately share indicators across groups.
- Legal boundaries: Some OSINT tools perform active scans (port scanning, banner grabbing). Confirm tool behavior before use against external targets without authorization.
1---2name: collecting-open-source-intelligence3description: Collects and synthesizes open-source intelligence (OSINT) about threat actors, malicious infrastructure, and attack campaigns using publicly available data sources, passive reconnaissance tools, and dark web monitoring. Use when investigating external threat actor infrastructure, performing pre-engagement reconnaissance for authorized red team assessments, or enriching CTI reports with publicly available adversary context. Activates for requests involving Maltego, Shodan, OSINT framework, SpiderFoot, or infrastructure reconnaissance.4license: Apache-2.05---6# Collecting Open-Source Intelligence
7
8## When to Use
9
10Use this skill when:
11- Investigating external infrastructure associated with a phishing campaign targeting your organization
12- Enriching threat actor profiles with publicly observable indicators (WHOIS, ASN data, SSL certificates)
13- Conducting authorized attack surface discovery to understand your organization's external exposure
14
15**Do not use** this skill for active scanning against targets without explicit written authorization — OSINT collection must remain passive (no packets sent to target systems) unless scope permits active recon.
16
17## Prerequisites
18
19- Maltego CE or commercial license for graph-based link analysis
20- Shodan API key (https://shodan.io) for internet-wide device/service discovery
21- OSINT Framework familiarity (https://osintframework.com) for tool selection
22- SpiderFoot HX or open-source SpiderFoot for automated OSINT correlation
23
24## Workflow
25
26### Step 1: Define Collection Requirements
27
28Establish the intelligence requirement (IR) before collecting. Document:
29- Target: threat actor group, malicious domain, IP range, or organization
30- Priority Intelligence Requirements (PIRs): What specific questions need answering?
31- Legal authority: Passive OSINT is legal; active probing requires authorization
32- Data handling: TLP classification for collected intelligence
33
34### Step 2: Passive DNS and WHOIS Investigation
35
36```bash
37# Passive DNS via SecurityTrails API
38curl "https://api.securitytrails.com/v1/domain/evil-domain.com/dns/a" \
39 -H "apikey: YOUR_KEY"
40
41# WHOIS history via ARIN / RIPE
42whois -h whois.arin.net evil-domain.com
43
44# Certificate transparency logs (no API key required)
45curl "https://crt.sh/?q=%.evil-domain.com&output=json" | jq '.[].name_value'
46```
47
48Certificate transparency logs reveal all subdomains for a target domain, often exposing staging, VPN, or internal infrastructure inadvertently made public.
49
50### Step 3: Shodan Infrastructure Mapping
51
52```python
53import shodan
54
55api = shodan.Shodan("YOUR_SHODAN_API_KEY")
56
57# Search for specific C2 framework signatures (Cobalt Strike beacon)
58results = api.search('product:"Cobalt Strike" port:443')
59for r in results['matches']:
60 print(r['ip_str'], r['port'], r['org'], r.get('ssl', {}).get('cert', {}).get('subject', ''))
61
62# Find infrastructure associated with a known threat actor's ASN
63results = api.search('asn:AS12345 http.title:"Redirector"')
64```
65
66Correlate Shodan results with passive DNS to build infrastructure clusters.
67
68### Step 4: Maltego Graph Analysis
69
70In Maltego, use these built-in transforms for threat actor infrastructure mapping:
711. Start with a known malicious domain (Entity: Domain)
722. Run "To IP Address [DNS]" → identifies hosting IPs
733. Run "To Shared Hosting" → identifies co-hosted domains (potentially same threat actor)
744. Run "To DNS Name [Reverse DNS]" → identifies PTR records
755. Run "To Whois" → identifies registrant email/organization
766. Pivot on registrant email → "To Domains [Registrant Email]" → expands to all domains registered with same email
77
78Maltego Maltego Cyber threat intelligence transforms (VirusTotal, Shodan, PassiveTotal, URLScan) extend graph coverage.
79
80### Step 5: Dark Web and Paste Site Monitoring
81
82Use SpiderFoot HX or manual searches for:
83- Paste sites (Pastebin, Ghostbin): search for leaked credentials, IOCs, malware configs
84- Dark web forums: via Tor browser with appropriate operational security
85- GitHub/GitLab: search for exposed credentials or organization-specific strings
86
87```bash
88# SpiderFoot CLI for automated OSINT
89python sf.py -s evil-domain.com -m sfp_shodan,sfp_virustotal,sfp_passivetotal \
90 -o TF -R result.json
91```
92
93## Key Concepts
94
95| Term | Definition |
96|------|-----------|
97| **Passive OSINT** | Intelligence collection that does not send any packets to target systems — uses public databases, search engines, cached data |
98| **PIR** | Priority Intelligence Requirement — specific question the intelligence collection must answer, preventing unfocused data gathering |
99| **Certificate Transparency** | Public log of all SSL/TLS certificates issued by CAs, enabling discovery of subdomains via crt.sh |
100| **Pivoting** | Using one data point (IP, email, registrant name) to discover related infrastructure or accounts |
101| **ASN** | Autonomous System Number — block of IP addresses under a single routing policy; useful for clustering threat actor infrastructure |
102| **Co-hosted Domains** | Multiple domains resolving to the same IP, potentially indicating shared attacker infrastructure |
103
104## Tools & Systems
105
106- **Maltego**: Graph-based link analysis platform with 50+ data source transforms for IP, domain, email, and social media analysis
107- **Shodan**: Internet-wide scanner database with 1B+ indexed devices; supports banner, port, SSL certificate, and vulnerability searches
108- **SpiderFoot**: Automated OSINT tool with 200+ modules covering DNS, WHOIS, dark web, breach data, and social media
109- **Recon-ng**: Python-based OSINT framework with modular design for domain, email, and social media reconnaissance
110- **crt.sh**: Free certificate transparency search engine for subdomain and certificate discovery
111- **OSINT Framework (osintframework.com)**: Curated directory of OSINT tools organized by intelligence category
112
113## Common Pitfalls
114
115- **Leaving digital footprints**: Visiting a threat actor's website or Shodan-queried IP can alert the adversary. Use Tor or VPN with a dedicated OSINT VM.
116- **Confirmation bias in graph analysis**: Maltego graphs can create false connections. Verify each pivot independently before treating as confirmed.
117- **Outdated data**: WHOIS privacy services and bulletproof hosting rotate frequently. Always check data timestamps — 6-month-old passive DNS may no longer be valid.
118- **Attribution overconfidence**: Infrastructure overlap does not guarantee same threat actor. False flag operations deliberately share indicators across groups.
119- **Legal boundaries**: Some OSINT tools perform active scans (port scanning, banner grabbing). Confirm tool behavior before use against external targets without authorization.
theheavenlyd3mon