Collecting Open-Source Intelligence
When to Use
Use this skill when:
- Investigating external infrastructure associated with a phishing campaign targeting your organization
- Enriching threat actor profiles with publicly observable indicators (WHOIS, ASN data, SSL certificates)
- Conducting authorized attack surface discovery to understand your organization's external exposure
Do not use this skill for active scanning against targets without explicit written authorization — OSINT collection must remain passive (no packets sent to target systems) unless scope permits active recon.
Prerequisites
- Maltego CE or commercial license for graph-based link analysis
- Shodan API key (https://shodan.io) for internet-wide device/service discovery
- OSINT Framework familiarity (https://osintframework.com) for tool selection
- SpiderFoot HX or open-source SpiderFoot for automated OSINT correlation
Workflow
Step 1: Define Collection Requirements
Establish the intelligence requirement (IR) before collecting. Document:
- Target: threat actor group, malicious domain, IP range, or organization
- Priority Intelligence Requirements (PIRs): What specific questions need answering?
- Legal authority: Passive OSINT is legal; active probing requires authorization
- Data handling: TLP classification for collected intelligence
Step 2: Passive DNS and WHOIS Investigation
# Passive DNS via SecurityTrails API
curl "https://api.securitytrails.com/v1/domain/evil-domain.com/dns/a" \
-H "apikey: YOUR_KEY"
# WHOIS history via ARIN / RIPE
whois -h whois.arin.net evil-domain.com
# Certificate transparency logs (no API key required)
curl "https://crt.sh/?q=%.evil-domain.com&output=json" | jq '.[].name_value'
Certificate transparency logs reveal all subdomains for a target domain, often exposing staging, VPN, or internal infrastructure inadvertently made public.
Step 3: Shodan Infrastructure Mapping
import shodan
api = shodan.Shodan("YOUR_SHODAN_API_KEY")
# Search for specific C2 framework signatures (Cobalt Strike beacon)
results = api.search('product:"Cobalt Strike" port:443')
for r in results['matches']:
print(r['ip_str'], r['port'], r['org'], r.get('ssl', {}).get('cert', {}).get('subject', ''))
# Find infrastructure associated with a known threat actor's ASN
results = api.search('asn:AS12345 http.title:"Redirector"')
Correlate Shodan results with passive DNS to build infrastructure clusters.
Step 4: Maltego Graph Analysis
In Maltego, use these built-in transforms for threat actor infrastructure mapping:
- Start with a known malicious domain (Entity: Domain)
- Run "To IP Address [DNS]" → identifies hosting IPs
- Run "To Shared Hosting" → identifies co-hosted domains (potentially same threat actor)
- Run "To DNS Name [Reverse DNS]" → identifies PTR records
- Run "To Whois" → identifies registrant email/organization
- Pivot on registrant email → "To Domains [Registrant Email]" → expands to all domains registered with same email
Maltego Maltego Cyber threat intelligence transforms (VirusTotal, Shodan, PassiveTotal, URLScan) extend graph coverage.
Step 5: Dark Web and Paste Site Monitoring
Use SpiderFoot HX or manual searches for:
- Paste sites (Pastebin, Ghostbin): search for leaked credentials, IOCs, malware configs
- Dark web forums: via Tor browser with appropriate operational security
- GitHub/GitLab: search for exposed credentials or organization-specific strings
# SpiderFoot CLI for automated OSINT
python sf.py -s evil-domain.com -m sfp_shodan,sfp_virustotal,sfp_passivetotal \
-o TF -R result.json
Key Concepts
| Term |
Definition |
| Passive OSINT |
Intelligence collection that does not send any packets to target systems — uses public databases, search engines, cached data |
| PIR |
Priority Intelligence Requirement — specific question the intelligence collection must answer, preventing unfocused data gathering |
| Certificate Transparency |
Public log of all SSL/TLS certificates issued by CAs, enabling discovery of subdomains via crt.sh |
| Pivoting |
Using one data point (IP, email, registrant name) to discover related infrastructure or accounts |
| ASN |
Autonomous System Number — block of IP addresses under a single routing policy; useful for clustering threat actor infrastructure |
| Co-hosted Domains |
Multiple domains resolving to the same IP, potentially indicating shared attacker infrastructure |
Tools & Systems
- Maltego: Graph-based link analysis platform with 50+ data source transforms for IP, domain, email, and social media analysis
- Shodan: Internet-wide scanner database with 1B+ indexed devices; supports banner, port, SSL certificate, and vulnerability searches
- SpiderFoot: Automated OSINT tool with 200+ modules covering DNS, WHOIS, dark web, breach data, and social media
- Recon-ng: Python-based OSINT framework with modular design for domain, email, and social media reconnaissance
- crt.sh: Free certificate transparency search engine for subdomain and certificate discovery
- OSINT Framework (osintframework.com): Curated directory of OSINT tools organized by intelligence category
Common Pitfalls
- Leaving digital footprints: Visiting a threat actor's website or Shodan-queried IP can alert the adversary. Use Tor or VPN with a dedicated OSINT VM.
- Confirmation bias in graph analysis: Maltego graphs can create false connections. Verify each pivot independently before treating as confirmed.
- Outdated data: WHOIS privacy services and bulletproof hosting rotate frequently. Always check data timestamps — 6-month-old passive DNS may no longer be valid.
- Attribution overconfidence: Infrastructure overlap does not guarantee same threat actor. False flag operations deliberately share indicators across groups.
- Legal boundaries: Some OSINT tools perform active scans (port scanning, banner grabbing). Confirm tool behavior before use against external targets without authorization.
1---
2name: collecting-open-source-intelligence
3description: 'Collects and synthesizes open-source intelligence (OSINT) about threat actors, malicious infrastructure, and
4 attack campaigns using publicly available data sources, passive reconnaissance tools, and dark web monitoring. Use when
5 investigating external threat actor infrastructure, performing pre-engagement reconnaissance for authorized red team assessments,
6 or enriching CTI reports with publicly available adversary context. Activates for requests involving Maltego, Shodan, OSINT
7 framework, SpiderFoot, or infrastructure reconnaissance.
8
9 '
10domain: cybersecurity
11subdomain: threat-intelligence
12tags:
13- OSINT
14- Maltego
15- Shodan
16- Recon-ng
17- SpiderFoot
18- threat-intelligence
19- ATT&CK-T1591
20- NIST-CSF
21version: 1.0.0
22author: mahipal
23license: Apache-2.0
24nist_csf:
25- ID.RA-01
26- ID.RA-05
27- DE.CM-01
28- DE.AE-02
29---
30
31# Collecting Open-Source Intelligence
32
33## When to Use
34
35Use this skill when:
36- Investigating external infrastructure associated with a phishing campaign targeting your organization
37- Enriching threat actor profiles with publicly observable indicators (WHOIS, ASN data, SSL certificates)
38- Conducting authorized attack surface discovery to understand your organization's external exposure
39
40**Do not use** this skill for active scanning against targets without explicit written authorization — OSINT collection must remain passive (no packets sent to target systems) unless scope permits active recon.
41
42## Prerequisites
43
44- Maltego CE or commercial license for graph-based link analysis
45- Shodan API key (https://shodan.io) for internet-wide device/service discovery
46- OSINT Framework familiarity (https://osintframework.com) for tool selection
47- SpiderFoot HX or open-source SpiderFoot for automated OSINT correlation
48
49## Workflow
50
51### Step 1: Define Collection Requirements
52
53Establish the intelligence requirement (IR) before collecting. Document:
54- Target: threat actor group, malicious domain, IP range, or organization
55- Priority Intelligence Requirements (PIRs): What specific questions need answering?
56- Legal authority: Passive OSINT is legal; active probing requires authorization
57- Data handling: TLP classification for collected intelligence
58
59### Step 2: Passive DNS and WHOIS Investigation
60
61```bash
62# Passive DNS via SecurityTrails API
63curl "https://api.securitytrails.com/v1/domain/evil-domain.com/dns/a" \
64 -H "apikey: YOUR_KEY"
65
66# WHOIS history via ARIN / RIPE
67whois -h whois.arin.net evil-domain.com
68
69# Certificate transparency logs (no API key required)
70curl "https://crt.sh/?q=%.evil-domain.com&output=json" | jq '.[].name_value'
71```
72
73Certificate transparency logs reveal all subdomains for a target domain, often exposing staging, VPN, or internal infrastructure inadvertently made public.
74
75### Step 3: Shodan Infrastructure Mapping
76
77```python
78import shodan
79
80api = shodan.Shodan("YOUR_SHODAN_API_KEY")
81
82# Search for specific C2 framework signatures (Cobalt Strike beacon)
83results = api.search('product:"Cobalt Strike" port:443')
84for r in results['matches']:
85 print(r['ip_str'], r['port'], r['org'], r.get('ssl', {}).get('cert', {}).get('subject', ''))
86
87# Find infrastructure associated with a known threat actor's ASN
88results = api.search('asn:AS12345 http.title:"Redirector"')
89```
90
91Correlate Shodan results with passive DNS to build infrastructure clusters.
92
93### Step 4: Maltego Graph Analysis
94
95In Maltego, use these built-in transforms for threat actor infrastructure mapping:
961. Start with a known malicious domain (Entity: Domain)
972. Run "To IP Address [DNS]" → identifies hosting IPs
983. Run "To Shared Hosting" → identifies co-hosted domains (potentially same threat actor)
994. Run "To DNS Name [Reverse DNS]" → identifies PTR records
1005. Run "To Whois" → identifies registrant email/organization
1016. Pivot on registrant email → "To Domains [Registrant Email]" → expands to all domains registered with same email
102
103Maltego Maltego Cyber threat intelligence transforms (VirusTotal, Shodan, PassiveTotal, URLScan) extend graph coverage.
104
105### Step 5: Dark Web and Paste Site Monitoring
106
107Use SpiderFoot HX or manual searches for:
108- Paste sites (Pastebin, Ghostbin): search for leaked credentials, IOCs, malware configs
109- Dark web forums: via Tor browser with appropriate operational security
110- GitHub/GitLab: search for exposed credentials or organization-specific strings
111
112```bash
113# SpiderFoot CLI for automated OSINT
114python sf.py -s evil-domain.com -m sfp_shodan,sfp_virustotal,sfp_passivetotal \
115 -o TF -R result.json
116```
117
118## Key Concepts
119
120| Term | Definition |
121|------|-----------|
122| **Passive OSINT** | Intelligence collection that does not send any packets to target systems — uses public databases, search engines, cached data |
123| **PIR** | Priority Intelligence Requirement — specific question the intelligence collection must answer, preventing unfocused data gathering |
124| **Certificate Transparency** | Public log of all SSL/TLS certificates issued by CAs, enabling discovery of subdomains via crt.sh |
125| **Pivoting** | Using one data point (IP, email, registrant name) to discover related infrastructure or accounts |
126| **ASN** | Autonomous System Number — block of IP addresses under a single routing policy; useful for clustering threat actor infrastructure |
127| **Co-hosted Domains** | Multiple domains resolving to the same IP, potentially indicating shared attacker infrastructure |
128
129## Tools & Systems
130
131- **Maltego**: Graph-based link analysis platform with 50+ data source transforms for IP, domain, email, and social media analysis
132- **Shodan**: Internet-wide scanner database with 1B+ indexed devices; supports banner, port, SSL certificate, and vulnerability searches
133- **SpiderFoot**: Automated OSINT tool with 200+ modules covering DNS, WHOIS, dark web, breach data, and social media
134- **Recon-ng**: Python-based OSINT framework with modular design for domain, email, and social media reconnaissance
135- **crt.sh**: Free certificate transparency search engine for subdomain and certificate discovery
136- **OSINT Framework (osintframework.com)**: Curated directory of OSINT tools organized by intelligence category
137
138## Common Pitfalls
139
140- **Leaving digital footprints**: Visiting a threat actor's website or Shodan-queried IP can alert the adversary. Use Tor or VPN with a dedicated OSINT VM.
141- **Confirmation bias in graph analysis**: Maltego graphs can create false connections. Verify each pivot independently before treating as confirmed.
142- **Outdated data**: WHOIS privacy services and bulletproof hosting rotate frequently. Always check data timestamps — 6-month-old passive DNS may no longer be valid.
143- **Attribution overconfidence**: Infrastructure overlap does not guarantee same threat actor. False flag operations deliberately share indicators across groups.
144- **Legal boundaries**: Some OSINT tools perform active scans (port scanning, banner grabbing). Confirm tool behavior before use against external targets without authorization.
theheavenlyd3mon