Collecting Open-Source Intelligence
When to Use
Use this skill when:
- Investigating external infrastructure associated with a phishing campaign targeting your organization
- Enriching threat actor profiles with publicly observable indicators (WHOIS, ASN data, SSL certificates)
- Conducting authorized attack surface discovery to understand your organization's external exposure
Do not use this skill for active scanning against targets without explicit written authorization — OSINT collection must remain passive (no packets sent to target systems) unless scope permits active recon.
Detection Gaps & Validation
- Coverage gaps from a single source: Shodan, crt.sh, and passive DNS each see only a slice. A C2 behind Cloudflare shows the CDN IP, not origin; cert transparency misses some wildcard certs. Cross-check at least three independent sources before concluding infrastructure is mapped.
- False attribution from shared infrastructure: co-hosted domains on a bulletproof/shared host, or a reused Cobalt Strike watermark, do not establish a single actor. Treat each Maltego pivot as a hypothesis and verify the link independently.
- Stale data treated as live: 6-month-old passive DNS or expired WHOIS still resolves in the graph. Stamp every indicator with collection date and source, and discard pivots whose timestamps predate the campaign window.
- OPSEC leakage is also a coverage problem: directly visiting a target's site or actively probing its IP can tip off the adversary and crosses into active recon. Keep collection passive (cached/third-party data) unless scope authorizes active probing.
- Validate before reporting: confirm a candidate IOC against an independent enrichment (VirusTotal/OTX/PassiveTotal), record a confidence score, and explicitly note remaining collection gaps rather than presenting partial coverage as complete.
Prerequisites
- Maltego CE or commercial license for graph-based link analysis
- Shodan API key (https://shodan.io) for internet-wide device/service discovery
- OSINT Framework familiarity (https://osintframework.com) for tool selection
- SpiderFoot HX or open-source SpiderFoot for automated OSINT correlation
Workflow
Step 1: Define Collection Requirements
Establish the intelligence requirement (IR) before collecting. Document:
- Target: threat actor group, malicious domain, IP range, or organization
- Priority Intelligence Requirements (PIRs): What specific questions need answering?
- Legal authority: Passive OSINT is legal; active probing requires authorization
- Data handling: TLP classification for collected intelligence
Step 2: Passive DNS and WHOIS Investigation
# Passive DNS via SecurityTrails API
curl "https://api.securitytrails.com/v1/domain/evil-domain.com/dns/a" \
-H "apikey: YOUR_KEY"
# WHOIS history via ARIN / RIPE
whois -h whois.arin.net evil-domain.com
# Certificate transparency logs (no API key required)
curl "https://crt.sh/?q=%.evil-domain.com&output=json" | jq '.[].name_value'
Certificate transparency logs reveal all subdomains for a target domain, often exposing staging, VPN, or internal infrastructure inadvertently made public.
Step 3: Shodan Infrastructure Mapping
import shodan
api = shodan.Shodan("YOUR_SHODAN_API_KEY")
# Search for specific C2 framework signatures (Cobalt Strike beacon)
results = api.search('product:"Cobalt Strike" port:443')
for r in results['matches']:
print(r['ip_str'], r['port'], r['org'], r.get('ssl', {}).get('cert', {}).get('subject', ''))
# Find infrastructure associated with a known threat actor's ASN
results = api.search('asn:AS12345 http.title:"Redirector"')
Correlate Shodan results with passive DNS to build infrastructure clusters.
Step 4: Maltego Graph Analysis
In Maltego, use these built-in transforms for threat actor infrastructure mapping:
- Start with a known malicious domain (Entity: Domain)
- Run "To IP Address [DNS]" → identifies hosting IPs
- Run "To Shared Hosting" → identifies co-hosted domains (potentially same threat actor)
- Run "To DNS Name [Reverse DNS]" → identifies PTR records
- Run "To Whois" → identifies registrant email/organization
- Pivot on registrant email → "To Domains [Registrant Email]" → expands to all domains registered with same email
Maltego Maltego Cyber threat intelligence transforms (VirusTotal, Shodan, PassiveTotal, URLScan) extend graph coverage.
Step 5: Dark Web and Paste Site Monitoring
Use SpiderFoot HX or manual searches for:
- Paste sites (Pastebin, Ghostbin): search for leaked credentials, IOCs, malware configs
- Dark web forums: via Tor browser with appropriate operational security
- GitHub/GitLab: search for exposed credentials or organization-specific strings
# SpiderFoot CLI for automated OSINT
python sf.py -s evil-domain.com -m sfp_shodan,sfp_virustotal,sfp_passivetotal \
-o TF -R result.json
Key Concepts
| Term |
Definition |
| Passive OSINT |
Intelligence collection that does not send any packets to target systems — uses public databases, search engines, cached data |
| PIR |
Priority Intelligence Requirement — specific question the intelligence collection must answer, preventing unfocused data gathering |
| Certificate Transparency |
Public log of all SSL/TLS certificates issued by CAs, enabling discovery of subdomains via crt.sh |
| Pivoting |
Using one data point (IP, email, registrant name) to discover related infrastructure or accounts |
| ASN |
Autonomous System Number — block of IP addresses under a single routing policy; useful for clustering threat actor infrastructure |
| Co-hosted Domains |
Multiple domains resolving to the same IP, potentially indicating shared attacker infrastructure |
Tools & Systems
- Maltego: Graph-based link analysis platform with 50+ data source transforms for IP, domain, email, and social media analysis
- Shodan: Internet-wide scanner database with 1B+ indexed devices; supports banner, port, SSL certificate, and vulnerability searches
- SpiderFoot: Automated OSINT tool with 200+ modules covering DNS, WHOIS, dark web, breach data, and social media
- Recon-ng: Python-based OSINT framework with modular design for domain, email, and social media reconnaissance
- crt.sh: Free certificate transparency search engine for subdomain and certificate discovery
- OSINT Framework (osintframework.com): Curated directory of OSINT tools organized by intelligence category
Common Pitfalls
- Leaving digital footprints: Visiting a threat actor's website or Shodan-queried IP can alert the adversary. Use Tor or VPN with a dedicated OSINT VM.
- Confirmation bias in graph analysis: Maltego graphs can create false connections. Verify each pivot independently before treating as confirmed.
- Outdated data: WHOIS privacy services and bulletproof hosting rotate frequently. Always check data timestamps — 6-month-old passive DNS may no longer be valid.
- Attribution overconfidence: Infrastructure overlap does not guarantee same threat actor. False flag operations deliberately share indicators across groups.
- Legal boundaries: Some OSINT tools perform active scans (port scanning, banner grabbing). Confirm tool behavior before use against external targets without authorization.
1---2name: collecting-open-source-intelligence3description: Collects and synthesizes open-source intelligence (OSINT) about threat actors, malicious infrastructure, and attack campaigns using publicly available data sources, passive reconnaissance tools, and dark web monitoring. Use when investigating external threat actor infrastructure, performing pre-engagement reconnaissance for authorized red team assessments, or enriching CTI reports with publicly available adversary context. Activates for requests involving Maltego, Shodan, OSINT framework, SpiderFoot, or infrastructure reconnaissance.4license: Apache-2.05---6# Collecting Open-Source Intelligence
7
8## When to Use
9
10Use this skill when:
11- Investigating external infrastructure associated with a phishing campaign targeting your organization
12- Enriching threat actor profiles with publicly observable indicators (WHOIS, ASN data, SSL certificates)
13- Conducting authorized attack surface discovery to understand your organization's external exposure
14
15**Do not use** this skill for active scanning against targets without explicit written authorization — OSINT collection must remain passive (no packets sent to target systems) unless scope permits active recon.
16
17## Detection Gaps & Validation
18
19- **Coverage gaps from a single source:** Shodan, crt.sh, and passive DNS each see only a slice. A C2 behind Cloudflare shows the CDN IP, not origin; cert transparency misses some wildcard certs. Cross-check at least three independent sources before concluding infrastructure is mapped.
20- **False attribution from shared infrastructure:** co-hosted domains on a bulletproof/shared host, or a reused Cobalt Strike watermark, do not establish a single actor. Treat each Maltego pivot as a hypothesis and verify the link independently.
21- **Stale data treated as live:** 6-month-old passive DNS or expired WHOIS still resolves in the graph. Stamp every indicator with collection date and source, and discard pivots whose timestamps predate the campaign window.
22- **OPSEC leakage is also a coverage problem:** directly visiting a target's site or actively probing its IP can tip off the adversary and crosses into active recon. Keep collection passive (cached/third-party data) unless scope authorizes active probing.
23- **Validate before reporting:** confirm a candidate IOC against an independent enrichment (VirusTotal/OTX/PassiveTotal), record a confidence score, and explicitly note remaining collection gaps rather than presenting partial coverage as complete.
24
25## Prerequisites
26
27- Maltego CE or commercial license for graph-based link analysis
28- Shodan API key (https://shodan.io) for internet-wide device/service discovery
29- OSINT Framework familiarity (https://osintframework.com) for tool selection
30- SpiderFoot HX or open-source SpiderFoot for automated OSINT correlation
31
32## Workflow
33
34### Step 1: Define Collection Requirements
35
36Establish the intelligence requirement (IR) before collecting. Document:
37- Target: threat actor group, malicious domain, IP range, or organization
38- Priority Intelligence Requirements (PIRs): What specific questions need answering?
39- Legal authority: Passive OSINT is legal; active probing requires authorization
40- Data handling: TLP classification for collected intelligence
41
42### Step 2: Passive DNS and WHOIS Investigation
43
44```bash
45# Passive DNS via SecurityTrails API
46curl "https://api.securitytrails.com/v1/domain/evil-domain.com/dns/a" \
47 -H "apikey: YOUR_KEY"
48
49# WHOIS history via ARIN / RIPE
50whois -h whois.arin.net evil-domain.com
51
52# Certificate transparency logs (no API key required)
53curl "https://crt.sh/?q=%.evil-domain.com&output=json" | jq '.[].name_value'
54```
55
56Certificate transparency logs reveal all subdomains for a target domain, often exposing staging, VPN, or internal infrastructure inadvertently made public.
57
58### Step 3: Shodan Infrastructure Mapping
59
60```python
61import shodan
62
63api = shodan.Shodan("YOUR_SHODAN_API_KEY")
64
65# Search for specific C2 framework signatures (Cobalt Strike beacon)
66results = api.search('product:"Cobalt Strike" port:443')
67for r in results['matches']:
68 print(r['ip_str'], r['port'], r['org'], r.get('ssl', {}).get('cert', {}).get('subject', ''))
69
70# Find infrastructure associated with a known threat actor's ASN
71results = api.search('asn:AS12345 http.title:"Redirector"')
72```
73
74Correlate Shodan results with passive DNS to build infrastructure clusters.
75
76### Step 4: Maltego Graph Analysis
77
78In Maltego, use these built-in transforms for threat actor infrastructure mapping:
791. Start with a known malicious domain (Entity: Domain)
802. Run "To IP Address [DNS]" → identifies hosting IPs
813. Run "To Shared Hosting" → identifies co-hosted domains (potentially same threat actor)
824. Run "To DNS Name [Reverse DNS]" → identifies PTR records
835. Run "To Whois" → identifies registrant email/organization
846. Pivot on registrant email → "To Domains [Registrant Email]" → expands to all domains registered with same email
85
86Maltego Maltego Cyber threat intelligence transforms (VirusTotal, Shodan, PassiveTotal, URLScan) extend graph coverage.
87
88### Step 5: Dark Web and Paste Site Monitoring
89
90Use SpiderFoot HX or manual searches for:
91- Paste sites (Pastebin, Ghostbin): search for leaked credentials, IOCs, malware configs
92- Dark web forums: via Tor browser with appropriate operational security
93- GitHub/GitLab: search for exposed credentials or organization-specific strings
94
95```bash
96# SpiderFoot CLI for automated OSINT
97python sf.py -s evil-domain.com -m sfp_shodan,sfp_virustotal,sfp_passivetotal \
98 -o TF -R result.json
99```
100
101## Key Concepts
102
103| Term | Definition |
104|------|-----------|
105| **Passive OSINT** | Intelligence collection that does not send any packets to target systems — uses public databases, search engines, cached data |
106| **PIR** | Priority Intelligence Requirement — specific question the intelligence collection must answer, preventing unfocused data gathering |
107| **Certificate Transparency** | Public log of all SSL/TLS certificates issued by CAs, enabling discovery of subdomains via crt.sh |
108| **Pivoting** | Using one data point (IP, email, registrant name) to discover related infrastructure or accounts |
109| **ASN** | Autonomous System Number — block of IP addresses under a single routing policy; useful for clustering threat actor infrastructure |
110| **Co-hosted Domains** | Multiple domains resolving to the same IP, potentially indicating shared attacker infrastructure |
111
112## Tools & Systems
113
114- **Maltego**: Graph-based link analysis platform with 50+ data source transforms for IP, domain, email, and social media analysis
115- **Shodan**: Internet-wide scanner database with 1B+ indexed devices; supports banner, port, SSL certificate, and vulnerability searches
116- **SpiderFoot**: Automated OSINT tool with 200+ modules covering DNS, WHOIS, dark web, breach data, and social media
117- **Recon-ng**: Python-based OSINT framework with modular design for domain, email, and social media reconnaissance
118- **crt.sh**: Free certificate transparency search engine for subdomain and certificate discovery
119- **OSINT Framework (osintframework.com)**: Curated directory of OSINT tools organized by intelligence category
120
121## Common Pitfalls
122
123- **Leaving digital footprints**: Visiting a threat actor's website or Shodan-queried IP can alert the adversary. Use Tor or VPN with a dedicated OSINT VM.
124- **Confirmation bias in graph analysis**: Maltego graphs can create false connections. Verify each pivot independently before treating as confirmed.
125- **Outdated data**: WHOIS privacy services and bulletproof hosting rotate frequently. Always check data timestamps — 6-month-old passive DNS may no longer be valid.
126- **Attribution overconfidence**: Infrastructure overlap does not guarantee same threat actor. False flag operations deliberately share indicators across groups.
127- **Legal boundaries**: Some OSINT tools perform active scans (port scanning, banner grabbing). Confirm tool behavior before use against external targets without authorization.
gabrielmoreira