OSINT Collection Methodology
Collection Planning
Before collecting, define:
- Objective: What specific intelligence do we need?
- Scope: What sources are relevant? What's out of scope?
- Keywords: Search terms, aliases, IOCs to look for
- Time constraints: How recent must the intelligence be?
- Documentation: How will findings be recorded?
Source Categories
| Category |
Sources |
Best For |
| Government advisories |
CISA, NCSC, CERT-EU, national CERTs |
Authoritative threat alerts, vulnerability warnings |
| Vendor reports |
Mandiant, CrowdStrike, Microsoft, Recorded Future, Talos |
Campaign analysis, actor profiles, malware analysis |
| Security research |
Blogs, conference talks, academic papers |
Novel techniques, deep dives, vulnerability research |
| Underground |
Forum mirrors, paste sites, Telegram (public), leak aggregators |
Threat actor communications, credential dumps, tools |
| Code repositories |
GitHub, GitLab |
Malware source, PoC exploits, attacker tools, leaked configs |
| Certificate transparency |
crt.sh, Censys |
Domain enumeration, infrastructure mapping |
| DNS/WHOIS |
SecurityTrails, DomainTools, RiskIQ (via web search) |
Infrastructure relationships, historical records |
| News/media |
Tech press, infosec news sites |
Incident reports, geopolitical context |
| Social media |
X/Twitter, LinkedIn, Reddit (public) |
OPSEC failures, community intelligence, actor claims |
Search Operator Cheat Sheet
Google Dorks
site:example.com filetype:pdf # PDFs on a specific site
"threat actor" AND "APT28" -site:reddit.com # Exclude noise
inurl:"/wp-content/uploads/" filetype:exe # Exposed uploads
"index of" "/backup" # Directory listings
intitle:"login" site:*.example.com # Login pages on subdomains
GitHub Search
org:target-org password # Leaked credentials
filename:.env DB_PASSWORD # Exposed environment files
"api_key" language:python # Hardcoded API keys
"BEGIN RSA PRIVATE KEY" # Exposed private keys
Certificate Transparency (crt.sh)
https://crt.sh/?q=%.example.com # All certs for subdomains
https://crt.sh/?q=example.com&output=json # JSON output
Shodan Dorks (via web)
ssl.cert.subject.cn:"example.com" # Certificates for domain
http.title:"Dashboard" org:"Target" # Web dashboards
product:"Cobalt Strike" # C2 servers
Collection Process
1. Broad Sweep
Start with broad searches to understand what's available:
- Search for the topic/actor/indicator across major sources
- Note the quantity and quality of available information
- Identify the most promising sources for deeper collection
2. Targeted Collection
Narrow down to specific sources and extract detailed intelligence:
- Read full reports, not just headlines
- Extract specific IOCs, TTPs, dates, and attribution claims
- Note each source's perspective and potential biases
3. Source Assessment
Apply Admiralty Scale to every finding (per source-assessment skill):
- Rate source reliability (A-F)
- Rate information credibility (1-6)
- Note whether information is corroborated across sources
4. Documentation
Record every finding with full attribution:
#### Finding: [Title]
- **Source**: [URL or description]
- **Source rating**: [Admiralty code, e.g., B2]
- **Date collected**: YYYY-MM-DD
- **Date of information**: YYYY-MM-DD (when was this information produced?)
- **Summary**: [Key intelligence extracted]
- **Relevance**: [How this relates to the collection objective]
Ethical and Legal Boundaries
- Only access publicly available information
- Do not create fake accounts or impersonate anyone
- Do not attempt to access restricted systems
- Respect robots.txt and terms of service
- Be aware of data protection regulations (GDPR)
- Do not engage with threat actors or participate in illegal activity
Related skills
When OSINT collection surfaces an indicator, route into the appropriate enrichment / pivot:
- Bulk IOC enrichment —
/ioc-enrichment-workflow chains /lookup-virustotal, /lookup-otx, /lookup-shodan, /lookup-abuseipdb, /lookup-greynoise, /lookup-urlscan, /lookup-censys, and /lookup-misp
- Single-indicator first hop —
/ip-investigation, /domain-investigation, /hash-investigation, /url-investigation
- Multi-hop graph walk —
/indicator-pivoting
- Internal correlation against your own catalogue —
/lookup-misp search-attributes / search-events
- Ransomware victim-status sweeps on org names —
/lookup-ransomwarelive search --q <orgname>
- Underground forums and Telegram —
/darkweb-collection (read its OPSEC primer before any DIY collection)
- Apply rigor to the finished product —
/score-source, /apply-tlp, /confidence-language, /likelihood-language
1---2name: osint-methodology3description: Structured OSINT collection methodology. Planning, collection techniques, search operators, and documentation. Loaded by the osint-researcher agent.4---56# OSINT Collection Methodology78## Collection Planning910Before collecting, define:111. **Objective**: What specific intelligence do we need?122. **Scope**: What sources are relevant? What's out of scope?133. **Keywords**: Search terms, aliases, IOCs to look for144. **Time constraints**: How recent must the intelligence be?155. **Documentation**: How will findings be recorded?1617## Source Categories1819| Category | Sources | Best For |20|----------|---------|----------|21| **Government advisories** | CISA, NCSC, CERT-EU, national CERTs | Authoritative threat alerts, vulnerability warnings |22| **Vendor reports** | Mandiant, CrowdStrike, Microsoft, Recorded Future, Talos | Campaign analysis, actor profiles, malware analysis |23| **Security research** | Blogs, conference talks, academic papers | Novel techniques, deep dives, vulnerability research |24| **Underground** | Forum mirrors, paste sites, Telegram (public), leak aggregators | Threat actor communications, credential dumps, tools |25| **Code repositories** | GitHub, GitLab | Malware source, PoC exploits, attacker tools, leaked configs |26| **Certificate transparency** | crt.sh, Censys | Domain enumeration, infrastructure mapping |27| **DNS/WHOIS** | SecurityTrails, DomainTools, RiskIQ (via web search) | Infrastructure relationships, historical records |28| **News/media** | Tech press, infosec news sites | Incident reports, geopolitical context |29| **Social media** | X/Twitter, LinkedIn, Reddit (public) | OPSEC failures, community intelligence, actor claims |3031## Search Operator Cheat Sheet3233### Google Dorks34```35site:example.com filetype:pdf # PDFs on a specific site36"threat actor" AND "APT28" -site:reddit.com # Exclude noise37inurl:"/wp-content/uploads/" filetype:exe # Exposed uploads38"index of" "/backup" # Directory listings39intitle:"login" site:*.example.com # Login pages on subdomains40```4142### GitHub Search43```44org:target-org password # Leaked credentials45filename:.env DB_PASSWORD # Exposed environment files46"api_key" language:python # Hardcoded API keys47"BEGIN RSA PRIVATE KEY" # Exposed private keys48```4950### Certificate Transparency (crt.sh)51```52https://crt.sh/?q=%.example.com # All certs for subdomains53https://crt.sh/?q=example.com&output=json # JSON output54```5556### Shodan Dorks (via web)57```58ssl.cert.subject.cn:"example.com" # Certificates for domain59http.title:"Dashboard" org:"Target" # Web dashboards60product:"Cobalt Strike" # C2 servers61```6263## Collection Process6465### 1. Broad Sweep66Start with broad searches to understand what's available:67- Search for the topic/actor/indicator across major sources68- Note the quantity and quality of available information69- Identify the most promising sources for deeper collection7071### 2. Targeted Collection72Narrow down to specific sources and extract detailed intelligence:73- Read full reports, not just headlines74- Extract specific IOCs, TTPs, dates, and attribution claims75- Note each source's perspective and potential biases7677### 3. Source Assessment78Apply Admiralty Scale to every finding (per source-assessment skill):79- Rate source reliability (A-F)80- Rate information credibility (1-6)81- Note whether information is corroborated across sources8283### 4. Documentation84Record every finding with full attribution:85```markdown86#### Finding: [Title]87- **Source**: [URL or description]88- **Source rating**: [Admiralty code, e.g., B2]89- **Date collected**: YYYY-MM-DD90- **Date of information**: YYYY-MM-DD (when was this information produced?)91- **Summary**: [Key intelligence extracted]92- **Relevance**: [How this relates to the collection objective]93```9495## Ethical and Legal Boundaries96- Only access publicly available information97- Do not create fake accounts or impersonate anyone98- Do not attempt to access restricted systems99- Respect robots.txt and terms of service100- Be aware of data protection regulations (GDPR)101- Do not engage with threat actors or participate in illegal activity102103## Related skills104105When OSINT collection surfaces an indicator, route into the appropriate enrichment / pivot:106107- **Bulk IOC enrichment** — `/ioc-enrichment-workflow` chains `/lookup-virustotal`, `/lookup-otx`, `/lookup-shodan`, `/lookup-abuseipdb`, `/lookup-greynoise`, `/lookup-urlscan`, `/lookup-censys`, and `/lookup-misp`108- **Single-indicator first hop** — `/ip-investigation`, `/domain-investigation`, `/hash-investigation`, `/url-investigation`109- **Multi-hop graph walk** — `/indicator-pivoting`110- **Internal correlation against your own catalogue** — `/lookup-misp search-attributes` / `search-events`111- **Ransomware victim-status sweeps on org names** — `/lookup-ransomwarelive search --q <orgname>`112- **Underground forums and Telegram** — `/darkweb-collection` (read its OPSEC primer before any DIY collection)113- **Apply rigor to the finished product** — `/score-source`, `/apply-tlp`, `/confidence-language`, `/likelihood-language`