Performing Open Source Intelligence Gathering

Performing Open Source Intelligence Gathering

Legal Notice: This skill is for authorized security testing and educational purposes only. Unauthorized use against systems you do not own or have written permission to test is illegal and may violate computer fraud laws.

Overview

Open Source Intelligence (OSINT) gathering is the first active phase of a red team engagement, where operators collect publicly available information about the target organization to identify attack surfaces, potential targets for social engineering, technology stacks, and credential exposures. Effective OSINT directly shapes initial access strategies and reduces operational risk.

When to Use

Trigger phrases:

• "performing open source intelligence gathering"

• "Open Source Intelligence (OSINT) gathering is the first active phase of a red te"

• When conducting security assessments that involve performing open source intelligence gathering

• When following incident response procedures for related security events

• When performing scheduled security testing or auditing activities

• When validating security controls through hands-on testing

Prerequisites

• Familiarity with red teaming concepts and tools
• Access to a test or lab environment for safe execution
• Python 3.8+ with required dependencies installed
• Appropriate authorization for any testing activities

Objectives

• Enumerate the target organization's external attack surface (domains, IPs, cloud assets)
• Identify employees and their roles for social engineering targeting
• Discover leaked credentials, API keys, and sensitive documents
• Map the organization's technology stack and vendors
• Identify physical locations, office layouts, and access control details
• Build target profiles for spearphishing campaign development

Core Concepts

This section covers core concepts for performing open source intelligence gathering.

• Ensure all prerequisites are met before proceeding
• Follow the documented workflow steps in sequence
• Record results and any anomalies encountered during this phase

OSINT Categories

MITRE ATT&CK Mapping

• T1595.001 - Active Scanning: Scanning IP Blocks
• T1595.002 - Active Scanning: Vulnerability Scanning
• T1592 - Gather Victim Host Information
• T1589 - Gather Victim Identity Information
• T1590 - Gather Victim Network Information
• T1591 - Gather Victim Org Information
• T1593 - Search Open Websites/Domains
• T1594 - Search Victim-Owned Websites
• T1596 - Search Open Technical Databases

Workflow

# Example: IOC detection
import re

IOC_PATTERNS = {
    "ip": r"\b(?:\d{1,3}\.){3}\d{1,3}\b",
    "domain": r"\b[a-z0-9-]+\.[a-z]{2,}\b",
    "hash_md5": r"\b[a-f0-9]{32}\b",
    "hash_sha256": r"\b[a-f0-9]{64}\b",
}

def extract_iocs(text: str) -> dict:
    return {k: re.findall(v, text) for k, v in IOC_PATTERNS.items()}

1. Scope and authorize — confirm written authorization and define target boundaries
2. Reconnaissance — enumerate targets, services, and potential attack surfaces
3. Exploitation — attempt exploitation of identified vulnerabilities within scope
4. Post-exploitation — document access level, lateral movement, and data exposure
5. Report and remediate — compile findings with reproduction steps and fix recommendations

Phase 1: Domain and Network Reconnaissance

1. Perform WHOIS lookups for target domains
2. Enumerate subdomains using Certificate Transparency logs, DNS brute-force, and web scraping
3. Identify IP ranges and ASN ownership
4. Scan for exposed services using Shodan/Censys
5. Check for cloud storage buckets (S3, Azure Blob, GCS)
6. Map CDN and hosting providers

Phase 2: Personnel and Social Intelligence

1. Enumerate employees via LinkedIn, company website, and conference speaker lists
2. Identify email naming conventions
3. Discover personal social media accounts of key targets
4. Map organizational hierarchy and reporting structure
5. Identify recently hired IT/security personnel
6. Check for conference presentations and technical publications

Phase 3: Credential and Data Leak Discovery

1. Search breach databases (Have I Been Pwned, DeHashed)
2. Check paste sites (Pastebin, GitHub Gists)
3. Search GitHub/GitLab for leaked secrets and API keys
4. Look for exposed configuration files and backups
5. Check for leaked internal documents via Google dorking

Phase 4: Technology Stack Identification

1. Analyze job postings for technology mentions
2. Use Wappalyzer/BuiltWith for web technology fingerprinting
3. Check for exposed admin panels and development environments
4. Identify VPN and remote access technologies
5. Map cloud services and SaaS applications

Defensive OSINT (Threat Intelligence)

When investigating external threat actors rather than preparing for red team operations, OSINT collection must remain strictly passive (no packets sent to target systems). Use this parallel workflow:

1. Scope — Define research questions (e.g., "which threat actor targets our sector?"), identify data sources, set time boundaries
2. Gather — Collect data from passive sources only: WHOIS records, passive DNS (SecurityTrails), certificate transparency logs (crt.sh), breach databases (Have I Been Pwned), dark web monitoring, Shodan historical data
3. Synthesize — Enrich threat actor profiles with publicly observable indicators (IPs, domains, SSL certs, ASN data). Cross-reference findings with MITRE ATT&CK and known campaign indicators. Produce actionable threat intelligence reports

Key differences from offensive OSINT:

• No active scanning (no Nmap, no direct DNS queries to target)
• Focus on attribution and campaign tracking, not attack surface mapping
• Output is threat intelligence report, not red team recon report
• Emphasis on IOC extraction and enrichment, not exploitation planning

When NOT to Use

• You don't have explicit written authorization to test
• Task is about defense/detection, not offense (use detection skills)
• You need to implement security controls (use implementing-* skills)
• Task requires compliance auditing (use auditing-* skills)
• You're investigating an incident (use incident response skills)
• Target is out of scope for your engagement
• Task is about vulnerability scanning only (use scanning tools)

Red Flags

• Performing actions without explicit written authorization from the asset owner
• Testing against production systems without a defined scope and rules of engagement
• Exceeding the authorized scope of the engagement
• Leaving persistent access mechanisms without explicit approval
• Causing denial-of-service on production systems during testing

Verification

• All steps executed successfully against a test environment before production use
• Output documented with screenshots or logs demonstrating expected behavior
• All exploited vulnerabilities documented with reproduction steps
• Scope boundaries confirmed — only authorized targets were tested
• Remediation recommendations included for every finding

Tools and Resources

Validation Criteria

• Complete list of target domains and subdomains
• Employee list with roles and email addresses
• Technology stack identified
• Credential leak assessment completed
• Attack surface map documented
• OSINT report compiled for engagement team

Process

1. Analyze the task requirements
2. Apply domain expertise
3. Verify output quality

Anti-Rationalization Table