Data Collection Agent

Data Collection Agent

Purpose

Phase 2 orchestrator responsible for gathering technical signals from all discovered assets through passive data collection techniques.

Responsibilities

1. HTTP Fingerprinting: Collect headers, cookies, error pages for each asset
2. DNS Intelligence: Query all DNS record types (MX, TXT, NS, CNAME, SRV)
3. TLS Analysis: Extract certificate metadata, JARM/JA3 fingerprints
4. JavaScript/DOM Analysis: Detect framework globals, bundle patterns
5. HTML Content Analysis: Parse meta tags, comments, script URLs
6. Code Repository Intel: Scan GitHub/GitLab for public repos
7. Job Posting Analysis: Extract tech requirements from career pages
8. Web Archive Analysis: Query Wayback Machine for historical snapshots

Skills Orchestrated

Execute in parallel groups where possible:

Group 1 (Parallel):

• http_fingerprinting
• dns_intelligence
• tls_certificate_analysis

Group 2 (Parallel):

• javascript_dom_analysis
• html_content_analysis

Group 3 (Parallel):

• code_repository_intel
• job_posting_analysis
• web_archive_analysis

Input

Asset inventory from Phase 1:

{
  "company": "string",
  "primary_domain": "string",
  "discovered_assets": {
    "domains": ["array"],
    "subdomains": ["array"],
    "ip_addresses": ["array"],
    "api_portals": ["array"]
  }
}

Output

Raw signals JSON organized by source type:

{
  "phase": 2,
  "company": "string",
  "signals": {
    "http_signals": [
      {
        "url": "string",
        "headers": {"object"},
        "cookies": ["array"],
        "status_code": "number",
        "server_signature": "string"
      }
    ],
    "dns_signals": {
      "mx_records": ["array"],
      "txt_records": ["array"],
      "ns_records": ["array"],
      "cname_records": ["array"],
      "srv_records": ["array"]
    },
    "tls_signals": [
      {
        "domain": "string",
        "issuer": "string",
        "protocol": "string",
        "cipher": "string",
        "jarm_hash": "string"
      }
    ],
    "javascript_signals": [
      {
        "url": "string",
        "globals_detected": ["array"],
        "dom_attributes": ["array"],
        "bundle_patterns": ["array"]
      }
    ],
    "html_signals": [
      {
        "url": "string",
        "meta_tags": ["array"],
        "generator_comments": ["array"],
        "script_urls": ["array"]
      }
    ],
    "repository_signals": {
      "org_found": "boolean",
      "org_url": "string",
      "repositories": [
        {
          "name": "string",
          "languages": ["array"],
          "dependencies": ["array"],
          "ci_configs": ["array"]
        }
      ]
    },
    "job_signals": {
      "ats_platform": "string",
      "tech_mentions": [
        {
          "technology": "string",
          "frequency": "number",
          "context": "string"
        }
      ]
    },
    "archive_signals": {
      "snapshots_analyzed": "number",
      "historical_tech": [
        {
          "technology": "string",
          "first_seen": "date",
          "last_seen": "date"
        }
      ]
    }
  },
  "timestamp": "ISO-8601"
}

Execution Flow

INPUT: Asset Inventory
         │
         ├─────────────────────────────────────┐
         │           GROUP 1 (Parallel)        │
         ├─► http_fingerprinting               │
         ├─► dns_intelligence                  │
         └─► tls_certificate_analysis          │
                       │                       │
         ├─────────────────────────────────────┤
         │           GROUP 2 (Parallel)        │
         ├─► javascript_dom_analysis           │
         └─► html_content_analysis             │
                       │                       │
         ├─────────────────────────────────────┤
         │           GROUP 3 (Parallel)        │
         ├─► code_repository_intel             │
         ├─► job_posting_analysis              │
         └─► web_archive_analysis              │
                       │
                       ▼
         OUTPUT: Raw Signals JSON

Error Handling

• Continue collection even if some sources fail
• Log failed sources with error details
• Never block on a single failed request
• Aggregate partial results when available

Rate Limiting

• HTTP requests: 30/minute per domain
• GitHub API: 60/hour (unauthenticated)
• Wayback Machine: 15/minute
• DNS queries: No hard limit, but 2s delay between batches