# Data Collection Agent

> Phase 2 orchestrator - Gathers technical signals from discovered assets

- Skill: `tools-only/data-collection-agent` (Agent Skill, multi-file: 3 files)
- Install (CLI): `npx skillmds add tools-only/data-collection-agent`
- Raw SKILL.md: https://api.skillmd.com/api/skills/tools-only/data-collection-agent/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: tools-only (https://skillmd.com/u/tools-only)
- Updated: 2026-09-09
- Page: https://skillmd.com/skills/tools-only/data-collection-agent

---


# Data Collection Agent

## Purpose

Phase 2 orchestrator responsible for gathering technical signals from all discovered assets through passive data collection techniques.

## Responsibilities

1. **HTTP Fingerprinting**: Collect headers, cookies, error pages for each asset
2. **DNS Intelligence**: Query all DNS record types (MX, TXT, NS, CNAME, SRV)
3. **TLS Analysis**: Extract certificate metadata, JARM/JA3 fingerprints
4. **JavaScript/DOM Analysis**: Detect framework globals, bundle patterns
5. **HTML Content Analysis**: Parse meta tags, comments, script URLs
6. **Code Repository Intel**: Scan GitHub/GitLab for public repos
7. **Job Posting Analysis**: Extract tech requirements from career pages
8. **Web Archive Analysis**: Query Wayback Machine for historical snapshots

## Skills Orchestrated

Execute in parallel groups where possible:

**Group 1 (Parallel):**
- `http_fingerprinting`
- `dns_intelligence`
- `tls_certificate_analysis`

**Group 2 (Parallel):**
- `javascript_dom_analysis`
- `html_content_analysis`

**Group 3 (Parallel):**
- `code_repository_intel`
- `job_posting_analysis`
- `web_archive_analysis`

## Input

Asset inventory from Phase 1:
```json
{
  "company": "string",
  "primary_domain": "string",
  "discovered_assets": {
    "domains": ["array"],
    "subdomains": ["array"],
    "ip_addresses": ["array"],
    "api_portals": ["array"]
  }
}
```

## Output

Raw signals JSON organized by source type:
```json
{
  "phase": 2,
  "company": "string",
  "signals": {
    "http_signals": [
      {
        "url": "string",
        "headers": {"object"},
        "cookies": ["array"],
        "status_code": "number",
        "server_signature": "string"
      }
    ],
    "dns_signals": {
      "mx_records": ["array"],
      "txt_records": ["array"],
      "ns_records": ["array"],
      "cname_records": ["array"],
      "srv_records": ["array"]
    },
    "tls_signals": [
      {
        "domain": "string",
        "issuer": "string",
        "protocol": "string",
        "cipher": "string",
        "jarm_hash": "string"
      }
    ],
    "javascript_signals": [
      {
        "url": "string",
        "globals_detected": ["array"],
        "dom_attributes": ["array"],
        "bundle_patterns": ["array"]
      }
    ],
    "html_signals": [
      {
        "url": "string",
        "meta_tags": ["array"],
        "generator_comments": ["array"],
        "script_urls": ["array"]
      }
    ],
    "repository_signals": {
      "org_found": "boolean",
      "org_url": "string",
      "repositories": [
        {
          "name": "string",
          "languages": ["array"],
          "dependencies": ["array"],
          "ci_configs": ["array"]
        }
      ]
    },
    "job_signals": {
      "ats_platform": "string",
      "tech_mentions": [
        {
          "technology": "string",
          "frequency": "number",
          "context": "string"
        }
      ]
    },
    "archive_signals": {
      "snapshots_analyzed": "number",
      "historical_tech": [
        {
          "technology": "string",
          "first_seen": "date",
          "last_seen": "date"
        }
      ]
    }
  },
  "timestamp": "ISO-8601"
}
```

## Execution Flow

```
INPUT: Asset Inventory
         │
         ├─────────────────────────────────────┐
         │           GROUP 1 (Parallel)        │
         ├─► http_fingerprinting               │
         ├─► dns_intelligence                  │
         └─► tls_certificate_analysis          │
                       │                       │
         ├─────────────────────────────────────┤
         │           GROUP 2 (Parallel)        │
         ├─► javascript_dom_analysis           │
         └─► html_content_analysis             │
                       │                       │
         ├─────────────────────────────────────┤
         │           GROUP 3 (Parallel)        │
         ├─► code_repository_intel             │
         ├─► job_posting_analysis              │
         └─► web_archive_analysis              │
                       │
                       ▼
         OUTPUT: Raw Signals JSON
```

## Error Handling

- Continue collection even if some sources fail
- Log failed sources with error details
- Never block on a single failed request
- Aggregate partial results when available

## Rate Limiting

- HTTP requests: 30/minute per domain
- GitHub API: 60/hour (unauthenticated)
- Wayback Machine: 15/minute
- DNS queries: No hard limit, but 2s delay between batches