datadog-labs Audit Trail: API Key Compromise Investigation
Reconstruct what a Datadog API key did, where requests originated, and which resources were affected.
Prerequisites
pup auth login # OAuth2 (recommended)
# or set DD_API_KEY + DD_APP_KEY with audit_logs_read scope
You need the key ID of the suspect key (not the key value). Find it in Datadog UI under Organization Settings > API Keys, or from context showing @metadata.api_key.id.
Investigation Workflow
Step 1 — Establish timeline
pup audit-logs search --query "@metadata.api_key.id:KEY_ID" --from 90d --limit 200 -o json \
| jq '[.data[] | {
timestamp: .attributes.timestamp,
action: .attributes.attributes.action,
event: .attributes.attributes.evt.name,
resource_type: .attributes.attributes.asset.type,
resource_id: .attributes.attributes.asset.id,
endpoint: .attributes.attributes.http.url_details.path,
method: .attributes.attributes.http.method,
ip: .attributes.attributes.network.client.ip,
city: .attributes.attributes.network.client.geoip.city.name,
country: .attributes.attributes.network.client.geoip.country.name,
asn: .attributes.attributes.network.client.geoip.as.name
}]'
Step 2 — Geo/IP breakdown
pup audit-logs search --query "@metadata.api_key.id:KEY_ID" --from 90d --limit 500 -o json \
| jq '[.data[] | {
country: .attributes.attributes.network.client.geoip.country.name,
asn: .attributes.attributes.network.client.geoip.as.name,
ip: .attributes.attributes.network.client.ip
}]
| group_by(.country)
| map({
country: .[0].country,
count: length,
asns: [.[].asn] | unique,
ips: [.[].ip] | unique
})
| sort_by(-.count)'
Step 3 — Endpoint breakdown
pup audit-logs search --query "@metadata.api_key.id:KEY_ID" --from 90d --limit 500 -o json \
| jq '[.data[] | {
method: .attributes.attributes.http.method,
path: .attributes.attributes.http.url_details.path
}]
| group_by(.path)
| map({path: .[0].path, methods: [.[].method] | unique, count: length})
| sort_by(-.count)'
Step 4 — Destructive action check
pup audit-logs search --query "@metadata.api_key.id:KEY_ID @action:deleted" --from 90d -o json \
| jq '[.data[] | {
timestamp: .attributes.timestamp,
resource_type: .attributes.attributes.asset.type,
resource_id: .attributes.attributes.asset.id,
ip: .attributes.attributes.network.client.ip,
country: .attributes.attributes.network.client.geoip.country.name
}]'
Step 5 — When was the key created and by whom?
pup audit-logs search --query "@asset.type:api_key @asset.id:KEY_ID @action:created" --from 90d -o json \
| jq '[.data[] | {
created_at: .attributes.timestamp,
created_by: .attributes.attributes.usr.email,
creator_ip: .attributes.attributes.network.client.ip,
creator_country: .attributes.attributes.network.client.geoip.country.name
}]'
Anomaly Flags
| Signal | Why it matters |
|---|---|
| Country not in org's normal baseline | Possible exfiltration from unexpected region |
| ASN is a cloud/VPN provider (AWS, Cloudflare, NordVPN, etc.) | Proxied traffic; obscured origin |
| DELETE actions on monitors, dashboards, or log pipelines | Potential sabotage |
| Burst of activity in short window | Automated scraping or bulk exfiltration |
| Activity outside business hours | Off-hours access |
| Key used from multiple IPs simultaneously | Key shared or stolen |
Investigation Output Format
Key ID: <key_id>
Created: <timestamp> by <user_email>
Active period: <first_seen> to <last_seen>
Total events: <N>
Origins:
- <Country> (<ASN>): <N> events — [NORMAL / FLAG: first-time origin]
Endpoints called (top 5):
- <METHOD> <path>: <N> calls
Destructive actions: <N> deletions — [resource types affected]
Recommended actions:
1. Revoke the key immediately if not already done
2. Review affected resources: [list]
3. Check if any deleted resources need restoration
4. Audit who else had access to this key
Remediation
Revoke in Datadog UI: Organization Settings > API Keys > Revoke.
Or via API (requires manage_api_keys scope):
pup api-keys delete KEY_ID