CloudTrail Log Analysis
Instructions
Step 1: Ingestion and Flattening
- Understand Source: CloudTrail logs are nested JSON. Refer to
references/cloudtrail_format.mdfor field mapping. - Flatten for Analysis: Use
jqto convert theRecordsarray into JSONL format for easier processing.cat *.json | jq -c '.Records[]' > flattened.jsonl - DuckDB Setup: Ingest the flattened data into DuckDB for high-performance SQL querying.
import duckdb con = duckdb.connect('analysis.db') con.execute("CREATE TABLE events AS SELECT * FROM read_json_auto('flattened.jsonl')")
Step 2: Investigation
- Identify Anomalies: Search for
errorCodespikes or unauthorized operations. - Trace Principals: Follow the
userIdentity.arnandrole_arnto map the scope of an incident. - Document Actions: Capture all commands and findings in
analyst_log-YY-MM-DD-HH-MM.md.
Working Agreements
- Script Retention: Always create and retain scripts (e.g.,
analyze_*.py) in the current project directory. DO NOT place scripts in/tmpor other directories outside the project, and NEVER delete generated helper scripts or analysis code, as they must be preserved for future reference and reproducibility. - Persistence: Save confident data as a persistent
.dbfile. - Memory Safety: Use
polars.scan_ndjson()or DuckDB disk spilling for large datasets to prevent OOM. - Python Style: Use
orjson,polars, andduckdb. Useuvfor environment management. - No Analogies: Keep technical explanations direct and professional.
Examples
Example 1: Hunting for Unauthorized Discovery
User says: "Search for any 'Access Denied' errors in the last log batch." Action:
- Query for events where
errorCodeisAccessDeniedorClient.UnauthorizedOperation. - Group by
eventNameanduserIdentity.arnto find the most aggressive principal.
Example 2: Detecting Log Tampering
User says: "Did anyone try to disable CloudTrail?" Action:
- Search for
StopLogging,DeleteTrail, orUpdateTrailin theeventNamefield. - Identify the
sourceIPAddressanduserAgentof the requester.
Troubleshooting
Error: "DuckDB Out of Memory"
Cause: Loading massive raw JSON files into memory.
Solution: Set PRAGMA memory_limit='2GB' and use DuckDB's native JSON reader which supports disk spilling.
Error: "Missing Records"
Cause: Searching in Management Events for Data Event activity (e.g., S3 GetObject).
Solution: Verify if Data Events were enabled in the trail configuration. Check references/cloudtrail_security_research.md for event categories.
Error: "Invalid JSON format"
Cause: CloudTrail files are often gzipped or contain a single Records object rather than JSONL.
Solution: Use zcat for gzipped files and the jq flattening recipe in Step 1.