undermybelt Implementing CloudTrail Log Analysis
When to Use
- When building security monitoring pipelines for AWS API activity
- When investigating security incidents to trace attacker actions across AWS services
- When compliance requires audit logging of all administrative and data access operations
- When creating detection rules for known attack patterns in AWS environments
- When establishing baseline API behavior for anomaly detection
Do not use for real-time threat detection (use GuardDuty which already analyzes CloudTrail), for application-level logging (use CloudWatch Application Logs), or for network traffic analysis (use VPC Flow Logs).
Prerequisites
- CloudTrail enabled with management events and optionally data events across all accounts
- S3 bucket configured as CloudTrail delivery channel with appropriate retention policies
- Amazon Athena configured with CloudTrail log table for ad-hoc queries
- CloudWatch Logs subscription for real-time analysis with Logs Insights
- SIEM integration (Splunk, Elastic, or Security Lake) for production monitoring
Workflow
Step 1: Configure CloudTrail for Comprehensive Logging
Ensure CloudTrail captures all relevant event types across the organization.
# Create an organization trail (captures all accounts)
aws cloudtrail create-trail \
--name org-security-trail \
--s3-bucket-name cloudtrail-logs-org-ACCOUNT \
--is-organization-trail \
--is-multi-region-trail \
--include-global-service-events \
--enable-log-file-validation \
--kms-key-id alias/cloudtrail-key \
--cloud-watch-logs-log-group-arn arn:aws:logs:us-east-1:ACCOUNT:log-group:cloudtrail-org:* \
--cloud-watch-logs-role-arn arn:aws:iam::ACCOUNT:role/CloudTrailCloudWatchRole
# Start logging
aws cloudtrail start-logging --name org-security-trail
# Enable data events for S3 and Lambda
aws cloudtrail put-event-selectors \
--trail-name org-security-trail \
--advanced-event-selectors '[
{
"Name": "S3DataEvents",
"FieldSelectors": [
{"Field": "eventCategory", "Equals": ["Data"]},
{"Field": "resources.type", "Equals": ["AWS::S3::Object"]}
]
},
{
"Name": "LambdaDataEvents",
"FieldSelectors": [
{"Field": "eventCategory", "Equals": ["Data"]},
{"Field": "resources.type", "Equals": ["AWS::Lambda::Function"]}
]
}
]'
# Verify trail configuration
aws cloudtrail describe-trails --trail-name-list org-security-trail
Step 2: Set Up Athena for CloudTrail Query Analysis
Create an Athena table for querying CloudTrail logs with SQL.
-- Create CloudTrail Athena table
CREATE EXTERNAL TABLE cloudtrail_logs (
eventVersion STRING,
userIdentity STRUCT<
type:STRING, principalId:STRING, arn:STRING,
accountId:STRING, invokedBy:STRING,
accessKeyId:STRING, userName:STRING,
sessionContext:STRUCT<
attributes:STRUCT<mfaAuthenticated:STRING, creationDate:STRING>,
sessionIssuer:STRUCT<type:STRING, principalId:STRING, arn:STRING, accountId:STRING, userName:STRING>
>
>,
eventTime STRING,
eventSource STRING,
eventName STRING,
awsRegion STRING,
sourceIPAddress STRING,
userAgent STRING,
errorCode STRING,
errorMessage STRING,
requestParameters STRING,
responseElements STRING,
additionalEventData STRING,
requestId STRING,
eventId STRING,
readOnly STRING,
resources ARRAY<STRUCT<arn:STRING, accountId:STRING, type:STRING>>,
eventType STRING,
apiVersion STRING,
recipientAccountId STRING,
sharedEventId STRING,
vpcEndpointId STRING
)
PARTITIONED BY (region STRING, year STRING, month STRING, day STRING)
ROW FORMAT SERDE 'org.apache.hive.hcatalog.data.JsonSerDe'
LOCATION 's3://cloudtrail-logs-org-ACCOUNT/AWSLogs/ORG_ID/';
-- Add partitions for recent data
ALTER TABLE cloudtrail_logs ADD
PARTITION (region='us-east-1', year='2026', month='02', day='23')
LOCATION 's3://cloudtrail-logs-org-ACCOUNT/AWSLogs/ORG_ID/ACCOUNT/CloudTrail/us-east-1/2026/02/23/';
Step 3: Run Security-Focused Athena Queries
Execute queries to detect common attack patterns and suspicious activity.
-- Detect console logins without MFA
SELECT eventtime, useridentity.username, sourceipaddress, useridentity.arn
FROM cloudtrail_logs
WHERE eventname = 'ConsoleLogin'
AND additionalEventData LIKE '%"MFAUsed":"No"%'
AND errorcode IS NULL
ORDER BY eventtime DESC;
-- Find IAM privilege escalation attempts
SELECT eventtime, useridentity.arn, eventname, errorcode, sourceipaddress
FROM cloudtrail_logs
WHERE eventname IN (
'CreatePolicyVersion', 'SetDefaultPolicyVersion', 'AttachUserPolicy',
'AttachRolePolicy', 'PutUserPolicy', 'PutRolePolicy',
'CreateAccessKey', 'CreateLoginProfile', 'UpdateLoginProfile',
'PassRole', 'AssumeRole'
)
ORDER BY eventtime DESC
LIMIT 100;
-- Detect CloudTrail tampering
SELECT eventtime, useridentity.arn, eventname, requestparameters, sourceipaddress
FROM cloudtrail_logs
WHERE eventname IN ('StopLogging', 'DeleteTrail', 'UpdateTrail', 'PutEventSelectors')
ORDER BY eventtime DESC;
-- Find API calls from Tor exit nodes or unusual IPs
SELECT eventtime, useridentity.arn, eventname, sourceipaddress, awsregion
FROM cloudtrail_logs
WHERE sourceipaddress NOT LIKE '10.%'
AND sourceipaddress NOT LIKE '172.%'
AND sourceipaddress NOT LIKE '192.168.%'
AND useridentity.type = 'IAMUser'
AND errorcode IS NULL
GROUP BY eventtime, useridentity.arn, eventname, sourceipaddress, awsregion
ORDER BY eventtime DESC
LIMIT 200;
-- Detect unauthorized API calls (AccessDenied patterns)
SELECT useridentity.arn, eventname, COUNT(*) as denied_count
FROM cloudtrail_logs
WHERE errorcode IN ('AccessDenied', 'UnauthorizedAccess', 'Client.UnauthorizedAccess')
AND eventtime > date_format(date_add('day', -7, now()), '%Y-%m-%dT%H:%i:%sZ')
GROUP BY useridentity.arn, eventname
HAVING COUNT(*) > 10
ORDER BY denied_count DESC;
Step 4: Build Real-Time Detection with CloudWatch Logs Insights
Create real-time queries for active security monitoring.
# Detect root account usage
aws logs start-query \
--log-group-name cloudtrail-org \
--start-time $(date -d "24 hours ago" +%s) \
--end-time $(date +%s) \
--query-string '
fields @timestamp, eventName, sourceIPAddress, userAgent
| filter userIdentity.type = "Root"
| sort @timestamp desc
'
# Detect security group changes
aws logs start-query \
--log-group-name cloudtrail-org \
--start-time $(date -d "24 hours ago" +%s) \
--end-time $(date +%s) \
--query-string '
fields @timestamp, userIdentity.arn, eventName, requestParameters.groupId, sourceIPAddress
| filter eventName in ["AuthorizeSecurityGroupIngress", "AuthorizeSecurityGroupEgress", "RevokeSecurityGroupIngress", "CreateSecurityGroup"]
| sort @timestamp desc
'
# Detect new IAM users or access keys created
aws logs start-query \
--log-group-name cloudtrail-org \
--start-time $(date -d "24 hours ago" +%s) \
--end-time $(date +%s) \
--query-string '
fields @timestamp, userIdentity.arn, eventName, requestParameters.userName, sourceIPAddress
| filter eventName in ["CreateUser", "CreateAccessKey", "CreateLoginProfile"]
| sort @timestamp desc
'
Step 5: Create CloudWatch Metric Filters and Alarms
Set up automated alerting for critical security events based on CIS Benchmark recommendations.
# CIS 3.1: Unauthorized API calls alarm
aws logs put-metric-filter \
--log-group-name cloudtrail-org \
--filter-name unauthorized-api-calls \
--filter-pattern '{($.errorCode = "*UnauthorizedAccess") || ($.errorCode = "AccessDenied*")}' \
--metric-transformations '[{"metricName":"UnauthorizedAPICalls","metricNamespace":"CISBenchmark","metricValue":"1"}]'
aws cloudwatch put-metric-alarm \
--alarm-name cis-unauthorized-api-calls \
--metric-name UnauthorizedAPICalls --namespace CISBenchmark \
--statistic Sum --period 300 --threshold 10 \
--comparison-operator GreaterThanThreshold --evaluation-periods 1 \
--alarm-actions arn:aws:sns:us-east-1:ACCOUNT:security-alerts
# CIS 3.3: Root account usage alarm
aws logs put-metric-filter \
--log-group-name cloudtrail-org \
--filter-name root-account-usage \
--filter-pattern '{$.userIdentity.type = "Root" && $.userIdentity.invokedBy NOT EXISTS && $.eventType != "AwsServiceEvent"}' \
--metric-transformations '[{"metricName":"RootAccountUsage","metricNamespace":"CISBenchmark","metricValue":"1"}]'
# CIS 3.4: IAM policy changes alarm
aws logs put-metric-filter \
--log-group-name cloudtrail-org \
--filter-name iam-policy-changes \
--filter-pattern '{($.eventName=CreatePolicy) || ($.eventName=DeletePolicy) || ($.eventName=AttachRolePolicy) || ($.eventName=DetachRolePolicy) || ($.eventName=AttachUserPolicy) || ($.eventName=DetachUserPolicy)}' \
--metric-transformations '[{"metricName":"IAMPolicyChanges","metricNamespace":"CISBenchmark","metricValue":"1"}]'
# CIS 3.5: CloudTrail configuration changes alarm
aws logs put-metric-filter \
--log-group-name cloudtrail-org \
--filter-name cloudtrail-changes \
--filter-pattern '{($.eventName = StopLogging) || ($.eventName = DeleteTrail) || ($.eventName = UpdateTrail)}' \
--metric-transformations '[{"metricName":"CloudTrailChanges","metricNamespace":"CISBenchmark","metricValue":"1"}]'
Key Concepts
| Term | Definition |
|---|---|
| CloudTrail | AWS service that records API calls made to AWS services, providing an audit trail of actions taken by users, roles, and services |
| Management Events | CloudTrail events for control plane operations like creating resources, modifying IAM, and configuring services |
| Data Events | CloudTrail events for data plane operations like S3 object access and Lambda function invocations, providing granular activity logging |
| Log File Validation | CloudTrail feature that creates a digest file for verifying that log files have not been tampered with after delivery |
| CloudTrail Lake | Managed data lake for CloudTrail events enabling SQL-based queries without managing Athena tables or S3 data |
| Organization Trail | Single trail that captures API activity across all accounts in an AWS Organization to a central S3 bucket |
Tools & Systems
- Amazon Athena: Serverless SQL query engine for analyzing CloudTrail logs stored in S3 at scale
- CloudWatch Logs Insights: Real-time log query service for interactive CloudTrail analysis within the last 30 days
- CloudTrail Lake: Managed event data lake with built-in SQL query capabilities and 7-year retention
- Amazon Security Lake: Centralized security data lake that normalizes CloudTrail data into OCSF format for SIEM consumption
- AWS CloudTrail: Core audit logging service capturing all API activity across AWS accounts and services
Common Scenarios
Scenario: Investigating an IAM Credential Compromise Through CloudTrail
Context: GuardDuty alerts on UnauthorizedAccess:IAMUser/MaliciousIPCaller for a developer's access key. The security team needs to trace all actions taken by the compromised credential.
Approach:
- Query CloudTrail for all events by the compromised AccessKeyId across all regions
- Build a timeline of API calls to understand the attack sequence
- Identify the initial access point (when did the key first appear from a malicious IP)
- Map all resources created, modified, or accessed by the attacker
- Check for persistence mechanisms (new users, access keys, Lambda functions, EC2 instances)
- Verify CloudTrail was not tampered with (check for StopLogging or UpdateTrail events)
- Document the full attack chain and scope of impact for the incident response report
Pitfalls: CloudTrail events can take up to 15 minutes to appear in S3 and CloudWatch Logs. For real-time visibility during active incidents, use CloudTrail Lake or CloudWatch Logs Insights rather than Athena queries against S3. Cross-region attacks require querying multiple region partitions in Athena.
Output Format
CloudTrail Security Analysis Report
======================================
Account: 123456789012
Analysis Period: 2026-02-16 to 2026-02-23
Trail: org-security-trail (organization-wide)
SECURITY EVENTS DETECTED:
Root account logins: 2
Console logins without MFA: 7
Privilege escalation attempts: 12
CloudTrail configuration changes: 0
Security group modifications: 34
Unauthorized API calls: 156
HIGH-PRIORITY FINDINGS:
[CT-001] Console Login Without MFA
User: admin-user
Time: 2026-02-22T14:30:00Z
IP: 203.0.113.50
Action Required: Enforce MFA via IAM policy
[CT-002] IAM Privilege Escalation
User: dev-user
Time: 2026-02-23T03:15:00Z
Events: CreatePolicyVersion -> AttachRolePolicy
IP: 185.x.x.x (suspicious)
Action Required: Investigate credential compromise
ALERTING STATUS:
CIS metric filters configured: 14 / 14
CloudWatch alarms active: 14 / 14
Alerts fired (last 7 days): 8