AWS Cloudwatch Investigation

AWS CloudWatch investigation

Take an AWS incident signal, transform it into scoped CloudWatch Logs, Metrics, Alarm, and CloudTrail evidence, and return a timeline, blast-radius conclusion, likely root event, and next validation steps.

When to invoke

• "Investigate this CloudWatch alarm."
• "Write Logs Insights queries for this error spike."
• "Did a deployment cause this AWS incident?"
• "Narrow the blast radius for these Lambda failures."
• "Build an incident timeline from CloudWatch and CloudTrail."

Prerequisites and context

• The user must provide or authorize access to the AWS account, region, log groups, alarm names, service namespace, and incident window.
• Prefer read-only investigation. Do not change alarms, dashboards, retention, Lambda memory, ECS services, or infrastructure unless separately asked.
• Use event timestamps, not ingestion timestamps, when comparing CloudWatch Logs, CloudWatch Metrics, CloudWatch Alarms, CloudTrail, and AWS Health.

Logs Insights query patterns

Deployment correlation

Use the alarm transition timestamp as the anchor. Query CloudTrail for deployment-related events in [alarm_time - 30min, alarm_time]:

SELECT eventTime, eventName, userIdentity.arn, requestParameters
FROM <event-data-store-id>
WHERE eventTime > '<alarm_time_minus_30m>'
  AND eventTime < '<alarm_time>'
  AND eventName IN (
    'UpdateFunctionCode', 'UpdateFunctionConfiguration',
    'UpdateService', 'CreateDeployment', 'RegisterTaskDefinition',
    'CreateChangeSet', 'ExecuteChangeSet',
    'StartPipelineExecution', 'PutImage'
  )
ORDER BY eventTime DESC

Strengthen the correlation by checking canary or synthetic monitor failures, scaling events, config changes, and whether any other environmental change happened in the same window.

Blast-radius decision tree

START
  |
  v
[1] ACCOUNT — Which account(s) show the alarm?
  |  - Multi-account: suspect shared service such as SSO, networking, or deployment pipeline
  |  - Single account: proceed to Region
  v
[2] REGION — Which region(s) are affected?
  |  - Multi-region: suspect global service such as IAM, Route53, or S3 global behavior
  |  - Single-region: proceed to Service
  v
[3] SERVICE — Which service namespace shows degradation?
  |  - Multiple services: suspect VPC, NAT, DNS, IAM, shared database, cache, or external API
  |  - Single service: proceed to Operation
  v
[4] OPERATION — Which API action, function, stage, resource, method, ECS service, or task definition is failing?
  |  - All operations: suspect service-level throttling or quota
  |  - Specific operation: proceed to Resource
  v
[5] RESOURCE — Which Function ARN, Task ID, DB instance identifier, or other resource instance is the investigation target?

When multiple services are affected, investigate in this order: VPC/Networking (NAT Gateway ErrorPortAllocation, packet drops, DNS), IAM/STS (ThrottlingException on AssumeRole, token vending latency), downstream dependency, shared deployment pipeline, then AWS Health Dashboard and Service Health.

Metric math patterns

Replace TARGET, FunctionName, ApiName, and TableName with the scoped resource. Treat 1000 as a default example account concurrency limit; use the account's actual quota when known.

Incident timeline reconstruction

Collect timestamped evidence and sort by event time:

Root event rule: walk backward from the first symptom to the most recent deploy, config change, scaling event, quota pressure, or external dependency shift that can explain all later symptoms.

Gotchas

• CloudWatch metric timestamps are end-of-period; a 1-minute datapoint at 14:05 covers 14:04-14:05.
• CloudTrail can have up to 15-minute delivery delay; use eventTime, not ingestion time.
• Log group timestamps depend on agent or SDK flush interval; allow 30-60 seconds of clock skew.
• Alarm state changes include evaluation delay: periods x evaluation periods; the anomaly often started earlier.

Source compatibility terms

Retain these CloudWatch incident terms in investigations and reports: 5/min, ALARM, AWS/ECS, Deployment/change, IAM/STS**, STRONG, StartTime/EndTime, VPC/Networking**, agent/SDK, alarm-to-deployment, assumed-role, assumed-role/github-actions-deploy/session, built-in, canary/synthetic, github-actions-deploy, multi-region, payment-processor, payments-api, service/task, single-region, stage/resource/method, us-east-1, EndTime, MetricName, MetricStat, PaymentProcessorErrors, and StartTime.

Output template

## CloudWatch investigation — <alarm, service, or incident>

**Status:** investigating | likely cause found | inconclusive | blocked
**Window:** <start> to <end> UTC
**Scope:** account=<id>, region=<region>, service=<namespace>, resource=<resource>

### Findings
| Time | Source | Evidence | Interpretation |
| --- | --- | --- | --- |
| `<timestamp>` | CloudTrail | `<eventName by actor>` | `<deploy/config/change candidate>` |
| `<timestamp>` | Logs Insights | `<query result>` | `<first symptom or dominant error>` |

### Blast radius
- Account: <single | multiple>
- Region: <single | multiple>
- Service: <single | multiple>
- Operation/resource: <specific target>

### Correlation
**Deployment correlation:** strong | medium | weak | none
**Root event:** <earliest plausible change or unknown>
**Confidence:** high | medium | low

### Queries used
- `<Logs Insights or MetricDataQueries summary>`

### Next checks
- <one concrete validation or mitigation step>

Quality gate

• Alarm transition time, first symptom time, and investigation window are explicit.
• Logs, metrics, alarms, CloudTrail, and AWS Health are considered or explicitly marked unavailable.
• Blast radius is narrowed in account → region → service → operation → resource order.
• Deployment correlation uses same resource, timing, actor, and prior-health evidence.
• Metric math names the namespace, metric, dimensions, period, statistic, and expression.
• Timeline entries use event timestamps and account for metric period, CloudTrail delay, log skew, and alarm evaluation delay.