Incident Investigator
Investigate production incidents by querying Datadog for logs, metrics, and traces, then synthesizing findings into a structured root-cause report.
When to Activate
- On-call alert triggered and developer needs to understand what's happening
- Service degradation, error spike, or latency increase detected
- User asks "what's causing this?" or "why is X failing in production?"
- Post-incident root cause analysis needed
- Monitoring dashboard shows anomaly requiring investigation
Steps
Gather incident context — confirm the affected service, error message or alert name, and approximate start time. If the user hasn't provided a time range, default to the last hour. Ask for any missing critical context before querying.
Query logs — search Datadog logs for error patterns in the incident window. Filter by service name and status:error. Surface the top recurring error messages and stack traces. Note frequency and distribution over time to establish whether the issue is steady-state or spiking.
Check metrics and monitors — retrieve error rate, latency (p50/p95/p99), and saturation metrics (CPU, memory, throughput) for the affected service. Check for triggered monitors around the incident start time to correlate alert onset with metric changes.
Pull traces — look up traces for failing requests to identify where in the call chain the failure originates. Note slow spans, timeouts, and downstream service dependencies that may indicate the root cause location.
Correlate events — search for deployment events, configuration changes, or upstream service issues in the same window. A deploy or config push immediately preceding the error onset is strong evidence of causation.
Generate report — synthesize findings into a structured incident report with timeline, root cause, confidence level, and actionable remediation steps.
Output Format
Keep the report under ~500 words outside quoted log lines and metric values. Evidence is what
earns length; narration is not. Omit a section that has nothing to report rather than padding it.
Incident Summary
- What happened, when it started, and estimated user/system impact
Timeline
- Chronological events: deploys, config changes, error onset, monitor triggers
Root Cause (High / Medium / Low confidence)
- Primary cause with supporting evidence from logs, metrics, and traces
Evidence
- Key log snippets showing the error pattern
- Metric anomalies (e.g., "error rate jumped from 0.1% to 12% at 14:32 UTC")
- Relevant trace IDs for further inspection
Next Steps
- Immediate remediation actions (rollback, config change, scaling)
- Follow-up investigation items if root cause is uncertain
- Suggested post-mortem tasks
Parameters
| Parameter |
Description |
Default |
service: |
Datadog service name to investigate |
(prompt if blank) |
since: |
Start of incident window (ISO 8601 or relative: 1h) |
1h ago |
until: |
End of incident window |
now |
error: |
Specific error message or alert name to focus on |
(optional) |
env: |
Deployment environment (prod, staging) |
prod |
Usage
Quick triage from an alert:
"Investigate the high error rate alert on the swap-router service"
With explicit time range:
"Root cause analysis for production error since:2024-01-15T14:00:00Z until:2024-01-15T15:30:00Z service:api-gateway"
Post-incident analysis:
"What caused the outage on the pricing service around 2pm yesterday?"
Scoped to a specific error:
"Triage production error 'upstream connect error or disconnect/reset before headers' on service:routing env:prod"
Notes
- Requires Datadog MCP server configured in the uniswap-integrations plugin
- If traces are unavailable, the investigation focuses on logs and metrics
- Use
env:staging to investigate pre-production incidents before they reach prod
- Confidence level in the root cause reflects evidence quality — "Low" means further investigation is recommended before acting
1---2name: investigate-incident3description: Investigate production incidents using Datadog. Use when user says "what's causing this alert", "investigate production incident", "root cause analysis for production error", "debug production issue", "what caused this outage", "analyze production logs", "triage production error", "why is the service down", "production is throwing errors", or "help me understand this incident". Always use this skill when diagnosing production issues, on-call alerts, or service degradations that require querying logs, metrics, or traces.4---56# Incident Investigator78Investigate production incidents by querying Datadog for logs, metrics, and traces, then synthesizing findings into a structured root-cause report.910## When to Activate1112- On-call alert triggered and developer needs to understand what's happening13- Service degradation, error spike, or latency increase detected14- User asks "what's causing this?" or "why is X failing in production?"15- Post-incident root cause analysis needed16- Monitoring dashboard shows anomaly requiring investigation1718## Steps19201. **Gather incident context** — confirm the affected service, error message or alert name, and approximate start time. If the user hasn't provided a time range, default to the last hour. Ask for any missing critical context before querying.21222. **Query logs** — search Datadog logs for error patterns in the incident window. Filter by service name and `status:error`. Surface the top recurring error messages and stack traces. Note frequency and distribution over time to establish whether the issue is steady-state or spiking.23243. **Check metrics and monitors** — retrieve error rate, latency (p50/p95/p99), and saturation metrics (CPU, memory, throughput) for the affected service. Check for triggered monitors around the incident start time to correlate alert onset with metric changes.25264. **Pull traces** — look up traces for failing requests to identify where in the call chain the failure originates. Note slow spans, timeouts, and downstream service dependencies that may indicate the root cause location.27285. **Correlate events** — search for deployment events, configuration changes, or upstream service issues in the same window. A deploy or config push immediately preceding the error onset is strong evidence of causation.29306. **Generate report** — synthesize findings into a structured incident report with timeline, root cause, confidence level, and actionable remediation steps.3132## Output Format3334Keep the report under ~500 words outside quoted log lines and metric values. Evidence is what35earns length; narration is not. Omit a section that has nothing to report rather than padding it.3637**Incident Summary**3839- What happened, when it started, and estimated user/system impact4041**Timeline**4243- Chronological events: deploys, config changes, error onset, monitor triggers4445**Root Cause** _(High / Medium / Low confidence)_4647- Primary cause with supporting evidence from logs, metrics, and traces4849**Evidence**5051- Key log snippets showing the error pattern52- Metric anomalies (e.g., "error rate jumped from 0.1% to 12% at 14:32 UTC")53- Relevant trace IDs for further inspection5455**Next Steps**5657- Immediate remediation actions (rollback, config change, scaling)58- Follow-up investigation items if root cause is uncertain59- Suggested post-mortem tasks6061## Parameters6263| Parameter | Description | Default |64| ---------- | ----------------------------------------------------- | ----------------- |65| `service:` | Datadog service name to investigate | (prompt if blank) |66| `since:` | Start of incident window (ISO 8601 or relative: `1h`) | `1h ago` |67| `until:` | End of incident window | `now` |68| `error:` | Specific error message or alert name to focus on | (optional) |69| `env:` | Deployment environment (`prod`, `staging`) | `prod` |7071## Usage7273Quick triage from an alert:7475```76"Investigate the high error rate alert on the swap-router service"77```7879With explicit time range:8081```82"Root cause analysis for production error since:2024-01-15T14:00:00Z until:2024-01-15T15:30:00Z service:api-gateway"83```8485Post-incident analysis:8687```88"What caused the outage on the pricing service around 2pm yesterday?"89```9091Scoped to a specific error:9293```94"Triage production error 'upstream connect error or disconnect/reset before headers' on service:routing env:prod"95```9697## Notes9899- Requires Datadog MCP server configured in the uniswap-integrations plugin100- If traces are unavailable, the investigation focuses on logs and metrics101- Use `env:staging` to investigate pre-production incidents before they reach prod102- Confidence level in the root cause reflects evidence quality — "Low" means further investigation is recommended before acting
uniswap