Reactive and read-only; covers every new or still-unresolved alert in
{{sentry_project}} found in a sweep, nothing carried over between sweeps.
Handle each alert as an independent unit: a failure or inconclusive diagnosis
on one alert never blocks triage of the others found in the same sweep.
Step 1 — Pull the alert from Sentry (read-only)
Via the Sentry connector, for every alert in {{sentry_project}} that is new or
still unresolved since the last sweep:
- The full stack trace and exception type.
- Event frequency (spike vs. steady trickle) and the first-seen timestamp.
- Whether Sentry has already grouped it under an existing, previously-seen
issue fingerprint.
Step 2 — Correlate the logs (read-only)
Via the Datadog connector, pull the log lines in a window around the
first-seen timestamp (default: 10 minutes before to 5 minutes after), scoped
to the service(s) named in the stack trace. Line the log entries up against
the trace to confirm the failure mode — a matching upstream timeout, a bad
payload, a dependency erroring — rather than trusting the exception message
alone.
Step 3 — Check recent deploys on GitHub (read-only)
gh pr list --repo {{target_repo}} --state merged --limit 20 \
--json number,title,mergedAt,url,author \
--search "merged:>=<first-seen-minus-2h>"
git log --since="<first-seen-minus-2h>" --until="<first-seen>" --oneline
Rank candidate deploys by how closely they precede the first-seen timestamp
and whether the changed files touch the failing service/module. Treat the
closest matching deploy as the suspect, not a certainty — say so in the
diagnosis.
Step 4 — Classify severity and benign vs. real
| Signal |
Classification |
| Matches a known flaky/noisy pattern (repeated transient timeout, third-party rate limit, a fingerprint seen before with no user impact) |
Benign — close with reasoning, no page |
| New error type, affects one user or a narrow code path, no matching deploy |
Real, low severity — post diagnosis, no page unless it recurs |
| Spike in frequency, matches a recent deploy, or affects a shared/critical path |
Real, elevated severity |
Marked fatal/critical in Sentry, or severity at or above {{page_severity_threshold}} |
Page immediately regardless of diagnosis confidence |
| Trace, logs, or deploy history are inconclusive after Steps 1–3 |
Unresolved — page, diagnosis attached as partial context |
Step 5 — Post the first-pass diagnosis
Post one message per alert to {{incident_channel}}: the error and where it's
from, the first-seen time and frequency, the correlated log lines, the
suspect deploy (PR/commit + author, or "no matching deploy found"), and the
severity classification from Step 4.
Step 6 — Page or close
- Page — at or above {{page_severity_threshold}}, or unresolved per Step
4: @-mention the on-call rotation in {{incident_channel}} on the same
message, with the diagnosis attached as context, not a substitute for
looking at it.
- Close — benign or low severity with a confident diagnosis: mark it
resolved in the post with the reasoning, no page.
Nothing else leaves the sandbox — no ticket creation, no deploy, no rollback,
no config change.
1---2name: incident-triage3description: On-call alert triage runbook for {{sentry_project}}. Pulls the Sentry stack trace, correlates Datadog logs around the spike, checks {{target_repo}} for the deploy that shipped just before, classifies severity against known-noisy patterns, posts a first-pass diagnosis to {{incident_channel}}, and pages a human only when severity is at or above {{page_severity_threshold}} or the alert can't be resolved.4---56<skill name="incident-triage">78<overview>9Do the mechanical first minutes of incident response before a human is woken10up. A sweep spawns a fresh, read-only session; this skill turns a raw alert11into a trace, a suspect deploy, a correlated log window, and a severity call —12then decides whether that diagnosis is enough to close the loop or whether it13has to page.1415Reactive and read-only; covers every new or still-unresolved alert in16{{sentry_project}} found in a sweep, nothing carried over between sweeps.17Handle each alert as an independent unit: a failure or inconclusive diagnosis18on one alert never blocks triage of the others found in the same sweep.19</overview>2021<when-to-load>22- The alert-triage sweep fires on its cadence.23- A human asks the agent to re-triage a specific alert or explain why one was24 paged (or wasn't).25</when-to-load>2627<workflow>2829## Step 1 — Pull the alert from Sentry (read-only)3031Via the Sentry connector, for every alert in {{sentry_project}} that is new or32still unresolved since the last sweep:3334- The full stack trace and exception type.35- Event frequency (spike vs. steady trickle) and the first-seen timestamp.36- Whether Sentry has already grouped it under an existing, previously-seen37 issue fingerprint.3839## Step 2 — Correlate the logs (read-only)4041Via the Datadog connector, pull the log lines in a window around the42first-seen timestamp (default: 10 minutes before to 5 minutes after), scoped43to the service(s) named in the stack trace. Line the log entries up against44the trace to confirm the failure mode — a matching upstream timeout, a bad45payload, a dependency erroring — rather than trusting the exception message46alone.4748## Step 3 — Check recent deploys on GitHub (read-only)4950```sh51gh pr list --repo {{target_repo}} --state merged --limit 20 \52 --json number,title,mergedAt,url,author \53 --search "merged:>=<first-seen-minus-2h>"54git log --since="<first-seen-minus-2h>" --until="<first-seen>" --oneline55```5657Rank candidate deploys by how closely they precede the first-seen timestamp58and whether the changed files touch the failing service/module. Treat the59closest matching deploy as the suspect, not a certainty — say so in the60diagnosis.6162## Step 4 — Classify severity and benign vs. real6364| Signal | Classification |65|---|---|66| Matches a known flaky/noisy pattern (repeated transient timeout, third-party rate limit, a fingerprint seen before with no user impact) | Benign — close with reasoning, no page |67| New error type, affects one user or a narrow code path, no matching deploy | Real, low severity — post diagnosis, no page unless it recurs |68| Spike in frequency, matches a recent deploy, or affects a shared/critical path | Real, elevated severity |69| Marked `fatal`/`critical` in Sentry, or severity at or above {{page_severity_threshold}} | Page immediately regardless of diagnosis confidence |70| Trace, logs, or deploy history are inconclusive after Steps 1–3 | Unresolved — page, diagnosis attached as partial context |7172## Step 5 — Post the first-pass diagnosis7374Post one message per alert to {{incident_channel}}: the error and where it's75from, the first-seen time and frequency, the correlated log lines, the76suspect deploy (PR/commit + author, or "no matching deploy found"), and the77severity classification from Step 4.7879## Step 6 — Page or close8081- **Page** — at or above {{page_severity_threshold}}, or unresolved per Step82 4: @-mention the on-call rotation in {{incident_channel}} on the same83 message, with the diagnosis attached as context, not a substitute for84 looking at it.85- **Close** — benign or low severity with a confident diagnosis: mark it86 resolved in the post with the reasoning, no page.8788Nothing else leaves the sandbox — no ticket creation, no deploy, no rollback,89no config change.9091</workflow>9293<guardrails>94- **Read-only across every connector.** Sentry, Datadog, and GitHub are95 investigated, never written to — no comments, no resolves-in-Sentry, no96 commits, no deploys, no rollbacks.97- **One diagnosis per alert.** Post exactly one message to {{incident_channel}}98 per alert per sweep; don't re-post an alert already diagnosed this sweep.99- **Independent per alert.** Treat every alert found in a sweep as its own100 case; a failure or inconclusive result investigating one never stops101 triage of the others found in the same sweep.102- **Page on doubt.** An inconclusive diagnosis pages a human — never guess103 "probably fine" into silence on anything ambiguous.104- **Severity floor is non-negotiable.** Anything at or above105 {{page_severity_threshold}} pages regardless of how confident the diagnosis106 is.107- **Scoped secrets.** Sentry, Datadog, and GitHub credentials are brokered108 server-side through connectors and the GitHub token is injected at runtime; scoped to this agent's grant.109- **No memory required between sweeps.** Each sweep is a fresh session — the110 "new or unresolved since last check" filter comes from Sentry's own issue111 state, not an agent-side ledger.112</guardrails>113114</skill>
kortix-ai