Pranavj17 Incident Triage
Use this skill when the user reports active failure. Goal: in under 5 tool calls, identify which service is bleeding, what pattern of error, and whether a recent deploy is responsible.
Playbook
Step 1 — Quantify the spike
Call aggregate_logs:
query: "logger_level:error"
field: "service"
rangeSeconds: 1800
Response gives you error counts per service for the last 30 minutes. The top 1–3 services are your suspects.
Step 2 — Baseline against the previous window
Run the same aggregate_logs query with from / to shifted back by 30 minutes (the previous half-hour). Compare:
- A service that went from low/zero to high count → new failure (likely a deploy or upstream).
- A service that was already high → ongoing issue, not the cause of this alert.
- All services up uniformly → likely a shared dependency (DB, network, auth) — investigate that service first.
Step 3 — Drill into the top offender
For the highest-delta service, call search_logs_relative:
query: "logger_level:error AND service:<top-offender>"
rangeSeconds: 1800
limit: 50
Read the sample error messages. Look for:
- One dominant error pattern → the bug is localized; capture the error text.
- A single trace_id repeating → hand off to
trace-debuggingskill with that trace_id. - Many distinct errors → likely a process crash / restart loop; check timestamps for clustering.
Step 4 — Did a deploy cause this?
If the spike has a clear start time, search a small window around it for startup markers:
query: '("Starting" OR "Booting" OR "version") AND service:<top-offender>'
from: <spike start - 5min>
to: <spike start + 1min>
limit: 20
A startup log line within 5 minutes before the spike is a strong deploy signal.
Step 5 — Pod / instance correlation
If errors cluster on one pod, narrow further:
aggregate_logs
query: "logger_level:error AND service:<top-offender>"
field: "pod"
rangeSeconds: 1800
A single bad pod → restart that pod (action for the user), no code change needed. Errors spread evenly across pods → real code or config issue.
Stop conditions
- Localized to one service + one error pattern + recent deploy → Done. Report findings and suggest a rollback or a targeted code-look.
- Multiple services failing simultaneously → Likely upstream (DB, queue, auth service). Investigate the shared dependency first; don't keep drilling into individual services.
- All errors in one pod → Report pod, suggest restart, stop investigating.
- Pattern is unclear after 5 tool calls → Surface what you have and ask the user what they know about recent changes; don't keep guessing.
What NOT to do
- Don't pull raw logs without aggregating first — you'll drown in noise.
- Don't filter to
erroronly on the first call and misswarnprecursors. After identifying the offender service, broaden to all levels in step 3 for a few sample windows. - Don't ignore baseline data. "Errors are high" is meaningless without "errors were low 30 minutes ago."