CI Incident

CI Incident Response

Guides a responder through a CI incident in camunda/camunda. Pulls live state from incident.io, context from the incident's Slack channel, the matching runbook from docs/monorepo-docs/ci-runbooks.md, and the process from docs/monorepo-docs/processes.md.

Prerequisites

• incident.io MCP — required. Verify the mcp__incident-io__incident_show tool is available. If not, tell the user to configure the incident.io MCP server and stop.
• Slack MCP — preferred but optional. If mcp__*__slack_read_channel is not available, warn the user that Slack context will be skipped and continue.

Throughout this skill, MCP tools are referenced by their short verb (incident_show, slack_read_channel); the actual identifier depends on which MCP server is configured.

Scope

Full incidents only. If the user passes an alert ID (or anything that isn't an incident), stop and ask for the incident ID.

Procedure

Step 1 — Parse the incident ID

Read the incident ID from the user's request. If it is missing, ask the user for it before proceeding. Do not guess from recent context.

Step 2 — Pull incident state

Call incident_show with the ID and include: ["investigation", "postmortem", "roles", "custom_fields"].

Extract and remember:

• Title, severity, status, opened-at
• Role assignments (lead, comms, etc.)
• Active alert name(s)
• Incident's Slack channel reference (channel ID or name)
• Any existing investigation / postmortem content

Step 3 — Pull Slack context

If the Slack MCP is available, read recent messages from the incident's Slack channel (slack_read_channel, ~50 most recent). Summarize for yourself:

• What has already been tried
• Who is actively engaged
• Timestamp of the last status update
• Any blockers or open questions raised

If the Slack MCP is not available, note this in the brief and continue.

Step 4 — Search for similar incidents

Check for a recurring pattern before matching a runbook:

• incident_list with query: "<title or active alert name>", scoped to resolved/closed incidents, excluding the current one.
• gh issue list --repo camunda/camunda --search "<alert or job/workflow name>" for existing tracking (flake tickets, deferred fixes, runbook follow-ups).

If found, pull its postmortem (incident_show with include: ["postmortem", "investigation"]) and note the prior root cause, fix, and whether its follow-up was completed. If the same root cause recurs despite that fix, flag it explicitly — the fix didn't hold.

Step 5 — Match a runbook

Read docs/monorepo-docs/ci-runbooks.md.

Match the incident title and active alert name(s) against the ### headings under both Alert Runbooks and Incident Runbooks sections. Fuzzy match is fine (e.g. alert "Merge Queue High Failure Rate" → runbook heading of the same name).

• One match: load its Troubleshooting and Solutions subsections.
• Multiple matches: list them and ask the user which applies.
• No match: list the available runbook headings and ask which applies, or confirm this is a novel incident with no runbook.

Step 6 — Locate the current process step

Read the CI Incident Management section of docs/monorepo-docs/processes.md.

Based on incident status + Slack history, determine which step the incident is in:

1. Identification
2. Response
3. Follow-Up

This tells you what action is expected next (e.g. assign roles, post update, create follow-ups).

Step 7 — Brief the responder

In a single concise message, give the responder:

• State: title, severity, status, lead, opened-at
• Slack TL;DR: what's been tried, who's active, last update time
• Similar incidents: past occurrences found (with what fixed it last time), or none found
• Runbook: matched heading + the Troubleshooting steps
• Process step: which of the three steps the incident is in, and what's expected
• Next action: one concrete recommendation

Then wait for the responder to direct the next move.

Step 8 — Drive the response

Work through the runbook with the responder. Prioritize mitigation (stopping the bleeding) over root causing and resolution. Use relevant CI skills from this repository, e.g. for troubleshooting. Specific rules:

• Status updates: always use incident_update from the incident.io MCP. Do not post directly to the incident's Slack channel — incident.io syndicates updates to Slack.
• Slack MCP usage: read-only by default. Only post via Slack MCP if the responder explicitly asks to message a channel other than the incident channel (e.g. #top-monorepo-ci).
• Confirm wording: never send an incident_update without showing the draft to the responder and getting explicit confirmation.
• Don't change state silently: never change severity, status, or role assignments without the responder explicitly instructing it.
• Don't stop at mitigation — finish root causing: alerts can self-resolve mid-investigation because of how failure-event alerting is implemented; a quiet alert doesn't mean the incident is understood. Keep investigating until the root cause is known.
• Keep incident open until all attached alerts are resolved: check for attached alerts that are still firing, and reject resolving or closing the incident if any are.

Step 9 — Wrap-up

When the incident moves to resolved or closed:

• Re-read the Follow-Up subsection of processes.md.
• Prompt the responder to create follow-ups via follow_up_create for each action item raised during response (postmortem tasks, runbook updates, fixes deferred during the incident).
• Offer to draft each follow-up's title and description from the Slack history and runbook gaps.

Guardrails

• Do not invent runbook content. If no runbook matches, say so.
• Do not assume the responder has tried something just because it's in the runbook — ask.
• If the runbook references commands or links that no longer exist in the repo, flag it as a follow-up rather than improvising.