Performance Incident Response
Purpose
Restore the user objective while preserving enough evidence to learn. Triage, mitigation and root-
cause analysis are distinct phases; a successful rollback proves a useful association, not the full
causal chain.
Incident contract
Record incident commander, technical lead, scribe, communications owner, affected user journey and
SLO/burn, start/detection/acknowledgement/mitigation/recovery timestamps with sources, recent changes,
evidence budget, safe actions, rollback authority and the next decision time.
Reuse the active incident process and existing authorization. One person may cover several
roles initially; missing role assignments or template fields must not delay an authorized
urgent mitigation. Keep impact/start-time uncertainty explicit rather than inventing timestamps.
Workflow
- Declare impact and scope from user-facing evidence. Separate missing telemetry from zero impact.
- Preserve an append-only timestamped change/evidence ledger. Capture volatile, cheap evidence
before destructive actions when feasible within the evidence budget. If ongoing impact
requires immediate mitigation, record what could not be captured and proceed; expensive
diagnostics must not become prerequisites (
incident-evidence-capture).
- Build a small hypothesis table from USE/resource, request-path and recent-change evidence. Assign
one discriminating check per hypothesis. This can run alongside mitigation and need not
establish root cause before applying a supported recovery action.
- Choose one bounded mitigation with predicted signal, blast radius, abort condition and rollback.
- Validate recovery against the user SLI, goodput, backlog/drain and resource guardrails; green CPU
or one cleared alert is insufficient.
- Hand off follow-up investigation with preserved artifacts. Run a blameless postmortem after the
system is stable.
Rules
- Prioritize restoring the user objective over exploratory tuning. Prefer reversible mitigation;
a targeted configuration/code correction may be necessary. State evidence, risk and validation
instead of treating either a familiar rollback or an optimization label as proof of safety.
- Preserve exact command output, timestamps, JDK/process/container identity and failures. An empty
profiler result is evidence about the tool path, not proof that the mechanism is absent.
- Coordinate interacting changes to the same affected scope, normally one material change at a
time. Independent authorized work can proceed under named owners. When urgent impact requires
bundled actions, record their timing and the resulting limits on causal attribution.
- Use separate clocks: detection, acknowledgement, first material mitigation and sustained recovery.
Define each metric before comparing incidents.
- A postmortem maps contributing conditions and failed defenses, not one linear “five whys” chain.
- Every action item has an owner, deadline, observable acceptance criterion and the control layer it
changes. “Be more careful” is not an action.
Handoff the current impact, actions and actual outcomes, remaining uncertainty, artifact locations,
active owners and next decision/check. A mitigation attempted, a workload temporarily quiet and
a confirmed sustained recovery are different states.
References
- War room and postmortem — read during response setup,
mitigation handoff or postmortem preparation.
1---2name: performance-incident-response3description: Coordinating a production performance incident from impact declaration through evidence-preserving triage, one-change mitigation, recovery validation and a blameless causal postmortem. Use when a latency, throughput, saturation or resource regression requires a war room; when responders are changing JVM flags before preserving evidence; or when MTTD, mitigation time and recovery time are being conflated. Evidence acquisition belongs to incident-evidence-capture; technical diagnosis to performance-methodology; this skill owns response sequencing and decision records.4---56# Performance Incident Response78## Purpose910Restore the user objective while preserving enough evidence to learn. Triage, mitigation and root-11cause analysis are distinct phases; a successful rollback proves a useful association, not the full12causal chain.1314## Incident contract1516Record incident commander, technical lead, scribe, communications owner, affected user journey and17SLO/burn, start/detection/acknowledgement/mitigation/recovery timestamps with sources, recent changes,18evidence budget, safe actions, rollback authority and the next decision time.19Reuse the active incident process and existing authorization. One person may cover several20roles initially; missing role assignments or template fields must not delay an authorized21urgent mitigation. Keep impact/start-time uncertainty explicit rather than inventing timestamps.2223## Workflow24251. Declare impact and scope from user-facing evidence. Separate missing telemetry from zero impact.262. Preserve an append-only timestamped change/evidence ledger. Capture volatile, cheap evidence27 before destructive actions when feasible within the evidence budget. If ongoing impact28 requires immediate mitigation, record what could not be captured and proceed; expensive29 diagnostics must not become prerequisites (`incident-evidence-capture`).303. Build a small hypothesis table from USE/resource, request-path and recent-change evidence. Assign31 one discriminating check per hypothesis. This can run alongside mitigation and need not32 establish root cause before applying a supported recovery action.334. Choose one bounded mitigation with predicted signal, blast radius, abort condition and rollback.345. Validate recovery against the user SLI, goodput, backlog/drain and resource guardrails; green CPU35 or one cleared alert is insufficient.366. Hand off follow-up investigation with preserved artifacts. Run a blameless postmortem after the37 system is stable.3839## Rules4041- Prioritize restoring the user objective over exploratory tuning. Prefer reversible mitigation;42 a targeted configuration/code correction may be necessary. State evidence, risk and validation43 instead of treating either a familiar rollback or an optimization label as proof of safety.44- Preserve exact command output, timestamps, JDK/process/container identity and failures. An empty45 profiler result is evidence about the tool path, not proof that the mechanism is absent.46- Coordinate interacting changes to the same affected scope, normally one material change at a47 time. Independent authorized work can proceed under named owners. When urgent impact requires48 bundled actions, record their timing and the resulting limits on causal attribution.49- Use separate clocks: detection, acknowledgement, first material mitigation and sustained recovery.50 Define each metric before comparing incidents.51- A postmortem maps contributing conditions and failed defenses, not one linear “five whys” chain.52- Every action item has an owner, deadline, observable acceptance criterion and the control layer it53 changes. “Be more careful” is not an action.5455Handoff the current impact, actions and actual outcomes, remaining uncertainty, artifact locations,56active owners and next decision/check. A mitigation attempted, a workload temporarily quiet and57a confirmed sustained recovery are different states.5859## References6061- [War room and postmortem](references/war-room-and-postmortem.md) — read during response setup,62 mitigation handoff or postmortem preparation.
robsonkades