# Rollback Runbook

> Plan and execute a safe rollback (or roll-forward) when a release misbehaves, with a documented sequence and an explicit decision point on rollback vs fix-forward.

- Skill: `poly-gents/rollback-runbook` (Agent Skill, multi-file: 2 files)
- Install (CLI): `npx skillmds@latest add poly-gents/rollback-runbook`
- Raw SKILL.md: https://api.skillmd.com/api/skills/poly-gents/rollback-runbook/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: poly-gents (https://skillmd.com/u/poly-gents)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/poly-gents/rollback-runbook

---


# Rollback runbook

A rollback is a live operation, not a vibe. Run it from a runbook, in this order, with a single named owner.

## When to use

- A release is causing customer-visible incorrect behaviour.
- A release is causing observability red (error rate spike, latency spike).
- A release reveals a security issue that can't be hotfixed in minutes.

## The decision (rollback vs fix-forward)

- **Rollback** when:
  - The bug is broad (affects > 1% of users or any paying customer).
  - The fix is non-trivial (> 30 minutes to write).
  - The change has a clean revert (no destructive migration).
- **Fix-forward** when:
  - The bug is narrow (a single feature flag can dark-launch it off).
  - The fix is a one-line change with low risk.
  - Rollback would itself break something (migration ran, data shape changed, external state mutated).

## Rollback sequence

1. **Declare.** Open the rollback in the incident thread; name the owner.
2. **Freeze.** Stop additional deploys to the same service.
3. **Verify the rollback target.** What commit are you reverting to? Has it itself been verified?
4. **Execute.** Use the deploy tool's revert path (`github-actions` rerun, kubectl rollout undo, blue/green flip).
5. **Watch.** For 15 minutes after the rollback, monitor `datadog` / `grafana` for the same SLIs that triggered the rollback.
6. **Confirm.** When SLIs return to baseline, post a confirmation in the thread.
7. **Hand off to postmortem.** Run `incident-postmortem-loop`.

## Special cases

- **DB migration ran.** Rollback the code, leave the schema, file an urgent task for a fix-forward migration. Never run a destructive down migration on production data without a backup verified within the last 24 hours.
- **Third-party config flipped.** The revert is in the third-party UI; do that step first, then the code revert.
- **Feature flag exists.** Flip the flag off before considering the full rollback; that's usually enough.

## Anti-patterns

- Rolling back to "the last known good commit" without checking which commit that actually is.
- Skipping the 15-minute watch.
- Doing the rollback alone without naming an owner in the thread.