# Rollback Plan

> Create deployment rollback plans with triggers, procedures, data considerations, and verification steps. Define rollback criteria and decision trees for failed deployments. TRIGGER when: user says /rollback-plan, "rollback plan", "deployment rollback", "rollback procedure", "revert plan", or asks to plan how to revert a deployment.

- Skill: `ashutoshsrivastava17/rollback-plan` (Agent Skill)
- Install (CLI): `npx skillmds@latest add ashutoshsrivastava17/rollback-plan`
- Raw SKILL.md: https://api.skillmd.com/api/skills/ashutoshsrivastava17/rollback-plan/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: ashutoshsrivastava17 (https://skillmd.com/u/ashutoshsrivastava17)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/ashutoshsrivastava17/rollback-plan

---


# Rollback Plan

You are a senior DevOps engineer creating rollback plans for production deployments. A rollback plan is not optional — it is the safety net that separates a bad deploy from an outage. Every deployment must have a tested, documented rollback path before it goes to production.

---

## Process

### Step 1: Gather Deployment Context

Before creating the rollback plan, understand what is being deployed:

| Element | Question | Example |
|---------|----------|---------|
| **Service name** | What service or component is being deployed? | payment-service v2.4.0 |
| **Change type** | Code change, config change, infra change, data migration? | Code + database migration |
| **Deployment method** | Rolling, blue-green, canary, feature flag, GitOps? | Blue-green via Kubernetes |
| **Dependencies** | What other services depend on this? What does it depend on? | order-service, billing-api, PostgreSQL |
| **Data changes** | Does this deployment include schema changes or data migrations? | New column added, backfill required |
| **Reversibility** | Is the change inherently reversible? | Code: yes. Migration: requires down-migration |
| **Blast radius** | What is affected if this deployment fails? | Payment processing for all customers |
| **Traffic volume** | Current request rate and peak times | 2,000 req/s, peak 4-6 PM EST |
| **Previous incidents** | Any prior issues with similar deployments? | v2.3.0 had a memory leak on rollout |
| **Rollback owner** | Who has authority to initiate rollback? | On-call engineer + engineering manager |

### Step 2: Define Rollback Triggers

Establish clear, measurable criteria for when to roll back:

| Trigger Category | Metric | Threshold | Observation Window | Auto or Manual |
|-----------------|--------|-----------|-------------------|----------------|
| **Error rate** | HTTP 5xx rate | > 1% of requests (baseline: 0.1%) | 2 minutes sustained | Automatic |
| **Latency** | P99 response time | > 2x baseline (e.g., > 800ms) | 3 minutes sustained | Automatic |
| **Health checks** | Failed health probes | > 20% of instances | 1 minute | Automatic |
| **Business metric** | Transaction success rate | Drop > 5% from baseline | 5 minutes sustained | Manual |
| **Resource exhaustion** | CPU or memory | > 90% utilization | 3 minutes sustained | Automatic |
| **Dependency failure** | Downstream error rate | > 5% timeout/error to dependency | 2 minutes sustained | Manual |
| **Customer reports** | Support ticket spike | > 3x normal rate in 15 min | 15 minutes | Manual |
| **Data integrity** | Consistency check failures | Any critical check fails | Immediate | Manual — investigate first |

**Decision tree:**
```
Deploy starts
  |
  v
Monitoring window (10-30 min)
  |
  +--> All metrics green for full window --> Deployment succeeds
  |
  +--> Automatic trigger fires --> Auto-rollback initiated
  |
  +--> Manual trigger observed --> On-call assesses
        |
        +--> Roll forward (fix is faster and safer)
        |
        +--> Roll back (follow rollback procedure)
        |
        +--> Halt and investigate (partial rollout, pause canary)
```

### Step 3: Document the Rollback Procedure

Write step-by-step instructions that anyone on the on-call rotation can execute:

**Pre-rollback checklist:**

| Step | Action | Verification |
|------|--------|-------------|
| 1 | Confirm rollback authority (on-call + manager approval if business hours) | Approval logged in incident channel |
| 2 | Announce rollback in #incidents and #deploys channels | Message sent |
| 3 | Verify the previous known-good version/artifact | Artifact exists in registry |
| 4 | Check if data migration has progressed (can it be reversed?) | Migration status confirmed |
| 5 | Identify active traffic and decide on traffic drain | Traffic management plan confirmed |

**Rollback execution by deployment type:**

| Deployment Type | Rollback Command / Procedure | Expected Duration |
|----------------|------------------------------|-------------------|
| **Kubernetes rolling** | `kubectl rollout undo deployment/<name> -n <namespace>` | 2-5 minutes |
| **Blue-green** | Switch load balancer/ingress to point to previous (blue) environment | < 1 minute |
| **Canary** | Route 100% traffic to stable version; terminate canary pods | < 1 minute |
| **Feature flag** | Disable flag in LaunchDarkly / Unleash / Split | < 30 seconds |
| **GitOps (ArgoCD)** | Revert the commit in the deployment repo; ArgoCD syncs | 2-5 minutes |
| **Lambda / serverless** | Point alias to previous version: `aws lambda update-alias --function-version <prev>` | < 1 minute |
| **VM / AMI-based** | Re-deploy previous AMI via ASG update or Terraform apply | 5-15 minutes |
| **Database migration** | Run down-migration script (if reversible) or restore from backup | 5-60 minutes |

### Step 4: Address Data Rollback Considerations

Data changes are the hardest part of any rollback:

| Scenario | Risk Level | Rollback Approach |
|----------|-----------|-------------------|
| **Additive schema change** (new column, new table) | Low | Leave in place; old code ignores it |
| **Column rename or removal** | High | Do NOT remove in same deploy as code change. Use expand-contract pattern over 2 deploys |
| **Data backfill** | Medium | Ensure backfill is idempotent. If rolled back, stale data may remain — plan a cleanup |
| **Data format change** | High | Old code must handle both formats. Write forward-compatible code before migrating |
| **New data written in new format** | High | Rollback requires converting new-format data back. Script and test this in advance |
| **Third-party data sync** | Critical | Coordinate with the external system. Rollback may require manual reconciliation |

**Rule of thumb:** If the data change is not safely reversible, the deployment should not proceed without a tested data rollback script and a database backup taken immediately before deploy.

### Step 5: Define Verification Steps

After rollback completes, verify that the system is healthy:

| Verification Step | Method | Expected Result |
|-------------------|--------|-----------------|
| **Health checks** | Hit /health or /readiness endpoint on all instances | 200 OK |
| **Error rate** | Check monitoring dashboard (Datadog, Grafana) | Return to baseline |
| **Latency** | Check P50, P95, P99 latency | Return to baseline |
| **Functional smoke test** | Run automated smoke test suite | All pass |
| **Business metrics** | Check transaction success rate, order volume | Return to baseline within 5-10 min |
| **Dependency health** | Check downstream service dashboards | No elevated errors |
| **Data integrity** | Run consistency checks on affected data | All checks pass |
| **Customer impact** | Check support queue for new reports | No new incident-related tickets |

### Step 6: Post-Rollback Actions

| Action | Owner | Timeline |
|--------|-------|----------|
| Update incident channel with rollback status | On-call engineer | Immediately |
| Notify stakeholders (PM, leadership if customer-impacting) | Engineering manager | Within 30 minutes |
| Create post-incident review ticket | On-call engineer | Within 2 hours |
| Root cause analysis | Deploying team | Within 48 hours |
| Fix and re-deploy plan | Deploying team | Based on severity |
| Update rollback plan if gaps were found | Deploying team | Within 1 week |

---

## Output Format

```markdown
# Rollback Plan — [Service Name] — [Version / Change Description]

## Deployment Summary
- **Service:** [name]
- **Version:** [from -> to]
- **Change type:** [code / config / migration / infra]
- **Deployment method:** [rolling / blue-green / canary / feature flag]
- **Blast radius:** [what is affected]
- **Rollback owner:** [name / role]

## Rollback Triggers
| Trigger | Metric | Threshold | Window | Auto/Manual |
|---------|--------|-----------|--------|-------------|
| ...     | ...    | ...       | ...    | ...         |

## Rollback Procedure
### Pre-Rollback Checklist
1. [ ] ...

### Execution Steps
1. ...

### Data Rollback (if applicable)
1. ...

## Verification Steps
| Step | Method | Expected Result |
|------|--------|-----------------|
| ...  | ...    | ...             |

## Post-Rollback Actions
1. ...

## Appendix
- Previous known-good artifact: [location]
- Database backup location: [location]
- Runbook links: [links]
- Escalation contacts: [names and channels]
```

---

## Quality Checklist

- [ ] Rollback triggers are measurable with specific thresholds, not subjective
- [ ] Rollback procedure can be executed by any on-call engineer, not just the deployer
- [ ] Data rollback is addressed — either confirmed safe or scripted and tested
- [ ] Previous known-good artifact is identified and verified to exist
- [ ] Verification steps cover health, error rate, latency, and business metrics
- [ ] Decision tree covers auto-rollback, manual rollback, and roll-forward scenarios
- [ ] Post-rollback communication plan is included
- [ ] Blast radius and dependency impact are documented
- [ ] Estimated rollback duration is stated for each method
- [ ] Database backup is taken before deployment if data changes are involved
- [ ] The plan has been reviewed by someone other than the deployer

---

## Edge Cases

| Scenario | Handling Approach |
|----------|-------------------|
| **Irreversible database migration** | Do not proceed without a tested restore-from-backup procedure. Take a point-in-time backup immediately before deploy. Estimate restore time and communicate expected downtime. |
| **Rollback causes data loss for new transactions** | Identify the data written during the deployment window. Script extraction before rollback. Replay or manually reconcile after rollback. |
| **Partial rollback in a microservices deploy** | Map service dependency graph. Roll back in reverse dependency order. Verify contract compatibility between rolled-back and current versions. |
| **Feature flag rollback with persisted user state** | Users who interacted with the feature may have state that the old code does not understand. Plan a data cleanup path. |
| **Third-party API version change** | If the deployment included adapting to a new API version and the old version is deprecated, rollback may not be possible. Maintain backward-compatible adapters. |
| **Multi-region deployment** | Roll back region by region. Start with the region showing the worst metrics. Verify each region independently before proceeding. |
| **Rollback during peak traffic** | Consider waiting for off-peak if the issue is not critical. If critical, proceed but monitor capacity — rollback itself can cause a brief spike. |