Create Rollback Plan

Create Rollback Plan

Overview

Create a comprehensive rollback plan for a deployment that covers trigger criteria, step-by-step procedures, data considerations, feature flag rollback, cache invalidation, communication protocols, and verification steps. A rollback plan must be written and tested before the deployment, not improvised during an incident.

Workflow

1. Read project context — Check .chalk/docs/engineering/ for:

   • Architecture and deployment topology docs (monolith vs. microservices, deploy targets, CDN setup)
   • Database migration patterns and tools
   • Feature flag infrastructure and current flag states
   • Previous rollback plans (to inherit project-specific procedures)
   • Release checklists for the current version
   • CI/CD pipeline configuration

2. Understand the deployment — From $ARGUMENTS and conversation:

   • What version is being deployed
   • What changes are included (features, migrations, config changes)
   • What is the deployment mechanism (blue-green, canary, rolling, big-bang)
   • What infrastructure is involved (servers, containers, serverless, CDN)
   • What database changes are included (if any)

3. Identify rollback risks — For this specific deployment:

   • Are there database migrations? Are they reversible?
   • Are there data format changes? Is old code compatible with new data?
   • Are there external API contract changes? Will consumers break?
   • Are there feature flags that can independently control new behavior?
   • Are there cache entries that will be stale after rollback?
   • Are there scheduled jobs or background workers affected?

4. Define trigger criteria — Specify concrete, measurable conditions that should trigger a rollback:

   • Error rate thresholds (absolute and relative to baseline)
   • Latency thresholds (P50, P95, P99)
   • Business metric thresholds (conversion rate, active users, revenue)
   • Critical flow failures (login, checkout, data persistence)
   • Manual trigger conditions (customer reports, support ticket spike)
   • Time-based triggers (if metrics do not stabilize within X minutes)

5. Write the rollback procedure — Step-by-step commands and actions:

   • Exact commands to execute the rollback (copy-pasteable)
   • Order of operations (which service first if multi-service)
   • Database rollback procedure (if migrations were run)
   • Feature flag changes (which flags to disable)
   • Cache and CDN invalidation steps
   • DNS or load balancer changes (if applicable)
   • Background job/worker considerations

6. Document data considerations — Critical for any deployment with data changes:

   • Can the migration be reversed without data loss?
   • Will data written by the new code be readable by the old code?
   • Are there irreversible data transformations?
   • What is the backup strategy and restore procedure?
   • How long does a data restore take?

7. Write the communication plan — Who to notify and when:

   • Engineering team (before rollback begins)
   • On-call engineer (if not already engaged)
   • Support team (to expect user reports)
   • Stakeholders (if user-visible impact exceeds threshold)
   • External consumers (if API changes are reverted)
   • Status page updates (if public-facing)

8. Define verification steps — How to confirm the rollback succeeded:

   • Health check endpoints to verify
   • Key user flows to test manually
   • Metrics to watch and expected values
   • Time to wait before declaring rollback complete
   • How to confirm no data corruption occurred

9. Determine the next file number — List files in .chalk/docs/engineering/ matching *_rollback_plan*. Find the highest number and increment by 1.

10. Write the plan — Save to .chalk/docs/engineering/<n>_rollback_plan_<version>.md.

11. Confirm — Present the plan with key highlights: trigger criteria summary, estimated rollback time, and any risks that cannot be mitigated by rollback alone.

Filename Convention

<number>_rollback_plan_<version>.md

Examples:

• 9_rollback_plan_v1_3_0.md
• 15_rollback_plan_v2_0_0.md

Rollback Plan Format

# Rollback Plan: <version>

Last updated: <YYYY-MM-DD>
**Deployment date**: <date or TBD>
**Rollback owner**: <name or role>
**Estimated rollback time**: <minutes>
**Risk level**: Low | Medium | High

## Deployment Summary

<1-2 sentences: what is being deployed, key changes, and why a rollback plan is needed.>

### Changes Included
- <Change 1: brief description and risk level>
- <Change 2: brief description and risk level>

### Rollback Complexity
| Factor | Assessment |
|--------|-----------|
| Database migrations | Reversible / Irreversible / None |
| Data format changes | Compatible / Incompatible / None |
| Feature flags available | Yes / Partial / No |
| External API changes | Breaking / Non-breaking / None |
| Multi-service coordination | Required / Not required |

## Trigger Criteria

### Automatic Rollback Triggers
Rollback should be initiated immediately if any of the following are observed:

| Metric | Baseline | Threshold | Monitoring Dashboard |
|--------|----------|-----------|---------------------|
| Error rate (5xx) | <X%> | > <Y%> for 5 min | <dashboard link> |
| P99 latency | <Xms> | > <Yms> for 5 min | <dashboard link> |
| <Critical flow> success rate | <X%> | < <Y%> for 3 min | <dashboard link> |

### Manual Rollback Triggers
Rollback should be considered if:
- Support reports > <n> tickets about <issue type> within <timeframe>
- <Critical business metric> drops below <threshold>
- On-call engineer observes <specific symptom>

### Decision Authority
- **Auto-trigger**: On-call engineer can initiate without approval
- **Manual trigger**: Requires sign-off from <role> during business hours, on-call engineer authority after hours

## Rollback Procedure

### Pre-Rollback (2 minutes)

1. **Announce** — Post in <channel>: "Initiating rollback of <version>. Reason: <trigger>. ETA: <minutes>."
2. **Snapshot current state** — Record current error rates, latency, and active user count for comparison.

### Step 1: Feature Flags (if applicable)

Disable the following feature flags immediately to reduce impact while the full rollback proceeds:

| Flag | Current State | Rollback State | Command |
|------|---------------|----------------|---------|
| `<flag_name>` | enabled | disabled | `<exact command>` |

### Step 2: Application Rollback

```bash
# Exact commands to roll back the application
<command to revert to previous version>

For multi-service deployments, roll back in this order:

1. — <command> (depends on nothing)
2. — <command> (depends on Service A)

Step 3: Database Rollback (if applicable)

# Exact commands to reverse the migration
<migration rollback command>

Pre-check before running:

• Confirm no data written by new code is incompatible with old schema
• Confirm migration reversal does not drop columns containing user data
• Take a database snapshot before reversing: <snapshot command>

Step 4: Cache and CDN Invalidation

# Invalidate stale cache entries
<cache invalidation command>

Step 5: Verify Rollback

See Verification section below.

Data Considerations

Data Compatibility Matrix

Irreversible Changes

Backup and Restore

Communication Plan

During Rollback

If User-Facing Impact

Verification

Immediate Checks (first 5 minutes)

• Health check endpoints returning 200: <curl command or URL>
• Application version confirmed as previous: <version check command>
• Error rate returning to baseline:
• P99 latency returning to baseline:

Functional Checks (next 15 minutes)

• <Critical user flow 1>:
• <Critical user flow 2>:
• <Critical user flow 3>:

Extended Monitoring (next 1-2 hours)

• Business metrics stable:
• No new error patterns in logs:
• Background jobs processing normally:

Rollback Complete Criteria

The rollback is declared complete when ALL of the following are true:

• Error rate is at or below pre-deployment baseline for 30 minutes
• All critical user flows verified working
• No new support tickets related to the deployment
• On-call engineer confirms system stability

## Anti-patterns

- **"Just revert the commit"** — Reverting code does not revert database migrations, cache state, feature flag changes, or data written by the new code. A rollback plan must address every layer of the system, not just the application code.
- **Not considering data migrations** — If the deployment includes a database migration that adds a column, reverting the code while leaving the new column is usually fine. But if it renames a column or changes data formats, the old code may break against the new schema. Always assess data compatibility.
- **No trigger criteria** — "We will roll back if something goes wrong" is not a plan. Define specific, measurable thresholds that trigger a rollback. Without them, the decision becomes a debate during an outage.
- **No verification step** — Rolling back and assuming it worked is dangerous. Define explicit checks that confirm the rollback succeeded: health endpoints, user flows, error rates, and business metrics.
- **Not testing the rollback** — A rollback plan that has never been tested is a hypothesis, not a plan. Test the rollback procedure on staging before the production deployment.
- **Missing communication plan** — Rolling back without notifying the team leads to confusion, duplicate debugging, and conflicting actions. Define who to notify, through which channel, and what to say.
- **Assuming rollback is instantaneous** — DNS propagation, cache expiration, client-side caching, and mobile app versions mean that some users will still see the new version after rollback. Account for this in the plan.
- **No data backup before migration** — If the migration is irreversible and something goes wrong, the only recovery is from a backup. Always take a backup before running migrations, and document the restore procedure with time estimates.