Rollback Strategy Advisor

Rollback Strategy Advisor

Provide safe and effective rollback strategies for failed deployments.

Core Capabilities

This skill helps recover from failed deployments by:

1. Assessing failure impact - Identifying what failed and what needs rollback
2. Recommending rollback strategy - Choosing the appropriate approach based on failure type
3. Providing step-by-step guidance - Clear procedural instructions for execution
4. Validating rollback success - Ensuring system returns to stable state
5. Preventing data loss - Protecting critical data during rollback operations

Rollback Strategy Workflow

Step 1: Assess the Failure

Understand what failed and the scope of impact.

Key Questions:

• What component failed? (Application, database, infrastructure, configuration)
• When did the failure occur? (During deployment, post-deployment, gradual degradation)
• What is the current system state? (Partially deployed, fully deployed, crashed)
• Is the system serving traffic? (Production load, maintenance mode, offline)
• Are there data changes? (Database migrations applied, user data modified)

Gather Information:

# Check deployment status
docker ps -a  # Container status
docker logs <container-name> --tail 100  # Recent logs

# Check system health
curl http://localhost:8080/health  # Health endpoint
docker stats  # Resource usage

# Identify deployment artifacts
docker images | grep <app-name>  # Available images
git log --oneline -10  # Recent commits

Output: Failure Assessment

Component: Application container
Failure Time: 5 minutes post-deployment
System State: New version deployed, returning 500 errors
Traffic: Receiving production traffic (degraded service)
Data Changes: No database migrations in this deployment

Step 2: Choose Rollback Strategy

Select the appropriate strategy based on failure type and system state.

Strategy Decision Tree:

Is database migration involved?
├─ YES → See "Database Rollback Strategy" (Step 3.4)
└─ NO → Continue

Is infrastructure changed?
├─ YES → See "Infrastructure Rollback Strategy" (Step 3.3)
└─ NO → Continue

Is configuration changed?
├─ YES → See "Configuration Rollback Strategy" (Step 3.2)
└─ NO → Application Code Rollback (Step 3.1)

Common Strategies:

Step 3: Execute Rollback

Perform the rollback with validation at each step.

Step 3.1: Application Code Rollback

Revert to the previous working application version.

Standard Procedure:

# 1. Identify previous working version
docker images | grep <app-name>
# Look for the previous tag (e.g., v1.2.3 if current is v1.2.4)

# 2. Stop current containers
docker-compose stop <service-name>

# 3. Update docker-compose.yml to previous version
# Change: image: myapp:v1.2.4 → image: myapp:v1.2.3

# 4. Start with previous version
docker-compose up -d <service-name>

# 5. Validate rollback (see Step 4)
curl http://localhost:8080/health
docker logs <service-name> --tail 50

Fast Rollback (if compose file unchanged):

# Restart with previous image tag
docker-compose stop <service-name>
docker run -d --name <service-name> \
  --network <network-name> \
  -p 8080:8080 \
  myapp:v1.2.3

# Or update compose and restart
sed -i 's/myapp:v1.2.4/myapp:v1.2.3/g' docker-compose.yml
docker-compose up -d <service-name>

Considerations:

• Keep previous images available (don't prune immediately after deploy)
• Tag images with version numbers or git commit SHAs
• Test the previous version still works in staging first if possible
• Monitor resource usage during rollback

Step 3.2: Configuration Rollback

Restore previous configuration files or environment variables.

Configuration File Rollback:

# 1. Locate configuration backup or git history
git log -- config/app.conf
git show HEAD~1:config/app.conf > config/app.conf

# 2. Update mounted config in docker-compose.yml if needed
# Ensure volume mount points to correct config

# 3. Restart services to load previous config
docker-compose restart <service-name>

# 4. Validate configuration loaded correctly
docker exec <service-name> cat /app/config/app.conf
curl http://localhost:8080/health

Environment Variable Rollback:

# 1. Edit docker-compose.yml to restore previous env vars
# Update the environment section or .env file

# 2. Recreate container with new env vars
docker-compose up -d --force-recreate <service-name>

# 3. Verify environment variables
docker exec <service-name> env | grep APP_

Feature Flag Rollback:

# If using feature flags, disable the problematic feature
# Update flag config or environment variable
# Example: FEATURE_NEW_CHECKOUT=false

docker-compose restart <service-name>

Considerations:

• Keep configuration in version control (git)
• Use .env files for environment-specific configs
• Backup configs before deployment
• Validate config syntax before applying

Step 3.3: Infrastructure Rollback

Revert infrastructure changes like network configurations, volume mounts, or docker-compose structure.

Docker Compose Rollback:

# 1. Restore previous docker-compose.yml from git
git checkout HEAD~1 -- docker-compose.yml

# 2. Recreate infrastructure
docker-compose down
docker-compose up -d

# 3. Validate all services running
docker-compose ps
docker-compose logs --tail 50

Network Configuration Rollback:

# If network configuration changed
# 1. Remove new network
docker network rm <new-network>

# 2. Recreate previous network
docker network create --driver bridge <old-network>

# 3. Reconnect containers
docker network connect <old-network> <container-name>

Volume Rollback:

# If volume mounts changed (be careful with data!)
# 1. Stop services
docker-compose stop

# 2. Update docker-compose.yml volume configuration
git checkout HEAD~1 -- docker-compose.yml

# 3. Restart services
docker-compose up -d

# Note: Data in volumes persists, only mount configuration changes

Considerations:

• Infrastructure changes may affect multiple services
• Test in staging environment first if possible
• Document infrastructure dependencies
• Consider using infrastructure-as-code tools (Terraform)

Step 3.4: Database Rollback

Reverse database migrations and restore schema to previous state.

Migration Rollback (with Migration Tool):

# Using Alembic (Python)
docker exec <db-container> alembic downgrade -1  # Rollback one migration
docker exec <db-container> alembic downgrade <revision>  # Rollback to specific revision

# Using Flyway (Java)
docker exec <app-container> flyway undo  # Rollback last migration

# Using Django
docker exec <app-container> python manage.py migrate <app> <migration>

# Using Rails
docker exec <app-container> rails db:rollback STEP=1

Manual Migration Rollback:

# 1. Identify the migration to reverse
docker exec <db-container> psql -U user -d dbname -c "\d+"  # List tables

# 2. Execute reverse migration SQL
docker exec <db-container> psql -U user -d dbname -f /migrations/rollback_v1.2.4.sql

# 3. Verify schema state
docker exec <db-container> psql -U user -d dbname -c "\d table_name"

Database Rollback with Application:

# CRITICAL: Rollback database BEFORE rolling back application
# to prevent new app code from working with old schema

# 1. Stop application (prevent new requests)
docker-compose stop app-service

# 2. Backup current database state
docker exec <db-container> pg_dump -U user dbname > backup_$(date +%Y%m%d_%H%M%S).sql

# 3. Rollback migration
docker exec <db-container> alembic downgrade -1

# 4. Rollback application to version compatible with old schema
docker-compose stop app-service
sed -i 's/myapp:v1.2.4/myapp:v1.2.3/g' docker-compose.yml
docker-compose up -d app-service

# 5. Validate
curl http://localhost:8080/health
docker logs app-service --tail 50

Considerations:

• Always backup before rollback - Database changes are risky
• Coordinate app and DB rollback carefully
• Test rollback migrations in staging
• Consider data loss implications (irreversible data changes)
• For destructive migrations (dropped columns), may need data restore from backup
• Use database versioning tools (Alembic, Flyway, Liquibase)

See references/database_rollback_patterns.md for detailed migration rollback examples and data preservation strategies.

Step 4: Validate Rollback Success

Confirm the system is working correctly after rollback.

Health Checks:

# 1. Container health
docker ps  # All containers running?
docker-compose ps  # Services in "Up" state?

# 2. Application health
curl http://localhost:8080/health
curl -I http://localhost:8080  # HTTP status code

# 3. Service logs
docker logs <service-name> --tail 100 | grep ERROR
docker logs <service-name> --tail 100 | grep WARN

# 4. Database connectivity
docker exec <app-container> psql -U user -d dbname -c "SELECT 1;"

# 5. Resource usage
docker stats --no-stream

Functional Testing:

# Test critical user flows
curl -X POST http://localhost:8080/api/login -d '{"user":"test","pass":"test"}'
curl http://localhost:8080/api/users/1

# Run smoke tests if available
docker exec <app-container> pytest tests/smoke/

# Check monitoring dashboards
# - Response times back to normal?
# - Error rates dropped?
# - Traffic being served?

Validation Checklist:

• ✓ All containers running
• ✓ Health endpoints returning 200
• ✓ No error spikes in logs
• ✓ Database queries executing
• ✓ Critical API endpoints responding
• ✓ Monitoring shows normal metrics
• ✓ Users can access the application

Step 5: Document and Communicate

Record the incident and inform stakeholders.

Incident Report Template:

## Deployment Rollback - [Date/Time]

**Summary:** Brief description of what failed and rollback action taken

**Timeline:**
- [Time] - Deployment started (v1.2.4)
- [Time] - Failure detected (500 errors)
- [Time] - Rollback initiated
- [Time] - Rollback completed
- [Time] - System validated stable

**Root Cause:** What caused the deployment to fail

**Rollback Actions:**
1. Stopped application service
2. Reverted docker-compose.yml to v1.2.3
3. Restarted service
4. Validated health checks

**Impact:**
- Downtime: X minutes
- Affected users: Y requests failed
- Data loss: None

**Follow-up Actions:**
- [ ] Fix root cause in v1.2.5
- [ ] Add test coverage for failure scenario
- [ ] Update deployment checklist
- [ ] Review rollback procedure effectiveness

Communication:

Team notification (Slack/email):

🚨 Deployment Rollback Completed

We rolled back the v1.2.4 deployment due to [issue].
System is now stable on v1.2.3.

Impact: X minutes downtime
Status: Fully operational
Next steps: Root cause analysis, fix in v1.2.5

For details see: [link to incident report]

Step 6: Prevent Future Failures

Analyze the incident and improve deployment practices.

Post-Incident Review:

1. What went wrong?

   • Code bug not caught in testing
   • Configuration incompatibility
   • Missing database index caused performance degradation
   • Infrastructure resource limits exceeded

2. Why wasn't it caught earlier?

   • Insufficient test coverage
   • Staging environment differs from production
   • Load testing not performed
   • Migration not tested with production data volume

3. What can prevent this?

   • Add integration test for failure scenario
   • Improve staging/production parity
   • Implement canary deployments
   • Add automated rollback triggers
   • Enhance monitoring and alerting

Deployment Improvements:

# Implement health checks in docker-compose.yml
services:
  app:
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8080/health"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 40s

Rollback Automation:

# Create rollback script for quick recovery
#!/bin/bash
# rollback.sh - Quick rollback to previous version

PREVIOUS_VERSION=$1

if [ -z "$PREVIOUS_VERSION" ]; then
  echo "Usage: ./rollback.sh <version>"
  exit 1
fi

echo "Rolling back to $PREVIOUS_VERSION..."
docker-compose stop app
sed -i "s/myapp:.*$/myapp:$PREVIOUS_VERSION/g" docker-compose.yml
docker-compose up -d app
docker-compose ps
echo "Rollback complete. Check logs: docker logs app"

Best Practices:

1. Maintain deployment history - Keep previous images, configs, and compose files
2. Test rollback procedures - Practice rollbacks in staging regularly
3. Automate health checks - Use docker healthchecks and monitoring
4. Version everything - Tag images, version configs, track migrations
5. Backup before risky changes - Database backups before migrations
6. Document dependencies - Track what depends on what for coordinated rollbacks
7. Gradual rollouts - Use canary or blue-green deployments when possible
8. Monitor post-deployment - Watch metrics closely for 30+ minutes after deploy

Quick Reference

Rollback Decision Matrix

Common Rollback Commands

# Quick app rollback
docker-compose stop <service>
sed -i 's/v1.2.4/v1.2.3/g' docker-compose.yml
docker-compose up -d <service>

# Config rollback
git checkout HEAD~1 -- config/
docker-compose restart <service>

# Database migration rollback
docker exec <db-container> alembic downgrade -1

# Full infrastructure rollback
git checkout HEAD~1 -- docker-compose.yml
docker-compose down && docker-compose up -d

Resources

• references/database_rollback_patterns.md - Detailed database migration rollback strategies and data preservation techniques
• references/platform_guides.md - Docker and Docker Compose specific rollback procedures and best practices

Best Practices

1. Always backup before rollback - Especially for database changes
2. Test rollback in staging first - If time permits
3. Stop traffic during risky rollbacks - Prevent inconsistent state
4. Rollback in reverse order - Undo changes in opposite sequence of deployment
5. Validate each step - Don't proceed if validation fails
6. Document everything - Create audit trail for compliance and learning
7. Communicate clearly - Keep stakeholders informed of status
8. Practice rollbacks regularly - Ensure procedures work when needed
9. Automate common rollbacks - Reduce human error and recovery time
10. Learn from failures - Use incidents to improve deployment process