Cloud Provider Outage Response

Cloud Provider Outage Response

Provider: {{ cloud_provider }} | Region: {{ affected_region }} Affected Services: {{ affected_services }}

Status Page URLs

Phase 1 — Confirm and Assess (0-15 min)

Confirm Provider Outage

• Check provider status page (may lag behind actual issues)
• Check AWS Health Dashboard / GCP Service Health / Azure Service Health
• Check community reports (Twitter/X, Hacker News, Downdetector)
• Verify from our own monitoring (distinguish our issue vs. provider issue)
• Check if the outage is regional or global

Assess Our Impact

• Which of our services are in {{ affected_region }}?
• Which cloud services do we depend on that are affected?
• Do we have multi-region redundancy?
• What is the customer-facing impact?
• Are there any data durability concerns?

Service Dependency Map

Phase 2 — Failover Decision (15-30 min)

Failover Decision Matrix

Failover Execution (if proceeding)

DNS-Based Failover

# Update DNS to point to healthy region
# Route53 health checks should auto-failover
# Manual override if needed:
aws route53 change-resource-record-sets --hosted-zone-id ZONE_ID \
  --change-batch '{"Changes":[{"Action":"UPSERT","ResourceRecordSet":{"Name":"api.example.com","Type":"A","TTL":60,"ResourceRecords":[{"Value":"HEALTHY_REGION_IP"}]}}]}'

Load Balancer Failover

• Remove unhealthy region from global load balancer
• Verify traffic routing to healthy region(s)
• Monitor healthy region for capacity (can it handle full load?)

Database Failover

• Promote read replica in healthy region to primary (if applicable)
• Update application connection strings
• Accept potential data loss based on replication lag
• Document the replication lag at time of failover

Capacity Verification

When failing over, the healthy region must handle increased load:

• Check current capacity utilization in healthy region
• Pre-scale compute resources if needed
• Verify database can handle doubled connection count
• Check rate limits and quotas in healthy region
• Monitor for capacity-related failures during failover

Phase 3 — During the Outage

Monitoring

• Set up dedicated monitoring for provider status updates
• Monitor healthy region(s) for stability under increased load
• Track customer-reported issues
• Document all timeline events for post-incident review

Customer Communication

We are currently experiencing service disruption due to an infrastructure
issue at our cloud provider ({{ cloud_provider }}) affecting the
{{ affected_region }} region.

Impact: [describe customer-facing impact]
Status: [Our team is actively working on failover / We are monitoring
         the provider's recovery efforts]
Next update: [time]

You can track the provider's status at: [status page URL]

What NOT to Do

• Do NOT assume the provider will recover quickly (plan for hours)
• Do NOT make changes to the affected region (API calls may fail/timeout)
• Do NOT restart services in the affected region (may make recovery harder)
• Do NOT blame the provider publicly (factual statements only)

Phase 4 — Recovery

When Provider Recovers

• Verify provider announces recovery on status page
• Test our services in the recovered region (do not trust blindly)
• Check data integrity in the recovered region
• Reconcile any data divergence between regions

Failback Procedure (if we failed over)

1. Verify recovered region is stable (wait 30+ minutes)
2. Sync data from active region back to recovered region
3. Gradually shift traffic back (10% → 25% → 50% → 100%)
4. Monitor for issues at each step
5. Restore original multi-region configuration
6. Verify all systems nominal

Data Reconciliation

• Check for data written to both regions during split
• Resolve any conflicts (last-write-wins, manual review, etc.)
• Verify database replication is healthy and caught up
• Check message queues for unprocessed items
• Verify cache consistency

Phase 5 — Post-Outage

SLA Credit Documentation

• Record exact start and end time of outage
• Document affected services and regions
• Calculate downtime percentage for SLA claim
• File SLA credit request with provider (within required timeframe)
• Track credit request to resolution

Architectural Review

• Were we adequately multi-region? If not, plan remediation
• Did failover work as expected?
• What was our recovery time vs. target?
• Do we need additional regions or providers?
• Should we adopt multi-cloud for critical services?
• Review and update disaster recovery plan

Counter-Rationalizations