cloudthinker-ai Load Balancer Failover Runbook Skill
Execute LB failover for {{ lb_name }} from {{ primary_lb }} to {{ secondary_lb }}. Type: {{ failover_type }}
Workflow
Phase 1 — Pre-Failover Health Check
PRE-FAILOVER HEALTH CHECK
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
PRIMARY LB STATUS
Name: {{ lb_name }}
Endpoint: {{ primary_lb }}
Current RPS: ___
Active connections: ___
Healthy backends: ___ / ___
SSL certificate expiry: ___
SECONDARY LB STATUS
Endpoint: {{ secondary_lb }}
Health check status: ___
Healthy backends: ___ / ___
SSL certificate valid: YES / NO
Last traffic served: ___
BACKEND PARITY CHECK
[ ] Same backend instances registered on both LBs
[ ] Health check configuration identical
[ ] Listener rules / routing rules match
[ ] SSL certificates match on both LBs
[ ] Security groups allow traffic from secondary LB to backends
[ ] Sticky sessions / session persistence configured identically
Phase 2 — Pre-Failover Validation
PRE-FAILOVER VALIDATION
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[ ] Test secondary LB with synthetic requests:
curl -H "Host: {{ lb_name }}" https://{{ secondary_lb }}/health
Response: ___ (expect 200 OK)
[ ] Verify response content matches primary LB output
[ ] Check latency from secondary LB: ___ ms
[ ] Verify logging and monitoring configured on secondary LB
[ ] Confirm DNS TTL is low enough for quick switchover: ___ seconds
[ ] Notify on-call team and stakeholders of planned failover
[ ] Prepare rollback command / procedure
Phase 3 — Traffic Shift Execution
TRAFFIC SHIFT
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
GRADUAL SHIFT (recommended for planned failovers):
1. [ ] Route 10% traffic to secondary LB
2. [ ] Monitor for 5 minutes: error rate ___, latency ___
3. [ ] Route 50% traffic to secondary LB
4. [ ] Monitor for 5 minutes: error rate ___, latency ___
5. [ ] Route 100% traffic to secondary LB
6. [ ] Confirm zero traffic on primary LB
IMMEDIATE SHIFT (for emergency failovers):
1. [ ] Update DNS record to point to {{ secondary_lb }}
2. [ ] Update Route53 health check / failover routing (if applicable)
3. [ ] Record switchover timestamp: ___
4. [ ] Verify DNS propagation
TRAFFIC SHIFT METHOD:
[ ] DNS-based (Route53 weighted/failover routing)
[ ] Global LB (CloudFront, Global Accelerator, Cloudflare)
[ ] Manual DNS update
Phase 4 — Post-Failover Backend Validation
BACKEND VALIDATION
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
TRAFFIC FLOW (check at T+5min, T+15min)
[ ] All traffic arriving at secondary LB backends
[ ] No traffic leaking to primary LB
[ ] Request distribution across backends is balanced
[ ] No backend instances marked unhealthy
APPLICATION HEALTH
[ ] Response times within SLA: ___ ms (SLA: ___ ms)
[ ] Error rate: ___% (baseline: ___%)
[ ] All API endpoints responding correctly
[ ] WebSocket / long-lived connections re-established
[ ] No session loss for sticky session workloads
MONITORING
[ ] Metrics flowing from secondary LB
[ ] Access logs being generated
[ ] Alerts configured and firing correctly
[ ] Dashboard updated to show secondary LB metrics
Phase 5 — Rollback Procedure
ROLLBACK (if issues detected)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Trigger: error rate > ___%, latency > ___ ms, backend failures
1. [ ] Shift traffic back to primary LB (reverse of Phase 3)
2. [ ] Verify primary LB receiving traffic
3. [ ] Confirm primary LB backends healthy
4. [ ] Monitor application health for 15 minutes
5. [ ] Notify stakeholders of rollback
6. [ ] Document failure reason for investigation
Phase 6 — Cleanup
CLEANUP
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[ ] Perform maintenance on primary LB (if planned maintenance)
[ ] Verify primary LB ready for failback when needed
[ ] Update monitoring to reflect new active LB
[ ] Update incident response runbooks with new primary
[ ] Document failover duration and any issues
[ ] Close change management ticket
[ ] Schedule failback (if applicable): ___
Counter-Rationalizations
| Shortcut | Counter | Why |
|---|---|---|
| "We can skip some steps for this case" | Adapt the workflow steps, don't skip them | Skipped steps are where incidents and oversights originate |
| "The user seems to already know what to do" | Complete all workflow phases with the user | The workflow catches blind spots that experience alone misses |
| "This is a minor case, full process is overkill" | Scale the process down, don't turn it off | Minor cases become major when unstructured; the process scales, not disappears |
| "I'll fill in the details later" | Complete each section before moving on | Deferred details are forgotten; real-time capture is more accurate |
| "The template output isn't necessary" | Always produce the structured output format | Structured output enables comparison, audit trails, and handoff to other teams |
Output Format
Produce a load balancer failover report with:
- Failover summary (LB name, primary, secondary, type, timestamps)
- Pre-failover health of both load balancers
- Traffic shift execution log with gradual percentages
- Backend validation results (latency, error rate, distribution)
- Issues and rollback details (if any)
- Post-failover status and next steps