cloudthinker-ai Database Failover Runbook Skill
Execute database failover for {{ database_name }} ({{ db_engine }}). Failover from {{ primary_endpoint }} to {{ replica_endpoint }}. Reason: {{ failover_reason }}
Workflow
Phase 1 — Pre-Failover Checks
PRE-FAILOVER CHECKLIST
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
REPLICATION HEALTH
[ ] Replication lag < 1 second on standby
[ ] Replication status: streaming / in-sync
[ ] No replication errors in last 24 hours
[ ] WAL / binlog position confirmed on both nodes
STANDBY VALIDATION
[ ] Standby is reachable: {{ replica_endpoint }}
[ ] Standby read queries returning correct data
[ ] Standby resource utilization acceptable (CPU, memory, disk)
[ ] Standby has identical schema version as primary
[ ] Standby connection pool capacity sufficient
APPLICATION READINESS
[ ] All application connection strings use DNS / virtual endpoint
[ ] No long-running transactions on primary
[ ] No active schema migrations or DDL operations
[ ] Maintenance window communicated to stakeholders
[ ] On-call engineer confirmed available
Phase 2 — Pre-Failover Preparation
PREPARATION
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[ ] Create fresh backup of primary database
[ ] Record current primary state:
- Active connections: ___
- Transactions per second: ___
- Replication lag: ___ ms
- WAL/binlog position: ___
[ ] Pause non-critical batch jobs and cron tasks
[ ] Reduce application connection pool sizes (if possible)
[ ] Drain active connections gracefully (planned failover only)
[ ] Notify monitoring team to expect alerts
Phase 3 — Failover Execution
FAILOVER EXECUTION
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
PLANNED FAILOVER:
1. [ ] Stop writes to primary (set read-only or block connections)
2. [ ] Wait for replication lag to reach zero
3. [ ] Promote standby to primary role
4. [ ] Record promotion timestamp: ___
5. [ ] Verify new primary accepts writes (test INSERT)
UNPLANNED FAILOVER:
1. [ ] Confirm primary is unavailable / degraded
2. [ ] Check replication lag at time of failure: ___ ms
3. [ ] Accept potential data loss window: ___ transactions
4. [ ] Force-promote standby to primary
5. [ ] Record promotion timestamp: ___
6. [ ] Verify new primary accepts writes (test INSERT)
Phase 4 — DNS and Endpoint Update
DNS / ENDPOINT UPDATE
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[ ] Update DNS record: {{ primary_endpoint }} -> new primary IP
[ ] Update virtual IP or proxy target (if applicable)
[ ] Update connection string in secrets manager (if hardcoded)
[ ] Verify DNS propagation (dig {{ primary_endpoint }})
[ ] Confirm applications reconnecting to new primary
[ ] Check connection count stabilizing on new primary
Phase 5 — Post-Failover Validation
POST-FAILOVER VALIDATION
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
HEALTH CHECKS (at T+5min, T+15min, T+1h)
[ ] New primary accepting reads and writes
[ ] Query latency within baseline (p50/p95/p99)
[ ] Error rate returned to normal
[ ] Connection pool utilization stable
[ ] No deadlocks or lock contention
DATA INTEGRITY
[ ] Row counts match expected values on critical tables
[ ] Recent transactions visible (spot-check latest records)
[ ] Auto-increment / sequence values correct
[ ] Foreign key constraints intact
APPLICATION HEALTH
[ ] All application instances connected to new primary
[ ] API response times within SLA
[ ] No 5xx errors related to database connectivity
[ ] Background jobs processing successfully
Phase 6 — Rebuild Standby and Cleanup
REBUILD STANDBY
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[ ] Configure old primary as new standby (or provision new standby)
[ ] Start replication from new primary to new standby
[ ] Verify replication streaming and lag decreasing
[ ] Wait for standby to fully catch up
[ ] Test read queries on new standby
CLEANUP
[ ] Re-enable batch jobs and cron tasks
[ ] Restore normal connection pool sizes
[ ] Update runbook with new primary/standby endpoints
[ ] Update monitoring targets and alert thresholds
[ ] Notify stakeholders of completed failover
[ ] Document any data loss or issues encountered
Counter-Rationalizations
| Shortcut | Counter | Why |
|---|---|---|
| "We can skip some steps for this case" | Adapt the workflow steps, don't skip them | Skipped steps are where incidents and oversights originate |
| "The user seems to already know what to do" | Complete all workflow phases with the user | The workflow catches blind spots that experience alone misses |
| "This is a minor case, full process is overkill" | Scale the process down, don't turn it off | Minor cases become major when unstructured; the process scales, not disappears |
| "I'll fill in the details later" | Complete each section before moving on | Deferred details are forgotten; real-time capture is more accurate |
| "The template output isn't necessary" | Always produce the structured output format | Structured output enables comparison, audit trails, and handoff to other teams |
Output Format
Produce a database failover execution report with:
- Failover summary (database, engine, reason, timestamps)
- Pre-failover health snapshot (replication lag, connection count)
- Execution log with step-by-step confirmation
- DNS update confirmation and propagation status
- Post-failover validation results (latency, error rate, data integrity)
- Standby rebuild status and next steps