cloudthinker-ai Network Partition Recovery Runbook Skill
Diagnose and recover network partition in {{ affected_segment }}. Symptoms: {{ symptoms }} | Affected: {{ affected_services }}
Workflow
Phase 1 — Initial Diagnosis
INITIAL DIAGNOSIS
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
SCOPE DETERMINATION
[ ] Identify all hosts/services affected by the partition
[ ] Determine if partition is complete (no connectivity) or partial (packet loss)
[ ] Map which source->destination pairs are failing
[ ] Check if issue is unidirectional or bidirectional
[ ] Identify the network boundary where connectivity fails
CONNECTIVITY TESTS
From: ___ To: ___ Result: [OK / FAIL / PARTIAL]
From: ___ To: ___ Result: [OK / FAIL / PARTIAL]
From: ___ To: ___ Result: [OK / FAIL / PARTIAL]
LAYER-BY-LAYER CHECK
[ ] Layer 2: ARP resolution working (arping)
[ ] Layer 3: IP routing correct (traceroute, ip route)
[ ] Layer 4: TCP connections establishing (telnet, nc)
[ ] Layer 7: Application-level connectivity (curl, health checks)
[ ] DNS: Name resolution working from affected hosts
Phase 2 — Root Cause Investigation
ROOT CAUSE INVESTIGATION
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
ROUTING
[ ] Check route tables for missing or incorrect routes
[ ] Verify default gateway reachable from affected hosts
[ ] Check for asymmetric routing issues
[ ] Review recent route table changes (cloud provider / on-prem)
[ ] Check BGP session status (if applicable)
SECURITY GROUPS / FIREWALLS
[ ] Review security group rules for recent changes
[ ] Check network ACLs for deny rules
[ ] Verify firewall rules (iptables, nftables, cloud firewall)
[ ] Check for IP blocklist entries matching affected hosts
[ ] Review WAF or DDoS protection rule changes
INFRASTRUCTURE
[ ] Check VPC peering / transit gateway status
[ ] Verify VPN tunnel status (if cross-network)
[ ] Check network interface status on affected hosts
[ ] Review cloud provider service health dashboard
[ ] Check for NIC driver issues or MTU mismatches:
Expected MTU: ___ Actual MTU: ___
RECENT CHANGES
[ ] Infrastructure-as-code deployments in last 24h
[ ] Security group or NACL modifications
[ ] Route table changes
[ ] VPN or peering configuration changes
[ ] DNS changes
Phase 3 — Route and Connectivity Fix
FIX APPLICATION
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
IDENTIFIED ROOT CAUSE: ___
FIX STEPS (select applicable):
ROUTING FIX:
[ ] Add/correct route: ___ via ___
[ ] Verify route propagation across all route tables
[ ] Test connectivity after route change
SECURITY GROUP / FIREWALL FIX:
[ ] Add/modify rule: allow ___ from ___ to ___ port ___
[ ] Verify rule applied (describe-security-groups / iptables -L)
[ ] Test connectivity after rule change
INFRASTRUCTURE FIX:
[ ] Restore peering connection / transit gateway attachment
[ ] Re-establish VPN tunnel
[ ] Restart network interface on affected host(s)
[ ] Fix MTU mismatch (ip link set dev ___ mtu ___)
ROLLBACK PLAN:
If fix causes wider issues:
[ ] Revert change: ___
[ ] Verify original connectivity pattern restored
Phase 4 — Connectivity Validation
CONNECTIVITY VALIDATION
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
NETWORK TESTS (repeat initial diagnosis tests):
From: ___ To: ___ Result: [OK / FAIL]
From: ___ To: ___ Result: [OK / FAIL]
From: ___ To: ___ Result: [OK / FAIL]
[ ] Packet loss rate: ___% (target: 0%)
[ ] Latency within baseline: ___ ms (baseline: ___ ms)
[ ] Traceroute path as expected (no unexpected hops)
[ ] TCP connections establishing within normal timeout
[ ] No retransmissions above baseline
APPLICATION VALIDATION
[ ] All previously affected services communicating
[ ] Health checks passing across all service pairs
[ ] Database connections re-established
[ ] Message queue consumers reconnected
[ ] API response times returned to baseline
Phase 5 — Prevention and Documentation
PREVENTION AND DOCUMENTATION
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[ ] Add monitoring for the specific failure mode detected
[ ] Create alert for route table changes in affected segment
[ ] Add network connectivity smoke tests to CI/CD
[ ] Update network diagrams if topology changed
[ ] Document root cause and fix in incident report
[ ] Review change management process for network changes
[ ] Consider adding redundant network paths
[ ] Update runbook with lessons learned
Counter-Rationalizations
| Shortcut | Counter | Why |
|---|---|---|
| "We can skip some steps for this case" | Adapt the workflow steps, don't skip them | Skipped steps are where incidents and oversights originate |
| "The user seems to already know what to do" | Complete all workflow phases with the user | The workflow catches blind spots that experience alone misses |
| "This is a minor case, full process is overkill" | Scale the process down, don't turn it off | Minor cases become major when unstructured; the process scales, not disappears |
| "I'll fill in the details later" | Complete each section before moving on | Deferred details are forgotten; real-time capture is more accurate |
| "The template output isn't necessary" | Always produce the structured output format | Structured output enables comparison, audit trails, and handoff to other teams |
Output Format
Produce a network partition recovery report with:
- Partition summary (segment, symptoms, scope, duration)
- Diagnosis results (layer-by-layer findings)
- Root cause identification with evidence
- Fix applied with before/after connectivity tests
- Validation results confirming full recovery
- Prevention measures to avoid recurrence