# Runbook Network Partition > Use when performing runbook network partition — network partition recovery procedure covering diagnosis, route fix, connectivity validation, and root cause analysis. Use when services experience network segmentation, intermittent connectivity, or cross-zone communication failures. - Skill: `cloudthinker-ai/runbook-network-partition` (Agent Skill) - Install (CLI): `npx skillmds@latest add cloudthinker-ai/runbook-network-partition` - Raw SKILL.md: https://api.skillmd.com/api/skills/cloudthinker-ai/runbook-network-partition/raw - Safety review: pending (external: skill-scanner PASS, skillspector PASS) - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Coding & Dev Tools - Author: cloudthinker-ai (https://skillmd.com/u/cloudthinker-ai) - Updated: 2026-09-08 - Page: https://skillmd.com/skills/cloudthinker-ai/runbook-network-partition --- # Network Partition Recovery Runbook Skill Diagnose and recover network partition in **{{ affected_segment }}**. Symptoms: **{{ symptoms }}** | Affected: **{{ affected_services }}** ## Workflow ### Phase 1 — Initial Diagnosis ``` INITIAL DIAGNOSIS ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ SCOPE DETERMINATION [ ] Identify all hosts/services affected by the partition [ ] Determine if partition is complete (no connectivity) or partial (packet loss) [ ] Map which source->destination pairs are failing [ ] Check if issue is unidirectional or bidirectional [ ] Identify the network boundary where connectivity fails CONNECTIVITY TESTS From: ___ To: ___ Result: [OK / FAIL / PARTIAL] From: ___ To: ___ Result: [OK / FAIL / PARTIAL] From: ___ To: ___ Result: [OK / FAIL / PARTIAL] LAYER-BY-LAYER CHECK [ ] Layer 2: ARP resolution working (arping) [ ] Layer 3: IP routing correct (traceroute, ip route) [ ] Layer 4: TCP connections establishing (telnet, nc) [ ] Layer 7: Application-level connectivity (curl, health checks) [ ] DNS: Name resolution working from affected hosts ``` ### Phase 2 — Root Cause Investigation ``` ROOT CAUSE INVESTIGATION ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ROUTING [ ] Check route tables for missing or incorrect routes [ ] Verify default gateway reachable from affected hosts [ ] Check for asymmetric routing issues [ ] Review recent route table changes (cloud provider / on-prem) [ ] Check BGP session status (if applicable) SECURITY GROUPS / FIREWALLS [ ] Review security group rules for recent changes [ ] Check network ACLs for deny rules [ ] Verify firewall rules (iptables, nftables, cloud firewall) [ ] Check for IP blocklist entries matching affected hosts [ ] Review WAF or DDoS protection rule changes INFRASTRUCTURE [ ] Check VPC peering / transit gateway status [ ] Verify VPN tunnel status (if cross-network) [ ] Check network interface status on affected hosts [ ] Review cloud provider service health dashboard [ ] Check for NIC driver issues or MTU mismatches: Expected MTU: ___ Actual MTU: ___ RECENT CHANGES [ ] Infrastructure-as-code deployments in last 24h [ ] Security group or NACL modifications [ ] Route table changes [ ] VPN or peering configuration changes [ ] DNS changes ``` ### Phase 3 — Route and Connectivity Fix ``` FIX APPLICATION ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ IDENTIFIED ROOT CAUSE: ___ FIX STEPS (select applicable): ROUTING FIX: [ ] Add/correct route: ___ via ___ [ ] Verify route propagation across all route tables [ ] Test connectivity after route change SECURITY GROUP / FIREWALL FIX: [ ] Add/modify rule: allow ___ from ___ to ___ port ___ [ ] Verify rule applied (describe-security-groups / iptables -L) [ ] Test connectivity after rule change INFRASTRUCTURE FIX: [ ] Restore peering connection / transit gateway attachment [ ] Re-establish VPN tunnel [ ] Restart network interface on affected host(s) [ ] Fix MTU mismatch (ip link set dev ___ mtu ___) ROLLBACK PLAN: If fix causes wider issues: [ ] Revert change: ___ [ ] Verify original connectivity pattern restored ``` ### Phase 4 — Connectivity Validation ``` CONNECTIVITY VALIDATION ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ NETWORK TESTS (repeat initial diagnosis tests): From: ___ To: ___ Result: [OK / FAIL] From: ___ To: ___ Result: [OK / FAIL] From: ___ To: ___ Result: [OK / FAIL] [ ] Packet loss rate: ___% (target: 0%) [ ] Latency within baseline: ___ ms (baseline: ___ ms) [ ] Traceroute path as expected (no unexpected hops) [ ] TCP connections establishing within normal timeout [ ] No retransmissions above baseline APPLICATION VALIDATION [ ] All previously affected services communicating [ ] Health checks passing across all service pairs [ ] Database connections re-established [ ] Message queue consumers reconnected [ ] API response times returned to baseline ``` ### Phase 5 — Prevention and Documentation ``` PREVENTION AND DOCUMENTATION ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ [ ] Add monitoring for the specific failure mode detected [ ] Create alert for route table changes in affected segment [ ] Add network connectivity smoke tests to CI/CD [ ] Update network diagrams if topology changed [ ] Document root cause and fix in incident report [ ] Review change management process for network changes [ ] Consider adding redundant network paths [ ] Update runbook with lessons learned ``` ## Counter-Rationalizations | Shortcut | Counter | Why | |----------|---------|-----| | "We can skip some steps for this case" | Adapt the workflow steps, don't skip them | Skipped steps are where incidents and oversights originate | | "The user seems to already know what to do" | Complete all workflow phases with the user | The workflow catches blind spots that experience alone misses | | "This is a minor case, full process is overkill" | Scale the process down, don't turn it off | Minor cases become major when unstructured; the process scales, not disappears | | "I'll fill in the details later" | Complete each section before moving on | Deferred details are forgotten; real-time capture is more accurate | | "The template output isn't necessary" | Always produce the structured output format | Structured output enables comparison, audit trails, and handoff to other teams | ## Output Format Produce a network partition recovery report with: 1. **Partition summary** (segment, symptoms, scope, duration) 2. **Diagnosis results** (layer-by-layer findings) 3. **Root cause** identification with evidence 4. **Fix applied** with before/after connectivity tests 5. **Validation** results confirming full recovery 6. **Prevention measures** to avoid recurrence