Amey-Thakur Leader election
"Exactly one instance runs this" is a lease renewed forever, plus the discipline to stop instantly when renewal fails. Split brain is not an edge case; it is the default outcome of skipping either half.
Method
- Elect by lease, not by deployment. A designated singleton pod is an availability hole. Every candidate runs; each tries to acquire or renew a lease (etcd/Consul session, Kubernetes Lease object, or a DB row with owner + expires_at compare-and-swapped); the holder leads.
- Renew early, resign fast. Renew at TTL/3. On renewal failure or timeout: stop leading immediately (halt loops, cancel in-flight dangerous work), do not finish the batch first. The window between your last successful renewal and your halt is your split-brain exposure; keep it seconds, not minutes.
- Fence the resources anyway. GC pauses and network stalls mean a deposed leader can act while deposed (same gap as distributed-locks). Carry the lease generation/token on every write the leader makes; receivers reject stale generations. Election chooses who should act; fencing enforces who may.
- Make leadership observable. Export who leads, since when, and election count. Frequent elections mean the TTL is tight against your GC/network reality or the store is unhealthy; alert on churn, and on zero leaders (minority partition or misconfig: the job silently not running is the failure users find last).
- Hand off cleanly on shutdown. Resign the lease (delete/release) during graceful-shutdown so the successor starts in seconds instead of waiting out the TTL. Rolling deploys otherwise add a full TTL of leaderless time per restart.
- Design followers to be warm. Followers keep caches and connections ready but take no actions; promotion is flipping a flag, not booting a subsystem. If promotion takes minutes, the lease TTL protects nothing.
Boundaries
- If the work can be partitioned (shard schedulers by key range, consumers by partition), partition it; leadership serializes everything through one node and caps throughput at its capacity.
- Leader election coordinates activity; it does not replicate state. A new leader reading its predecessor's half-written state needs that state transactional or checkpointed (see background-jobs).
- Cross-region leadership means cross-region lease latency; either accept it, or run per-region leaders with a partitioned domain.