Leader Election

Leader Election

Purpose

Leader election assigns a distinguished role for an epoch; a lease is one common failure- detection/expiry mechanism, but session locks and quorum terms are alternatives. A distributed lock usually protects a critical section while an election owns a long-lived role, though both need the same stale-owner analysis (distributed-locks-and-leases). A lock need not be a lease. A mutex is in-process exclusion and unrelated (java-memory-model). Lease expiry is one implementation choice, not a requirement. Ownership by partition is the alternative that removes the singleton altogether by assigning keys to instances (sharding-and-partitioning). Consensus is how the election is actually decided (consensus-and-quorums); election is a consumer of it, not a synonym.

Two failures. The first is the one that brings people here: a @Scheduled job that ran once becomes a job that runs N times when the deployment scales to N replicas, silently — the problem statement is stateless-service-design. The second is worse and is what this skill is really about: the leader that has already lost its lease and does not know yet. Between the instant a lease expires and the instant its former holder notices, two instances are both acting as leader, and neither logs anything unusual.

Workflow

Inspect the project's Java baseline, election-library/provider versions, grant/renewal semantics and every protected sink before adapting an example. The Java deadline illustration uses Java 17 language features; it does not authorize a runtime or dependency upgrade.

1. Ask whether the work needs a singleton at all. If it can be partitioned by key, every instance owns a disjoint subset and the global singleton disappears. Running everywhere additionally requires concurrency-safe effects and acceptable duplicate load; repeatability alone does not prove either. One active worker can be a capacity ceiling; an elected coordinator can also delegate partitioned work.
2. State the failover budget as a number. Detection + election + warm-up is a period with no leader, user-visible if anything waits on the leader's work. That number sets the lease length, not the other way round.
3. Size the lease against the measured pause distribution, not a round number: a lease shorter than the worst stop-the-world pause or network blip produces failovers that are pure churn (pause-attribution). Arithmetic in references/lease-and-split-brain.md.
4. Write the leader loop to stop before its conservative local validity deadline. A single timed-out renewal is ambiguous and need not stop work immediately if sufficient lease budget remains; it must never extend that deadline. Stop admission early enough for in-flight work to quiesce, and do not wait to be told another leader won.
5. Prevent stale authority from violating safety. Enforce a monotonically increasing term/ fence at every mutable resource, put the effect and authority check in the same transaction, or ensure repeated and concurrent effects preserve the required invariant, including any reconciliation window. Local leader belief is never the enforcement boundary.
6. Handle the rolling deploy explicitly. Stop new work, quiesce or hand off in-flight work, persist a checkpoint, then release/transfer authority. Releasing first can overlap the successor with unfinished effects (kubernetes-service-lifecycle).
7. Instrument and test the split. Export an is_leader gauge per instance; in a test, partition the leader from the store and assert both that it stopped and that its late write was rejected.

Decision block

Elect a leader when:
- the work must happen once per interval across the fleet and cannot be partitioned — a
  reconciliation sweep, a global aggregate, a single outbound connection to a peer that admits
  only one client
- a duplicate run is expensive and the work can be fenced or made idempotent
- one instance's throughput is sufficient for the whole workload, now and after growth
Avoid electing when:
- the work is already idempotent and safe on every replica: coordination buys nothing
- one instance cannot keep up — a leader does not scale, and adding replicas adds standbys,
  not capacity
- the failover window (lease + election + warm-up) is longer than the work's tolerance for
  having no owner
Prefer instead when:
- the work is per-key and the key space can be split: partitioned ownership gives one owner per
  key, scales with instances, and needs no election (sharding-and-partitioning)
- what is actually wanted is "do not run this twice concurrently" for a scheduled job that is
  idempotent — a lease-per-execution row (ShedLock-style) is simpler and honest about its limits

Rules

• A leader stops before it can no longer prove its grant valid, not on the first failed renewal and not when another actor reports winning. Convert a successful grant response into a conservative monotonic deadline accounting for request/response uncertainty, clock- rate drift and quiescence time. Timeout never extends it.
• Split-brain is two instances acting as leader at once, and its cost is whatever the work does twice — a doubled batch, two conflicting reconciliations, two outbound connections a peer rejects. Name that cost in the design; it decides how much fencing is worth.
• A leader must be prepared to lose authority mid-operation. Surviving designs include resource-enforced term/fence, mutation and authority check in one transaction, or idempotent/reconcilable work. Cancellation alone cannot retract a committed remote effect.
• Export local role, term, lease-deadline margin, renewal result/latency and useful-work age. sum(is_leader) != 1 is a diagnostic of sampled local belief, not proof: scrape gaps and stale metrics lie. Alert primarily on no useful progress and fence rejections; compare multiple leaders by overlapping terms and resource-side evidence.
• Two peers cannot guarantee both exclusive authority and continued progress after either becomes unreachable in this failure model. Requiring both votes or standing down can preserve safety while losing availability; unilateral promotion cannot distinguish a failed peer from a partition without additional assumptions or an arbiter. Two instances electing through a quorum-backed store are fine — the store is the arbiter.
• Failover time is detection/remaining grant + election + state recovery + warm-up/backlog. With an exclusive lease, a successor may need to wait up to the remaining duration, not at least the full lease. Session failure detectors may react earlier. Measure time to first correct useful result, not time to set a leader flag.
• The lease-length trade is explicit: short leases give fast failover and false failovers under a GC pause or a network blip, each of which costs a warm-up and a burst of churn; long leases give stability and a longer outage. Pick from the pause distribution and the budget.
• ShedLock and equivalents are not leader election. They are "do not run this twice" mechanisms built on a database row with an expiry (lockAtMostFor), and the expiry is a lease with the usual defect: a node still executing after it expires is not stopped, and there is no fencing token. That is adequate for a job that is idempotent or tolerant of a skipped or duplicated run, and not adequate when a second concurrent run corrupts data — set lockAtMostFor above the job's worst observed duration and treat overlap as possible anyway.
• The Kubernetes Lease object is a renewable record of a holder identity and a duration, and the basis of the lease-based election controllers use. Same property: it establishes who should lead and does not stop a stalled former holder from writing.
• A rolling deploy can terminate the leader. On SIGTERM, stop admission; continue renewal only as needed for a bounded safe drain, then checkpoint and release authority. If the grace period expires, rely on fence/idempotency rather than an unsafe release. A deliberate handoff can reduce gaps but must use a new term and acknowledgement protocol.

Safety and liveness contract

Safety: no stale term can commit an effect that violates the invariant.
Liveness: when a quorum/store and at least one eligible member remain reachable long enough,
          a leader eventually performs useful work.

Election gives local role information; it does not automatically enforce safety at databases, object stores, brokers or third-party APIs. List every sink and show how it rejects stale terms or tolerates duplicate/concurrent effects. A highest-seen-term fence rejects old terms only after the new term is installed at that sink; it does not itself reject every post-expiry write. Establish the sink's authority transition before successor work, or atomically validate current authority with the effect when strict expiry exclusion is required. If a sink cannot enforce or tolerate the required invariant, election alone is insufficient.

References

• Leases, renewal and the split-brain window — renewal timing, the stop-acting rule as code on a monotonic clock, the split-brain sequence, lease selection against pauses and blips, the failover budget, and rolling-deploy behaviour. Read when choosing a lease duration or after a duplicate run.
• Mechanisms and alternatives — the ways to avoid electing at all, then coordination-store lease, Kubernetes Lease and ShedLock-style database rows compared on fencing, failover time, dependencies and what each is adequate for, with a decision block. Read when choosing a mechanism or reviewing one already in place.