ZeroMQ Reliability
Goal
Choose the least-complex recovery mechanism that handles the failures the application actually needs to survive.
Workflow
- Enumerate failures separately: process crash, restart, busy loop, overload, link loss, partition, duplicate request, stale reply, broker loss, and storage loss.
- Classify each operation as idempotent or non-idempotent.
- Give retried non-idempotent work a stable request ID and a dedupe/reply policy before enabling automatic retries.
- Bound every wait with a deadline or cancellation path.
- Start with simple timeout + retry/failover; add heartbeats only when they detect a failure faster or more accurately than ordinary traffic.
- Count any valid peer traffic as liveness when the protocol permits it.
- Use reconnect backoff and jitter; do not spin on a dead endpoint.
- Recreate strict-state sockets after protocol state becomes unusable.
- Persist requests/results only when disconnected or durable execution is a requirement.
- Test crash points before receive, during work, after side effect, before reply, and during reconnect.
Rules
- Reconnect is transport behavior, not application-level exactly-once delivery.
- Never retry a side effect blindly.
- Sequence or generation-fence stale replies and state updates.
- Prefer a small pool/failover set over elaborate HA unless requirements demand consensus-level behavior.
- Treat Binary Star and other guide-era HA examples as teaching patterns, not substitutes for modern consensus when split-brain safety is required.