ZeroMQ Operations
Goal
Debug and operate ZeroMQ systems by observing message flow, queue pressure, peer state, and lifecycle transitions instead of guessing from connection state.
Workflow
- Reproduce with the smallest topology and log protocol-level send/receive events with monotonic timestamps, peer IDs, generations, and message IDs.
- Check startup ordering/handshake before blaming message loss.
- Check subscription filters and routing identities.
- Check HWM, send/receive timeouts, EAGAIN/drop behavior, and application processing rate; never treat configured HWM as an exact network queue size.
- Confirm one application thread owns each classic ZeroMQ socket.
- Make shutdown explicit: stop producers, drain/cancel intentionally, close sockets with chosen linger semantics, then terminate context.
- Use socket monitoring for transport lifecycle and application traces for protocol semantics.
- Randomize churn tests: connect, disconnect, restart, endpoint reuse, delay, malformed messages, slow peers, and process death.
- Run many isolated peer actors in one test process when practical.
- Turn every rare assertion/race into a deterministic regression test.
Rules
- Assertions express invariants in tests/internal code; they are not user-facing error handling.
- A single blocked peer must not block the whole routing loop.
- Bound queues, retries, timers, message sizes, and per-peer state.
- Prefer message passing between threads over shared mutable socket state.
- Verify whether a chosen modern socket type is thread-safe and stable/draft in the deployed runtime before relying on it.