oimiragieo Deliver Durably
Overview
A handler that writes its database and then publishes to a broker/channel in two separate steps has no atomicity across them — a crash between the two silently loses the message. "Fire and forget" backed only by an in-memory callback is exactly this bug: the work finishes, but nothing durable remembers a delivery is owed.
When to use
- Any async dispatch that must reach a user/system later (Telegram/email/webhook/pipeline trigger).
- Any restartable worker whose result must survive a process bounce.
- A delivery that "worked" in the session logs but the recipient never got it.
The technique
- Transactional outbox. Persist a "delivery owed" row in the SAME transaction as the business write, at dispatch — not at completion:
(id, target, channel, payload_ref, idempotency_key, created_at, delivered_at NULL, attempts). The run_id→recipient mapping must survive even if the in-process task dies. - Relay + reconciliation poller. A restartable poller drains
WHERE delivered_at IS NULL, delivers, marks done (at-least-once). A longer-interval reconciler replays rows older than N minutes. After N failed attempts →quarantined, never silently dropped. Distinguish a transient delivery failure from a truly orphaned task before marking anything failed. - Idempotency. The consumer checks
idempotency_key(INSERT ... ON CONFLICT DO NOTHING) before acting, so a redelivery is a no-op. Source the payload from durable state (e.g. the job store) so a finished job is always re-deliverable.
Common mistakes
- Persisting "owed" at completion instead of dispatch — a crash before completion loses the mapping.
- An in-memory callback / detached task as the only delivery path (GC or a restart strands it).
- Non-idempotent retry (double-sends on redelivery).
- Marking delivered before the send is confirmed; treating a flaky network error as a terminal failure.
Real-world impact
Example: a background job completed successfully but its notification never reached the user — delivery hung on one detached in-process task with no durable "delivery owed" record, so a read-timeout stranded it. The fix: persist the owed delivery at dispatch + a reconciliation poller that re-delivers from the durable job state. Related failure class: a health check reports OK (e.g. /healthz 200) while the poll loop is dead, so nothing recovers the owed reply — reconcile from durable state, don't trust a liveness signal.
Sources: microservices.io/patterns/data/transactional-outbox.html.