# Deliver Durably

> Use when building any fire-and-forget async delivery or dispatch (send a message, notify, post, trigger a pipeline, enqueue a task) that must survive a process crash or restart — or when a delivery "succeeded" in-session but never arrived.

- Skill: `oimiragieo/deliver-durably` (Agent Skill)
- Install (CLI): `npx skillmds@latest add oimiragieo/deliver-durably`
- Raw SKILL.md: https://api.skillmd.com/api/skills/oimiragieo/deliver-durably/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: DevOps & Infra
- Author: oimiragieo (https://skillmd.com/u/oimiragieo)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/oimiragieo/deliver-durably

---


# Deliver Durably

## Overview

A handler that writes its database and *then* publishes to a broker/channel in two separate steps has **no atomicity across them** — a crash between the two silently loses the message. "Fire and forget" backed only by an in-memory callback is exactly this bug: the work finishes, but nothing durable remembers a delivery is owed.

## When to use

- Any async dispatch that must reach a user/system later (Telegram/email/webhook/pipeline trigger).
- Any restartable worker whose result must survive a process bounce.
- A delivery that "worked" in the session logs but the recipient never got it.

## The technique

1. **Transactional outbox.** Persist a "delivery owed" row in the SAME transaction as the business write, **at dispatch — not at completion**: `(id, target, channel, payload_ref, idempotency_key, created_at, delivered_at NULL, attempts)`. The run_id→recipient mapping must survive even if the in-process task dies.
2. **Relay + reconciliation poller.** A restartable poller drains `WHERE delivered_at IS NULL`, delivers, marks done (at-least-once). A longer-interval reconciler replays rows older than N minutes. After N failed attempts → **`quarantined`, never silently dropped.** Distinguish a transient delivery failure from a truly orphaned task before marking anything failed.
3. **Idempotency.** The consumer checks `idempotency_key` (`INSERT ... ON CONFLICT DO NOTHING`) before acting, so a redelivery is a no-op. Source the payload from durable state (e.g. the job store) so a finished job is always re-deliverable.

## Common mistakes

- Persisting "owed" at *completion* instead of *dispatch* — a crash before completion loses the mapping.
- An in-memory callback / detached task as the only delivery path (GC or a restart strands it).
- Non-idempotent retry (double-sends on redelivery).
- Marking delivered before the send is confirmed; treating a flaky network error as a terminal failure.

## Real-world impact

Example: a background job *completed* successfully but its notification **never reached the user** — delivery hung on one detached in-process task with no durable "delivery owed" record, so a read-timeout stranded it. The fix: persist the owed delivery at dispatch + a reconciliation poller that re-delivers from the durable job state. Related failure class: a health check reports OK (e.g. `/healthz` 200) while the poll loop is dead, so nothing recovers the owed reply — reconcile from durable state, don't trust a liveness signal.

Sources: microservices.io/patterns/data/transactional-outbox.html.