# Async Messaging

> Build reliable event-driven flows with the Transactional Outbox pattern — write state and event in one transaction, relay asynchronously, achieve at-least-once delivery + consumer idempotency. Use when an action must reliably trigger downstream work, or when events are lost on crash (dual-write problem). Not for simple background work without state+event reliability (use background-jobs) or outbound HTTP webhook specifics (use webhook-design).

- Skill: `jaykim88/async-messaging` (Agent Skill)
- Install (CLI): `npx skillmds@latest add jaykim88/async-messaging`
- Raw SKILL.md: https://api.skillmd.com/api/skills/jaykim88/async-messaging/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Integrations & APIs
- License: MIT
- Author: JayKim88 (https://skillmd.com/u/jaykim88)
- Updated: 2026-09-10
- Page: https://skillmd.com/skills/jaykim88/async-messaging

---


# Async Messaging (Transactional Outbox)

## Purpose
Make "change state AND notify the world" reliable. The naive approach (write DB, then publish to a queue) loses events on crash between the two steps — the dual-write problem. The Outbox pattern solves it.

**Universal** — the Outbox pattern, the dual-write problem, and "exactly-once = at-least-once + consumer idempotency" are distributed-systems fundamentals independent of broker/language.

## Procedure

1. **Recognize the dual-write problem**
   - "Write DB, then publish to Kafka/Redis" has no atomicity — crash after DB write but before publish = lost event; crash after publish but before commit = phantom event
   - You cannot atomically write to two systems without a distributed transaction (which you should avoid)

2. **Apply the Transactional Outbox**
   - Write the business state change AND an `outbox` row in ONE local DB transaction (see `transaction-management`)
   - Both commit together or neither does — atomicity guaranteed by the single DB

3. **Relay the outbox asynchronously**
   - **Polling publisher**: a `background-jobs` task reads unsent outbox rows, publishes to the broker, marks sent
   - **CDC** (Change Data Capture, e.g., Debezium): tail the DB log, publish outbox inserts — lower latency, more infra
   - Start with polling; move to CDC only if latency demands it

4. **Design consumers for at-least-once (idempotency)**
   - The relay guarantees at-least-once → consumers WILL see duplicates
   - "Exactly-once" delivery is a myth; achieve effective exactly-once via at-least-once + idempotent consumers (dedupe on event id / idempotency key — see `resilience-patterns`)

4b. **Version events from day one (additive-only)**
   - Include `eventType` + `eventVersion` (or `schemaVersion`) on every payload — consumers will outlive any one publisher
   - **Additive-only changes** on a live event type: new fields nullable, never remove or repurpose; bump the version and run old+new in parallel during cutover. Treat events like a public API contract (see `api-contract`)
   - Never publish two semantically-different shapes under the same `eventType` + version

4c. **Dead-letter queue (DLQ) + poison-message handling**
   - A retried-N-times failure is a *poison message*: route to a DLQ after the retry budget rather than blocking the queue forever
   - DLQ needs alerting (a growing DLQ = silent failure) + a manual replay path with the original payload + reason
   - Cap retry attempts and use backoff + jitter (see `resilience-patterns` / `background-jobs`)

5. **Choose the broker by needs (criteria, not products)**
   - **Simple in-app queue** — low setup, good for in-process jobs
   - **Routing broker** — work queues + routing topologies, moderate scale
   - **High-throughput log** — event replay, multiple consumer groups, high volume
   - **Ordering needs a partition key**: same-entity events go to the same partition (`order_id`, `user_id`). Within a partition: ordered; across partitions: no guarantees. Choose the key by the invariant you actually need
   - Don't reach for a heavyweight log broker by default — it's heavy infra (specific products in Implementation)

5b. **Retain only what you need**
   - The outbox table grows on every write — schedule a cleanup job to delete (or archive) sent rows past N days, or it eats the DB
   - For broker retention: replay needs ≠ storage forever; tune by recovery window (e.g., 7 days for backfill, not 1 year "just in case")

6. **Validate (validation loop)**
   - Crash the service between state write and relay → verify the event is still delivered (outbox row persisted)
   - Deliver an event twice → verify the consumer processes it once (idempotent)
   - If an event is lost → the write wasn't in the same transaction as the outbox row; fix and re-test

## Anti-patterns

| ❌ Anti-pattern | ✅ Correct |
|---|---|
| Write DB then publish to broker (dual write) | Write state + outbox row in one transaction, relay async |
| Assuming the broker gives exactly-once | At-least-once + idempotent consumers |
| Publishing inside the business transaction (network I/O in tx) | Outbox row in tx; relay outside |
| Heavyweight log broker for a simple in-app job | Simple queue first; log broker only at real scale |
| Consumer with no dedup | Dedupe on event id |
| Two payload shapes under the same `eventType` (silent break) | `eventVersion` + additive-only changes; run old+new in parallel on cutover |
| Poison messages blocking the queue indefinitely | DLQ after retry budget; alert on DLQ growth; manual replay |
| Outbox table growing unbounded | Scheduled cleanup of sent rows past retention window |
| Ordering assumed without a partition key | Choose a partition key matching the invariant (same-entity events to the same partition) |

## Severity tiers

| Tier | Examples | Action SLA |
|---|---|---|
| **Critical** | Dual-write losing payment/order events on crash; consumer with no idempotency double-charging; breaking schema change shipped on a live `eventType` (consumers crash on next message) | Fix immediately |
| **Major** | Publishing inside a transaction (lock contention); no DLQ for poison messages; outbox table grows without cleanup (DB-eating); no partition key on a flow that requires ordering | Fix this sprint |
| **Minor** | Polling interval untuned; broker over-provisioned (Kafka for tiny load); broker retention not aligned with actual replay needs | Schedule within 2 sprints |

## Completion Criteria
- [ ] State change + event written in one transaction (Outbox)
- [ ] Relay (polling or CDC) publishes + marks sent
- [ ] Consumers idempotent (dedup on event id)
- [ ] Every event payload carries `eventVersion`; schema changes are additive-only
- [ ] DLQ wired + alerted on growth; manual replay path exists
- [ ] Outbox cleanup scheduled; broker retention matches the actual replay window
- [ ] Partition key chosen wherever ordering is required
- [ ] Broker choice justified by actual throughput/replay needs
- [ ] Crash + duplicate scenarios verified

## Output
- **Outbox table + relay**: schema + polling publisher (or CDC config)
- **Consumer handlers**: idempotent, with dedup
- **Architecture doc**: event catalog + delivery guarantees
- **Commit format**: `feat(events): add outbox for <event>` / `feat(events): idempotent consumer for <event>`

## Implementation

### TypeScript + Prisma + BullMQ/Redis (default)
- Outbox: `outbox(id, type, payload, created_at, sent_at)` written inside `prisma.$transaction` with the state change
- Relay: a BullMQ repeatable job polls `WHERE sent_at IS NULL`, publishes, sets `sent_at`
- Consumers: BullMQ workers with dedup table on event id
- Scale path: Redis Streams → Kafka (KafkaJS) if replay/throughput demands

### Other stacks
- **Python**: SQLAlchemy outbox + Celery relay; or Debezium CDC → Kafka
- **Go**: outbox + worker; Watermill library for messaging abstractions
- **Universal**: Outbox is DB-agnostic; Debezium provides CDC for Postgres/MySQL/Mongo; the at-least-once + idempotency principle is broker-independent

## Related skills
- `transaction-management` — the outbox row is written in the same transaction as the state change
- `background-jobs` — the outbox relay / consumers run as jobs
- `webhook-design` — outbound webhooks can be driven by the outbox

## Reference
- **Key insight encoded**: Solve the dual-write problem by writing business state and the event to an outbox table in one local transaction, then relay asynchronously (polling or CDC) — this is what makes at-least-once delivery achievable; "exactly-once" is really at-least-once + consumer idempotency. Treat events like a public API: version from day one, additive-only changes, partition key on flows that need ordering. Operational hygiene matters: a DLQ for poison messages + an outbox-cleanup job (else the outbox eats the DB).