Async Messaging

Async Messaging (Transactional Outbox)

Purpose

Make "change state AND notify the world" reliable. The naive approach (write DB, then publish to a queue) loses events on crash between the two steps — the dual-write problem. The Outbox pattern solves it.

Universal — the Outbox pattern, the dual-write problem, and "exactly-once = at-least-once + consumer idempotency" are distributed-systems fundamentals independent of broker/language.

Procedure

1. Recognize the dual-write problem

   • "Write DB, then publish to Kafka/Redis" has no atomicity — crash after DB write but before publish = lost event; crash after publish but before commit = phantom event
   • You cannot atomically write to two systems without a distributed transaction (which you should avoid)

2. Apply the Transactional Outbox

   • Write the business state change AND an outbox row in ONE local DB transaction (see transaction-management)
   • Both commit together or neither does — atomicity guaranteed by the single DB

3. Relay the outbox asynchronously

   • Polling publisher: a background-jobs task reads unsent outbox rows, publishes to the broker, marks sent
   • CDC (Change Data Capture, e.g., Debezium): tail the DB log, publish outbox inserts — lower latency, more infra
   • Start with polling; move to CDC only if latency demands it

4. Design consumers for at-least-once (idempotency)

   • The relay guarantees at-least-once → consumers WILL see duplicates
   • "Exactly-once" delivery is a myth; achieve effective exactly-once via at-least-once + idempotent consumers (dedupe on event id / idempotency key — see resilience-patterns)

4b. Version events from day one (additive-only)

• Include eventType + eventVersion (or schemaVersion) on every payload — consumers will outlive any one publisher
• Additive-only changes on a live event type: new fields nullable, never remove or repurpose; bump the version and run old+new in parallel during cutover. Treat events like a public API contract (see api-contract)
• Never publish two semantically-different shapes under the same eventType + version

4c. Dead-letter queue (DLQ) + poison-message handling

• A retried-N-times failure is a poison message: route to a DLQ after the retry budget rather than blocking the queue forever
• DLQ needs alerting (a growing DLQ = silent failure) + a manual replay path with the original payload + reason
• Cap retry attempts and use backoff + jitter (see resilience-patterns / background-jobs)

5. Choose the broker by needs (criteria, not products)
   • Simple in-app queue — low setup, good for in-process jobs
   • Routing broker — work queues + routing topologies, moderate scale
   • High-throughput log — event replay, multiple consumer groups, high volume
   • Ordering needs a partition key: same-entity events go to the same partition (order_id, user_id). Within a partition: ordered; across partitions: no guarantees. Choose the key by the invariant you actually need
   • Don't reach for a heavyweight log broker by default — it's heavy infra (specific products in Implementation)

5b. Retain only what you need

• The outbox table grows on every write — schedule a cleanup job to delete (or archive) sent rows past N days, or it eats the DB
• For broker retention: replay needs ≠ storage forever; tune by recovery window (e.g., 7 days for backfill, not 1 year "just in case")

6. Validate (validation loop)
   • Crash the service between state write and relay → verify the event is still delivered (outbox row persisted)
   • Deliver an event twice → verify the consumer processes it once (idempotent)
   • If an event is lost → the write wasn't in the same transaction as the outbox row; fix and re-test

Anti-patterns

Severity tiers

Completion Criteria

• State change + event written in one transaction (Outbox)
• Relay (polling or CDC) publishes + marks sent
• Consumers idempotent (dedup on event id)
• Every event payload carries eventVersion; schema changes are additive-only
• DLQ wired + alerted on growth; manual replay path exists
• Outbox cleanup scheduled; broker retention matches the actual replay window
• Partition key chosen wherever ordering is required
• Broker choice justified by actual throughput/replay needs
• Crash + duplicate scenarios verified

Output

• Outbox table + relay: schema + polling publisher (or CDC config)
• Consumer handlers: idempotent, with dedup
• Architecture doc: event catalog + delivery guarantees
• Commit format: feat(events): add outbox for <event> / feat(events): idempotent consumer for <event>

Implementation

TypeScript + Prisma + BullMQ/Redis (default)

• Outbox: outbox(id, type, payload, created_at, sent_at) written inside prisma.$transaction with the state change
• Relay: a BullMQ repeatable job polls WHERE sent_at IS NULL, publishes, sets sent_at
• Consumers: BullMQ workers with dedup table on event id
• Scale path: Redis Streams → Kafka (KafkaJS) if replay/throughput demands

Other stacks

• Python: SQLAlchemy outbox + Celery relay; or Debezium CDC → Kafka
• Go: outbox + worker; Watermill library for messaging abstractions
• Universal: Outbox is DB-agnostic; Debezium provides CDC for Postgres/MySQL/Mongo; the at-least-once + idempotency principle is broker-independent

Related skills

• transaction-management — the outbox row is written in the same transaction as the state change
• background-jobs — the outbox relay / consumers run as jobs
• webhook-design — outbound webhooks can be driven by the outbox

Reference

• Key insight encoded: Solve the dual-write problem by writing business state and the event to an outbox table in one local transaction, then relay asynchronously (polling or CDC) — this is what makes at-least-once delivery achievable; "exactly-once" is really at-least-once + consumer idempotency. Treat events like a public API: version from day one, additive-only changes, partition key on flows that need ordering. Operational hygiene matters: a DLQ for poison messages + an outbox-cleanup job (else the outbox eats the DB).