Design a Dead-Letter Queue
A record that fails processing should leave the pipeline with everything needed to understand and re-run it, not vanish into a log line. A DLQ is a durable store plus the metadata that makes a replay possible.
Procedure
- Store the full original payload, serialized as received, not just the error message. Without the payload there is nothing to replay.
- Attach metadata:
job_name, source, partition_or_window, failed_at, attempt_count, error_class, error_message, and the record's natural key.
- Key on natural key plus arrival time so a repeated failure of the same record shows as a rising
attempt_count, not endless duplicate rows.
- Separate transient failures (timeout, rate limit) from permanent ones (schema invalid, referential failure). Only permanent failures belong in the DLQ; retry transient ones first.
- Set a max attempt count; past it, move the record to a terminal state and alert.
- Make replay first-class: a tool that reads DLQ rows, re-runs them through the same validation and load, and clears each on success.
- Bound the queue: a retention policy and an alarm when depth grows past a threshold (for example >100 rows or +20% per hour).
- Never let growth be silent. A pipeline that always succeeds while dropping 5% of records is worse than one that fails.
- Include a
state column (pending, retrying, terminal) so consumers of the DLQ can filter what is actionable.
- Store the schema version of the payload, so a replay months later parses it with the code that produced it.
Pitfalls
- Logging the error but not the payload makes the DLQ useless for replay.
- A DLQ with no consumer grows forever and is discovered only when a table is missing rows.
- Retrying a permanently malformed record to max attempts wastes the budget meant for transient errors.
- No dedupe key means a record that fails every run fills the queue with duplicates.
- Storing only the error string loses the field values that caused it.
- Treating the DLQ as a graveyard: nothing is fixed, the gap just becomes permanent.
- A DLQ on the same storage as the target means a target outage also loses the failures.
- Storing a PoC payload without its schema version makes replay impossible after the code moves on.
Verification
kafka-consumer-groups --bootstrap-server $BROKER --describe --group dlq-inspector
psql -c "SELECT count(*), max(failed_at) FROM dead_letters"
Depth is under threshold and the oldest entry is recent, meaning someone is draining it. Report DLQ depth, the top error classes, and the path of the replay tool.
1---2name: design-a-dead-letter-queue3description: Use when records fail validation or processing and must not be silently dropped or block the pipeline. Captures each failure with payload, error, and origin so it can be inspected and replayed.4metadata:5 source: https://skillmd.com/skills/jiayaoqijia/design-a-dead-letter-queue6---78# Design a Dead-Letter Queue910A record that fails processing should leave the pipeline with everything needed to understand and re-run it, not vanish into a log line. A DLQ is a durable store plus the metadata that makes a replay possible.1112## Procedure13141. Store the full original payload, serialized as received, not just the error message. Without the payload there is nothing to replay.152. Attach metadata: `job_name`, `source`, `partition_or_window`, `failed_at`, `attempt_count`, `error_class`, `error_message`, and the record's natural key.163. Key on natural key plus arrival time so a repeated failure of the same record shows as a rising `attempt_count`, not endless duplicate rows.174. Separate transient failures (timeout, rate limit) from permanent ones (schema invalid, referential failure). Only permanent failures belong in the DLQ; retry transient ones first.185. Set a max attempt count; past it, move the record to a terminal state and alert.196. Make replay first-class: a tool that reads DLQ rows, re-runs them through the same validation and load, and clears each on success.207. Bound the queue: a retention policy and an alarm when depth grows past a threshold (for example >100 rows or +20% per hour).218. Never let growth be silent. A pipeline that always succeeds while dropping 5% of records is worse than one that fails.229. Include a `state` column (`pending`, `retrying`, `terminal`) so consumers of the DLQ can filter what is actionable.2310. Store the schema version of the payload, so a replay months later parses it with the code that produced it.2425## Pitfalls2627- Logging the error but not the payload makes the DLQ useless for replay.28- A DLQ with no consumer grows forever and is discovered only when a table is missing rows.29- Retrying a permanently malformed record to max attempts wastes the budget meant for transient errors.30- No dedupe key means a record that fails every run fills the queue with duplicates.31- Storing only the error string loses the field values that caused it.32- Treating the DLQ as a graveyard: nothing is fixed, the gap just becomes permanent.33- A DLQ on the same storage as the target means a target outage also loses the failures.34- Storing a PoC payload without its schema version makes replay impossible after the code moves on.3536## Verification3738```sh39kafka-consumer-groups --bootstrap-server $BROKER --describe --group dlq-inspector40psql -c "SELECT count(*), max(failed_at) FROM dead_letters"41```4243Depth is under threshold and the oldest entry is recent, meaning someone is draining it. Report DLQ depth, the top error classes, and the path of the replay tool.