Make operations idempotent and retries safe
Networks deliver at least once or not at all. Exactly-once effect comes only from an
idempotent operation plus retries. The hard part is not the happy path; it is two copies
of the same request arriving at once, and a crash between "did it" and "wrote it down".
Sources: IETF draft-ietf-httpapi-idempotency-key-header; Stripe's idempotency design
(Brandur Leach, "Implementing Stripe-like idempotency keys in Postgres"); AWS Builders'
Library "Timeouts, retries, and backoff with jitter".
Steps
- Classify the operation. Naturally idempotent (PUT that sets state, DELETE),
idempotent with a key (create order, send email, charge card), or deliberately
non-idempotent (say why). Call
search_lore for the repo's key header and retry policy.
- Choose the key. API creates: client-supplied
Idempotency-Key header (a string,
UUID recommended). Webhooks and queue consumers: the event/message id. Internal jobs:
a natural key (order id + step). Scope it to the caller: the store key is
(caller, key) so two tenants cannot collide.
- Store the outcome with a fingerprint. A table with a UNIQUE constraint on
(caller, key) holding a request fingerprint (hash of method, path and canonical
body), a status (in_progress, completed), the stored response (status + body),
and locked_until/created_at. Replay rules (IETF draft semantics):
- key missing on an endpoint that requires it →
400;
- same key, different fingerprint →
422, never a silent success;
- same key while the first is still
in_progress → 409 (client retries later);
- same key, completed → return the stored response byte-for-byte, including stored
errors from after execution began (Stripe stores
500s too). Validation errors
raised before execution are not stored, so a corrected retry works.
- Claim the key atomically, never check-then-insert.
INSERT … ON CONFLICT DO NOTHING RETURNING (Postgres) / INSERT IGNORE + affected-rows (MySQL) / a unique
index violation you catch. Whoever inserts owns the execution; everyone else reads the
row. A SELECT followed by INSERT is a race that double-executes under load.
- Make local effects atomic with the record. When the effect is a database write,
do it in the same transaction that sets
completed and stores the response.
- Bracket external effects with recovery points. For a charge, email or third-party
call you cannot enlist in the transaction: commit
in_progress with the downstream
idempotency key first, call out passing that key, then commit completed. A crash
leaves in_progress past its locked_until; a reconciler (or the next retry) resumes
by re-calling the dependency with the same downstream key, which collapses to one
effect. Taking over a stale in_progress row is a time lease, so fence it: the
takeover bumps an attempt counter, and the final completed write is conditional on
it (… WHERE key = ? AND attempt = ?, 0 rows → roll back and stop), so a paused original holder
cannot overwrite the recovered result (the concurrency-and-async skill, step 5).
Emit follow-on messages through a transactional outbox, not directly.
- Retry only what is safe, at one layer. Retry idempotent or keyed calls on
transient failures only: timeouts, connection resets,
502/503/504, and 429/503
honouring Retry-After. Never retry other 4xx. Backoff with full jitter:
sleep = random(0, min(cap, base * 2^attempt)); cap attempts and set a total
deadline. Retry at a single layer of the stack (3 retries at each of 5 layers is 243×
load) and bound retries with a retry budget or token bucket so an outage is not
amplified. Timeouts come from measured latency (e.g. above p99.9), never infinite.
- Expire keys deliberately. Keep records at least as long as clients retry (Stripe:
24 h), sweep them with a job (the
scheduled-jobs skill), and document the window in
the API contract (the openapi-contract skill).
Tests (each exercises the guarantee itself)
- Same key twice, sequentially → one effect, identical response (status and body).
- Concurrency: N (≥ 20) parallel requests with the same key → exactly one effect
(count rows/charges on a fake); every response is either
409 or byte-identical to
the one original success; none 500. Run it with real parallelism against the real database, not mocks (the
concurrency-and-async skill's step 9 recipe).
- Same key, different body →
422, no second effect.
- Crash recovery: kill (throw) after the external call but before
completed;
the reconciler/retry finishes with exactly one downstream effect.
- Retry policy with an injected clock and random source: stops at the cap, respects
Retry-After, never retries a 400, surfaces the final error.
Done when
The concurrent-same-key test and the crash-recovery test pass alongside the sequential
ones, and each acceptance criterion has evidence via the record-evidence skill.
Review checklist
Capture lore
This skill is one of the places durable, reusable knowledge naturally surfaces:
While making the operation safe to repeat you learn the idempotency-key header, the outcome table and the retry policy the repo standardises on. That kind of fact is lore. Capture it via the lore-capture
protocol in your brief (CLAUDE.factory.md, step 11 "Memory contribution"):
call the Memory MCP suggest_lore once at the close of your work — reusable
conventions, gotchas, decisions, and boundaries only, never per-ticket trivia.
1---2name: idempotency-and-retries3description: Use when a ticket makes an operation safe to repeat or adds retries — payment or order creation that must not double-charge, a webhook handler that receives duplicates, a client retry policy, an outbound call to a flaky dependency — and the design must guarantee exactly-once effect under at-least-once delivery. Invoke for "it charged twice", "add retries", "handle duplicate events", or any side effect reached over a network.4stack: [node, python, go, java, rust, csharp, kotlin, ruby]5area: backend6metadata:7 source: https://skillmd.com/skills/tmj-90/idempotency-and-retries8---910# Make operations idempotent and retries safe1112Networks deliver at least once or not at all. Exactly-once _effect_ comes only from an13idempotent operation plus retries. The hard part is not the happy path; it is two copies14of the same request arriving at once, and a crash between "did it" and "wrote it down".15Sources: IETF draft-ietf-httpapi-idempotency-key-header; Stripe's idempotency design16(Brandur Leach, "Implementing Stripe-like idempotency keys in Postgres"); AWS Builders'17Library "Timeouts, retries, and backoff with jitter".1819## Steps20211. **Classify the operation.** Naturally idempotent (PUT that sets state, DELETE),22 idempotent with a key (create order, send email, charge card), or deliberately23 non-idempotent (say why). Call `search_lore` for the repo's key header and retry policy.242. **Choose the key.** API creates: client-supplied `Idempotency-Key` header (a string,25 UUID recommended). Webhooks and queue consumers: the event/message id. Internal jobs:26 a natural key (order id + step). Scope it to the caller: the store key is27 `(caller, key)` so two tenants cannot collide.283. **Store the outcome with a fingerprint.** A table with a UNIQUE constraint on29 `(caller, key)` holding a request fingerprint (hash of method, path and canonical30 body), a status (`in_progress`, `completed`), the stored response (status + body),31 and `locked_until`/`created_at`. Replay rules (IETF draft semantics):32 - key missing on an endpoint that requires it → `400`;33 - same key, different fingerprint → `422`, never a silent success;34 - same key while the first is still `in_progress` → `409` (client retries later);35 - same key, completed → return the stored response byte-for-byte, including stored36 errors from after execution began (Stripe stores `500`s too). Validation errors37 raised before execution are not stored, so a corrected retry works.384. **Claim the key atomically, never check-then-insert.** `INSERT … ON CONFLICT DO39 NOTHING RETURNING` (Postgres) / `INSERT IGNORE` + affected-rows (MySQL) / a unique40 index violation you catch. Whoever inserts owns the execution; everyone else reads the41 row. A `SELECT` followed by `INSERT` is a race that double-executes under load.425. **Make local effects atomic with the record.** When the effect is a database write,43 do it in the same transaction that sets `completed` and stores the response.446. **Bracket external effects with recovery points.** For a charge, email or third-party45 call you cannot enlist in the transaction: commit `in_progress` with the downstream46 idempotency key first, call out passing that key, then commit `completed`. A crash47 leaves `in_progress` past its `locked_until`; a reconciler (or the next retry) resumes48 by re-calling the dependency with the same downstream key, which collapses to one49 effect. Taking over a stale `in_progress` row is a time lease, so fence it: the50 takeover bumps an attempt counter, and the final `completed` write is conditional on51 it (`… WHERE key = ? AND attempt = ?`, 0 rows → roll back and stop), so a paused original holder52 cannot overwrite the recovered result (the `concurrency-and-async` skill, step 5).53 Emit follow-on messages through a transactional outbox, not directly.547. **Retry only what is safe, at one layer.** Retry idempotent or keyed calls on55 transient failures only: timeouts, connection resets, `502/503/504`, and `429`/`503`56 honouring `Retry-After`. Never retry other 4xx. Backoff with full jitter:57 `sleep = random(0, min(cap, base * 2^attempt))`; cap attempts and set a total58 deadline. Retry at a single layer of the stack (3 retries at each of 5 layers is 243×59 load) and bound retries with a retry budget or token bucket so an outage is not60 amplified. Timeouts come from measured latency (e.g. above p99.9), never infinite.618. **Expire keys deliberately.** Keep records at least as long as clients retry (Stripe:62 24 h), sweep them with a job (the `scheduled-jobs` skill), and document the window in63 the API contract (the `openapi-contract` skill).6465## Tests (each exercises the guarantee itself)6667- Same key twice, sequentially → one effect, identical response (status and body).68- **Concurrency:** N (≥ 20) parallel requests with the same key → exactly one effect69 (count rows/charges on a fake); every response is either `409` or byte-identical to70 the one original success; none `500`. Run it with real parallelism against the real database, not mocks (the71 `concurrency-and-async` skill's step 9 recipe).72- Same key, different body → `422`, no second effect.73- **Crash recovery:** kill (throw) after the external call but before `completed`;74 the reconciler/retry finishes with exactly one downstream effect.75- Retry policy with an injected clock and random source: stops at the cap, respects76 `Retry-After`, never retries a `400`, surfaces the final error.7778## Done when7980The concurrent-same-key test and the crash-recovery test pass alongside the sequential81ones, and each acceptance criterion has evidence via the `record-evidence` skill.8283## Review checklist8485- [ ] Every networked side effect has a key strategy; keys are caller-scoped.86- [ ] Claim is an atomic insert on a unique constraint, not a read-then-write.87- [ ] Fingerprint mismatch → `422`; in-flight → `409`; completed → stored response.88- [ ] External effects bracketed by recovery points; downstream key propagated.89- [ ] Retries: transient only, full jitter, capped, deadline, single layer.90- [ ] Concurrent and crash tests exist, not only a sequential double call.9192## Capture lore9394This skill is one of the places durable, reusable knowledge naturally surfaces:95**While making the operation safe to repeat you learn the idempotency-key header, the outcome table and the retry policy the repo standardises on.** That kind of fact is *lore*. Capture it via the **lore-capture96protocol in your brief** (`CLAUDE.factory.md`, step 11 "Memory contribution"):97call the Memory MCP `suggest_lore` once at the close of your work — reusable98conventions, gotchas, decisions, and boundaries only, never per-ticket trivia.