Retry And Backoff Patterns

Retry and Backoff Patterns

This skill activates when a practitioner needs to implement resilient retry logic for Apex callouts — covering exponential backoff with jitter, idempotency via External Id upserts, dead-letter queue handling, and circuit breaker patterns using Custom Metadata. It does not cover native platform retry mechanisms (Outbound Messages, Platform Events) beyond documenting them as alternatives.

Before Starting

Gather this context before working on anything in this domain:

• Execution context of the callout: Is it currently synchronous (trigger, Visualforce, LWC controller)? If so, a retry loop in the same transaction is not viable — the callout must be moved to a Queueable first.
• Idempotency of the target system: Can the same payload be safely sent twice? If not, an External Id + upsert guard is mandatory before any retry is safe.
• Governor limits already consumed: Each Queueable re-enqueue counts against the org's daily async Apex limit (250,000 per 24 hours for most orgs). A burst retry storm can exhaust this budget.

Core Concepts

No Thread.sleep() in Apex — Use the Queueable Delay Parameter

Apex has no Thread.sleep(), and a synchronous retry loop cannot introduce delay. But Apex can schedule a delayed retry explicitly — the claim that it can only hope for scheduler jitter is out of date. System.enqueueJob takes an optional delay: "use the System.enqueueJob(queueable, delay) method to add queueable jobs to the asynchronous execution queue with a specified minimum delay (0–10 minutes)."

System.enqueueJob(new MyQueueableClass(), 5);            // 5-minute minimum

AsyncOptions opts = new AsyncOptions();                   // equivalent form
opts.MinimumQueueableDelayInMinutes = 5;
System.enqueueJob(new RetryCalloutJob(orderId, retryCount), opts);

• The delay is a minimum, not a guarantee, capped at 10 minutes. Longer backoff needs Scheduled Apex.
• Admins can set a default org-wide enqueue delay (1–600 seconds) in Apex Settings for jobs enqueued without one. An explicit delay "ignores any org-wide enqueue delay setting" — so a job you did not give a delay to may still be delayed by org config.

Exponential Backoff with Jitter

Exponential backoff calculates the delay between retries as baseDelay * 2^retryCount. Without jitter, every concurrent failing job wakes at the same moment and hammers the external system simultaneously — the thundering-herd problem. Adding jitter (+ (Math.random() * baseDelay)) scatters retry times across the time window.

Because System.enqueueJob(queueable, delay) takes whole minutes, 0–10, convert the computed backoff to minutes and clamp it — do not pass a seconds value into a minutes parameter:

Integer delayMinutes = (Integer) Math.min(
    10, Math.max(0, Math.ceil((baseDelaySeconds * Math.pow(2, retryCount) + jitter) / 60))
);
System.enqueueJob(new RetryCalloutJob(payload, retryCount + 1, maxRetries), delayMinutes);

The 10-minute ceiling is the real design constraint: 1/2/4/8 minutes fits, anything past that flattens to 10. Log the calculated delay alongside the clamped one so a flattened retry is visible in the data rather than looking like a scheduler anomaly.

Idempotency Key via External Id Upsert

On retry, the same payload may be sent and accepted by the external system, but if the original request succeeded and only the response was lost (network timeout), a naive retry creates a duplicate. The defense is to assign a stable idempotency key before the first attempt — typically a UUID stored in External_Id__c on the driving record — and pass it as a request header or body field. On the Salesforce side, use upsert on External_Id__c to prevent duplicate SObject creation from reprocessing. The key must survive retries unchanged.

Max Retry Guard and Dead-Letter Queue

Every retry implementation must have an explicit maximum retry count (3–5 is typical). When retryCount >= maxRetries, the job must not re-enqueue. Instead, write a dead-letter record — a Failed_Integration_Log__c custom object capturing the payload, error message, HTTP status, retry count, and timestamp — and optionally fire an alert (Platform Event or email). Without this guard, a permanently failing integration exhausts the async Apex limit silently.

Common Patterns

Pattern 1: Queueable Retry Chain with Exponential Backoff

When to use: An Apex callout fails with a transient error (HTTP 429, 503, or timeout) and must be retried automatically with increasing delay.

How it works:

1. The initial callout attempt is made from a Queueable execute() method.
2. On failure, increment retryCount on the job instance. Calculate delaySeconds = baseDelay * Math.pow(2, retryCount) + (Math.random() * baseDelay), then convert to whole minutes clamped to 0–10.
3. If retryCount < maxRetries, call System.enqueueJob(new RetryCalloutJob(payload, retryCount, maxRetries), delayMinutes) — pass the delay rather than relying on scheduler jitter.
4. If retryCount >= maxRetries, write a Failed_Integration_Log__c record and stop.

Why not a for-loop retry in the same transaction: Each Http.send() call consumes a callout from the 100-per-transaction limit. More critically, you cannot introduce real delay in a synchronous loop, so you just hammer the endpoint repeatedly in milliseconds — worse than no retry.

Pattern 2: Circuit Breaker via Custom Metadata

When to use: An external system is degraded for extended periods. Rather than retrying every call and burning async limits, a circuit breaker detects open-circuit state and skips the callout entirely until the system recovers.

How it works:

1. Create a Circuit_Breaker_Config__c Custom Metadata record with fields: Is_Open__c (Boolean), Opened_At__c (DateTime), Cool_Down_Minutes__c (Number).
2. At the start of the Queueable execute(), query (or cache via a static variable) the CMDT record.
3. If Is_Open__c = true AND Opened_At__c + Cool_Down_Minutes__c > now, skip the callout and write a log entry.
4. If the cool-down has elapsed, treat the circuit as half-open: attempt one callout. Success → flip Is_Open__c = false (via an Apex update or a named flow). Failure → keep open and reset Opened_At__c.
5. Toggling Is_Open__c manually also enables operators to manually open or close the circuit without a code deploy.

Why not a flag on a custom object: CMDT records are cached at the platform level and do not consume SOQL queries per transaction (after the first load in a request). Custom objects do.

Pattern 3: Idempotency Key Guard

When to use: The external system does not natively deduplicate requests (no idempotency-key header support), and a duplicate call creates duplicate data.

How it works:

1. Before the first callout attempt, generate String idempotencyKey = [String UUID or ExternalId from driving record].
2. Include the key in the request body or a custom header (X-Idempotency-Key).
3. On the Salesforce side, use Database.upsert(record, Schema.SObject.Fields.External_Id__c, false) to prevent double-insert on reprocessing.
4. Log the key on Failed_Integration_Log__c so support teams can trace retried requests.

Decision Guidance

Recommended Workflow

Step-by-step instructions for an AI agent or practitioner implementing retry logic for an Apex callout:

1. Confirm execution context: Verify that the callout is already in a Queueable or Batch context. If it originates from a trigger or controller, extract it to a Queueable job first before adding retry logic.
2. Define retry parameters: Decide maxRetries (3–5), baseDelaySeconds (1–5), and maxDelaySeconds (30–60). Document these as Custom Metadata fields in Retry_Config__mdt so they can be adjusted without a deploy.
3. Add retry counter and idempotency key fields: Add Retry_Count__c (Integer, default 0) and Integration_Idempotency_Key__c (Text, External Id) to the driving SObject. Populate the idempotency key before the first enqueue.
4. Implement the Queueable retry chain: In the execute() method, wrap the callout in a try-catch. On caught exceptions or non-2xx responses, increment the counter, calculate the backoff, convert it to whole minutes clamped to 0–10, and re-enqueue with System.enqueueJob(job, delayMinutes) if under the limit. Log both the calculated and the clamped delay.
5. Implement the dead-letter path: When retryCount >= maxRetries, insert a Failed_Integration_Log__c record with payload, error, HTTP status, retry count, and timestamp. Optionally publish a Platform Event to notify an operations flow.
6. Add the circuit breaker check: At the start of execute(), read Circuit_Breaker_Config__mdt. If the circuit is open and cool-down has not elapsed, log and return without attempting the callout.
7. Test failure modes explicitly: Write Apex tests that mock HTTP 429, 503, and timeout responses. Assert that retry count increments, dead-letter records are written at max retries, and idempotency keys are preserved across re-enqueues.

Review Checklist

Run through these before marking integration retry work complete:

• Callout is in a Queueable or Batch context — no synchronous retry loops
• maxRetries is explicitly defined and enforced — no unbounded retry possible
• Exponential backoff formula is present with jitter (Math.random())
• Idempotency key (External_Id__c) is generated before first attempt and passed in the request
• Dead-letter path writes Failed_Integration_Log__c when max retries exceeded
• Circuit breaker CMDT flag is checked before each callout attempt
• Apex tests mock all failure scenarios (429, 503, timeout) and assert dead-letter creation
• Retry config (maxRetries, baseDelay) is in Custom Metadata — not hardcoded

Salesforce-Specific Gotchas

Non-obvious platform behaviors that cause real production problems:

1. The enqueue delay is a floor, not an appointment — System.enqueueJob(queueable, delay) accepts a "specified minimum delay (0–10 minutes)". The platform will not run the job before that delay, but does not promise to run it at that delay — execution still waits for a worker. So: passing >10 or a seconds-scaled value is a bug, not a longer wait; and a job enqueued with no delay may still be delayed by the org-wide default (1–600 seconds) in Apex Settings, which an explicit delay "ignores". The retry interval is bounded below and approximate above.
2. Callout-after-DML rule applies inside Queueable — If you write a Failed_Integration_Log__c record (DML) and then attempt a callout in the same execute() method, you hit the "callout after uncommitted work" exception. Always perform DML after the callout block, or use a separate inner Queueable for the logging path.
3. Daily async Apex limit is shared across all jobs — Unlimited retry storms (e.g., a broken endpoint during peak processing) can consume the 250,000 daily Queueable executions, blocking all other background processing. The maxRetries guard and circuit breaker are critical safety valves, not optional.
4. System.enqueueJob() is limited to 50 per transaction — A batch of failing records all trying to enqueue retry jobs in the same transaction will hit this limit. Design retry logic so each job re-enqueues itself (1 per transaction), not so a parent job enqueues N children.
5. Native platform retries are separate from Apex retries — Outbound Messages retry automatically for up to 24 hours at platform-managed intervals. Platform Events can be replayed for up to 3 days. Do not add Apex retry logic on top of these — it results in double-processing.

Output Artifacts

Related Skills

• callout-limits-and-async-patterns — governor limits on callouts (100 per transaction, daily async limits); complements retry design
• apex-queueable-patterns — Queueable chaining patterns without callouts; prerequisite for understanding the async execution model
• named-credentials-setup — configuring Named Credentials for the endpoint used in retried callouts
• integration-framework-design — higher-level integration architecture decisions where retry strategy is one component