Apex Callout Retry And Resilience

Apex Callout Retry and Resilience

Activate when an Apex integration must survive transient failures from a downstream system: 5xx errors, network timeouts, 429 rate limits, brief endpoint outages. This skill is the strategy layer — it tells you when to retry, how many times, on what schedule, when to stop, and where to park failures. The HTTP plumbing itself lives in templates/apex/HttpClient.cls.

Before Starting

• Confirm the operation is idempotent (safe to repeat) before retrying writes. If not, an Idempotency-Key contract with the downstream is mandatory.
• Decide sync vs async up front. Synchronous retries fight a 120-second cumulative cap; async retries (Queueable / Platform Event) get a fresh transaction per attempt.
• Identify what's retry-eligible vs not. 5xx, network timeouts, 408, 429, 503 retry. 400, 401, 403, 404, 422 do NOT retry; these are caller bugs or auth failures.

Core Concepts

Retry classification

The Apex synchronous constraint

A synchronous Apex transaction has a 120-second cumulative callout cap and no usable Thread.sleep (Apex offers no sleep primitive — busy-wait loops via System.now() polling are governor-killers and forbidden). This forces synchronous retries to be:

• Bounded — typical 3 attempts max.
• Short-backoff — 100ms / 500ms / 2000ms via the only legal "delay": staying inside the same callout's longer timeout, OR doing minimal CPU work between attempts.
• Aware of the 100-callout-per-transaction limit — every retry counts against it.

For real exponential backoff (seconds-to-minutes), go async.

Async retry — Queueable chain or Platform Event

The right pattern for backoff > a few seconds:

1. First attempt fires sync (or from a Queueable).
2. On retryable failure, the Queueable enqueues itself with attempt + 1 and a stored nextAttemptAt.
3. A Scheduled Apex job (or Platform Event subscriber) picks up the work when nextAttemptAt <= now.

Each retry runs in a fresh transaction with fresh governor limits. Backoff schedules of 1s / 5s / 30s / 5min are achievable.

Circuit breaker

Track per-endpoint failure rate in Cache.Org (org-partition Platform Cache). Three states:

• CLOSED — calls flow normally; failures increment a counter.
• OPEN — counter exceeds threshold (e.g. 10 failures in 60 seconds); subsequent calls short-circuit and throw immediately. No callout consumed.
• HALF-OPEN — after a cooldown (e.g. 30 seconds), one probe call is allowed. Success returns state to CLOSED; failure returns to OPEN with a longer cooldown.

The cache key MUST be per-endpoint (e.g. ckt:payment-api), not global. One bad endpoint shouldn't blackhole every integration.

Idempotency

For retryable writes, the downstream must dedup. Two patterns:

• Idempotency-Key HTTP header — generate a UUID, send it on every retry of the same logical operation. Downstream returns the original response on duplicate. Stripe, Square, and most modern payment APIs support this.
• Salesforce-side dedup table — Outbound_Callout_Log__c keyed by (endpoint + payload_hash + minute_bucket). Before retrying, query the log; if already succeeded, skip.

Dead-letter pattern

When retries exhaust (e.g. 5 attempts over 1 hour), write the failed payload to Failed_Callout__c with: endpoint, payload, last response, attempt count, last error. A scheduled job or admin UI can reprocess. Without this, retries that exhaust silently disappear.

Governor budget reminder

• 100 callouts per transaction (sync or async). Retries count.
• 120s cumulative callout time per transaction.
• Platform Cache: 10MB org partition free; 1KB per cache entry recommended.

Testing

MockHttpResponseGenerator can return a sequence of responses by storing call-count state in the mock class. Test patterns: 3 timeouts then success; 5 5xx triggering circuit-open; 4xx never retried.

Recommended Workflow

1. Classify the integration: idempotent vs non-idempotent; sync vs async tolerable; latency budget.
2. Pick the retry policy: max attempts, backoff schedule, jitter (10-20% randomization to avoid thundering herd).
3. Decide circuit-breaker thresholds per endpoint and store config in Custom Metadata (Callout_Resilience_Config__mdt).
4. Implement Idempotency-Key generation (UUID per logical request, persisted on the source record so retries reuse it).
5. Design the dead-letter sObject and the reprocessing path (scheduled Apex + admin reprocess UI).
6. Write MockHttpResponseGenerator returning a sequence — assert retry count, circuit-open behavior, and dead-letter writes.
7. Deploy with circuit-breaker thresholds tuned conservatively; monitor Failed_Callout__c volume in week 1.

Review Checklist

• 4xx responses (except 408/429) NEVER trigger retry
• Synchronous retry bounded to <= 3 attempts and total time < 120s
• Async retries use Queueable chain or Platform Event, not sync polling
• Circuit-breaker state stored per endpoint in Cache.Org, not as static var
• Idempotency-Key header sent for write operations, persisted on source record
• Dead-letter sObject populated on exhaustion; reprocessing path documented
• No System.now() busy-wait loops anywhere
• Limits.getCallouts() checked before issuing retry inside a transaction
• Test class with MockHttpResponseGenerator covers: success-on-retry, exhaustion-to-dead-letter, circuit-open, 4xx-not-retried

Salesforce-Specific Gotchas

1. 120-second cumulative callout cap is HARD. Three retries with 30s timeouts each plus backoff burns the budget fast. Measure actual P95 latency before sizing retries.
2. Platform Cache keys are case-sensitive. ckt:Payment-Api and ckt:payment-api are different breakers — pick a convention and lint it.
3. Limits.getCallouts() is per-transaction, not per-class. A trigger that fires three handlers each issuing 40 callouts will hit 100 even though no single class did.
4. Test.setMock returns a single mock instance. To return a sequence, store call-count state inside the mock class and switch on it.

Output Artifacts

Related Skills

• apex/apex-named-credentials-patterns — auth and endpoint config (NOT this skill)
• apex/callout-and-dml-transaction-boundaries — DML-then-callout ordering (NOT this skill)
• apex/apex-queueable-patterns — chaining queueables for async retry
• apex/apex-platform-cache-patterns — Cache.Org partitioning for circuit state
• templates/apex/HttpClient.cls — the underlying HTTP plumbing