Rate Limit Handler
Stay under an upstream's quota by backing off cooperatively, obeying what the server tells you, and de-synchronizing your fleet - retrying harder is how a transient limit becomes an outage.
Workflow
- Classify the response before reacting. Retry only on 429, 502, 503, 504, and connection/timeout errors. Never retry 400, 401, 403, 404, or 422 - they will fail identically. Retry only idempotent or idempotency-keyed operations; a bare non-idempotent POST can duplicate effects.
- Honor the server first. If the response carries
Retry-After, obey it exactly and skip local backoff math - parse both the delta-seconds and the HTTP-date form. Read X-RateLimit-Remaining and X-RateLimit-Reset proactively and slow down as remaining approaches zero rather than waiting to be rejected - throttling when remaining drops below ~10% of the window's quota is a reasonable trigger.
- Back off with full jitter when there is no
Retry-After. Compute delay = base * 2^attempt with a base of 100-500ms, clamp it to a max (e.g. 30-60s), then sleep a uniform random value between 0 and that clamped cap - not the cap itself. Full jitter is what de-synchronizes many clients retrying the same downed service.
- Bound the retries. Cap both the attempt count (e.g. 5) and the total elapsed budget (e.g. 30-60s, and always below the caller's own timeout) so a request fails fast instead of hanging a caller for minutes. Surface the final failure to the caller; do not swallow it.
- Throttle proactively with a token bucket. Put a client-side limiter in front of the API sized to roughly 80% of the published quota - the headroom absorbs retries, clock skew, and other callers you forgot about - so bursts are smoothed and you stay under the limit by design rather than reacting to 429s. When many workers hit one upstream, back the bucket with a shared store (e.g. Redis) so the aggregate rate - not the per-process rate - stays bounded.
- Make it observable. Emit metrics for retry count, backoff duration, and 429 rate. A rising 429 rate is the early signal to lower concurrency or request a quota increase before users see errors; sustained 429s above ~1% of requests means the token bucket is mis-sized or missing.
Deliverable
Produce the retry/throttle layer (a wrapper, interceptor, or middleware around the upstream client) plus a short configuration summary stating: the retryable status set, base delay, cap, and jitter mode, the attempt and total-time budgets, the token-bucket rate and burst size relative to the published quota, whether the bucket is per-process or shared across the fleet, and the metrics emitted. The summary is what the next engineer tunes when the upstream changes its limits.
Quality bar
- No fixed-delay or no-jitter retry loop survives - every backoff path applies jitter.
Retry-After always wins over computed backoff.- Every retry path is bounded by both attempt count and total time budget.
- Non-idempotent operations are never retried without an idempotency key.
- The 4xx set above is never retried.
Do NOT
- Do NOT retry on 4xx client errors (except 429) - the request is malformed or unauthorized and will fail again.
- Do NOT use fixed or cap-only backoff - synchronized clients recreate the thundering herd you are trying to avoid.
- Do NOT retry forever or without a total-time budget.
- Do NOT enforce per-process limits when a fleet shares one quota - the aggregate will still blow the limit.
- Do NOT add the full backoff-plus-bucket stack to a single best-effort background call; let it fail and move on, and reserve the machinery for hot paths and high-volume integrations.
1---2name: rate-limit-handler3description: Adds retry-with-backoff, Retry-After handling, and client-side throttling so a caller stays under an upstream API's rate limits instead of hammering it. Use when you call a rate-limited or quota-enforced third-party API, see 429 or 503 responses or Retry-After headers, or have a worker fleet that needs to share one upstream's quota. Do NOT use to fail fast when a dependency is down - use circuit-breaker-builder instead; do NOT use to scaffold the HTTP client itself - use api-client-generator instead.4---5# Rate Limit Handler67Stay under an upstream's quota by backing off cooperatively, obeying what the server tells you, and de-synchronizing your fleet - retrying harder is how a transient limit becomes an outage.89## Workflow10111. **Classify the response before reacting.** Retry only on 429, 502, 503, 504, and connection/timeout errors. Never retry 400, 401, 403, 404, or 422 - they will fail identically. Retry only idempotent or idempotency-keyed operations; a bare non-idempotent POST can duplicate effects.122. **Honor the server first.** If the response carries `Retry-After`, obey it exactly and skip local backoff math - parse both the delta-seconds and the HTTP-date form. Read `X-RateLimit-Remaining` and `X-RateLimit-Reset` proactively and slow down as remaining approaches zero rather than waiting to be rejected - throttling when remaining drops below ~10% of the window's quota is a reasonable trigger.133. **Back off with full jitter when there is no `Retry-After`.** Compute `delay = base * 2^attempt` with a base of 100-500ms, clamp it to a max (e.g. 30-60s), then sleep a uniform random value between 0 and that clamped cap - not the cap itself. Full jitter is what de-synchronizes many clients retrying the same downed service.144. **Bound the retries.** Cap both the attempt count (e.g. 5) and the total elapsed budget (e.g. 30-60s, and always below the caller's own timeout) so a request fails fast instead of hanging a caller for minutes. Surface the final failure to the caller; do not swallow it.155. **Throttle proactively with a token bucket.** Put a client-side limiter in front of the API sized to roughly 80% of the published quota - the headroom absorbs retries, clock skew, and other callers you forgot about - so bursts are smoothed and you stay under the limit by design rather than reacting to 429s. When many workers hit one upstream, back the bucket with a shared store (e.g. Redis) so the aggregate rate - not the per-process rate - stays bounded.166. **Make it observable.** Emit metrics for retry count, backoff duration, and 429 rate. A rising 429 rate is the early signal to lower concurrency or request a quota increase before users see errors; sustained 429s above ~1% of requests means the token bucket is mis-sized or missing.1718## Deliverable1920Produce the retry/throttle layer (a wrapper, interceptor, or middleware around the upstream client) plus a short configuration summary stating: the retryable status set, base delay, cap, and jitter mode, the attempt and total-time budgets, the token-bucket rate and burst size relative to the published quota, whether the bucket is per-process or shared across the fleet, and the metrics emitted. The summary is what the next engineer tunes when the upstream changes its limits.2122## Quality bar2324- No fixed-delay or no-jitter retry loop survives - every backoff path applies jitter.25- `Retry-After` always wins over computed backoff.26- Every retry path is bounded by both attempt count and total time budget.27- Non-idempotent operations are never retried without an idempotency key.28- The 4xx set above is never retried.2930## Do NOT3132- Do NOT retry on 4xx client errors (except 429) - the request is malformed or unauthorized and will fail again.33- Do NOT use fixed or cap-only backoff - synchronized clients recreate the thundering herd you are trying to avoid.34- Do NOT retry forever or without a total-time budget.35- Do NOT enforce per-process limits when a fleet shares one quota - the aggregate will still blow the limit.36- Do NOT add the full backoff-plus-bucket stack to a single best-effort background call; let it fail and move on, and reserve the machinery for hot paths and high-volume integrations.
SkillMedev