API Rate Limit Handler

API Rate Limit Handler

Overview

A skill for implementing production-grade rate limiting, exponential backoff, and retry strategies when integrating with external APIs. Prevents cascading failures, respects upstream quotas, and keeps your application resilient under load.

When to Use This Skill

• Use when calling external APIs that enforce rate limits (OpenAI, Stripe, GitHub, etc.)
• Use when you receive 429 Too Many Requests or 5xx errors and need graceful recovery
• Use when building a client that must respect Retry-After headers
• Use when designing a system that fans out to multiple API providers
• Use when the user says "handle rate limits", "add retry logic", "backoff strategy", or "don't get throttled"

How It Works

Step 1: Classify the response

Determine whether a failed request is retryable or terminal.

Step 2: Parse rate limit headers

Always check upstream hints before computing your own delay.

function getRetryDelay(
  response: Response,
  attempt: number,
  maxDelayMs = 60_000
): number {
  // Prefer upstream hints
  const retryAfter = response.headers.get("Retry-After");
  if (retryAfter) {
    const seconds = Number(retryAfter);
    if (Number.isFinite(seconds) && seconds >= 0) {
      return Math.min(seconds * 1000, maxDelayMs);
    }
    // HTTP-date format
    const date = new Date(retryAfter).getTime();
    if (Number.isFinite(date)) {
      return Math.min(Math.max(0, date - Date.now()), maxDelayMs);
    }
  }

  // GitHub documents x-ratelimit-reset as Unix epoch seconds.
  const githubReset = Number(response.headers.get("x-ratelimit-reset"));
  if (Number.isFinite(githubReset)) {
    return Math.min(
      Math.max(0, githubReset * 1000 - Date.now()),
      maxDelayMs
    );
  }

  // Fallback: capped exponential backoff with full jitter.
  const cap = Math.min(1000 * 2 ** attempt, maxDelayMs);
  return Math.floor(Math.random() * cap);
}

Provider-specific reset headers do not share one unit or format. For example, some APIs return durations while GitHub returns epoch seconds. Parse an additional header only after checking that provider's current documentation.

Step 3: Implement the retry loop

async function fetchWithRetry(
  url: string,
  options: RequestInit,
  maxRetries = 3,
  maxElapsedMs = 120_000,
  retryNonIdempotent = false
): Promise<Response> {
  const startedAt = Date.now();
  const method = (options.method ?? "GET").toUpperCase();
  const replaySafe = ["GET", "HEAD", "OPTIONS", "PUT", "DELETE"].includes(method)
    || retryNonIdempotent;

  for (let attempt = 0; attempt <= maxRetries; attempt++) {
    const response = await fetch(url, options);

    if (response.ok) return response;

    // Terminal errors — do not retry
    if ([400, 401, 403, 404, 422].includes(response.status)) {
      throw new Error(`Terminal error ${response.status}: ${response.statusText}`);
    }

    if (!replaySafe) {
      throw new Error(
        `${method} was not retried because replay safety was not explicitly established`
      );
    }

    // Retryable — but exhausted attempts
    if (attempt === maxRetries) {
      throw new Error(`Failed after ${maxRetries} retries: ${response.status}`);
    }

    const remaining = maxElapsedMs - (Date.now() - startedAt);
    const delay = Math.min(getRetryDelay(response, attempt), remaining);
    if (delay <= 0) {
      throw new Error(`Retry deadline exceeded after ${maxElapsedMs}ms`);
    }

    // Release the connection before waiting when the body is not needed.
    await response.body?.cancel();
    console.warn(
      `Request failed (${response.status}), retrying in ${Math.round(delay)}ms (attempt ${attempt + 1}/${maxRetries})`
    );
    await new Promise(resolve => setTimeout(resolve, delay));
  }

  throw new Error("Unreachable");
}

Step 4: Add a client-side rate limiter (proactive)

Prevent hitting upstream limits in the first place with a token bucket or sliding window.

class TokenBucket {
  private tokens: number;
  private lastRefill: number;
  private queue: Promise<void> = Promise.resolve();

  constructor(
    private maxTokens: number,
    private refillRate: number // tokens per second
  ) {
    this.tokens = maxTokens;
    this.lastRefill = Date.now();
  }

  async acquire(): Promise<void> {
    const ticket = this.queue.then(() => this.acquireOnce());
    this.queue = ticket.catch(() => undefined);
    return ticket;
  }

  private async acquireOnce(): Promise<void> {
    this.refill();
    if (this.tokens < 1) {
      const waitMs = ((1 - this.tokens) / this.refillRate) * 1000;
      await new Promise(resolve => setTimeout(resolve, waitMs));
      this.refill();
    }
    this.tokens -= 1;
  }

  private refill(): void {
    const now = Date.now();
    const elapsed = (now - this.lastRefill) / 1000;
    this.tokens = Math.min(this.maxTokens, this.tokens + elapsed * this.refillRate);
    this.lastRefill = now;
  }
}

// Usage: limit to 60 requests/minute
const limiter = new TokenBucket(60, 1);

async function rateLimitedFetch(url: string, options: RequestInit) {
  await limiter.acquire();
  return fetchWithRetry(url, options);
}

Examples

Example 1: Idempotent API read with retry

const response = await fetchWithRetry(
  "https://api.github.com/repos/OWNER/REPO",
  {
    method: "GET",
    headers: {
      "Accept": "application/vnd.github+json",
      "Authorization": `Bearer ${githubToken}`,
    },
  },
  3
);

For a POST or another operation with side effects, leave retryNonIdempotent false unless the provider documents an idempotency mechanism and the same stable idempotency key is reused for every attempt.

Example 2: Python implementation

import time
import random
import httpx

def fetch_with_retry(url: str, max_retries: int = 3, **kwargs) -> httpx.Response:
    for attempt in range(max_retries + 1):
        response = httpx.request("GET", url, **kwargs)

        if response.is_success:
            return response

        if response.status_code in (400, 401, 403, 404, 422):
            response.raise_for_status()

        if attempt == max_retries:
            response.raise_for_status()

        # Parse Retry-After or compute backoff
        retry_after = response.headers.get("retry-after")
        if retry_after and retry_after.isdigit():
            delay = int(retry_after)
        else:
            delay = min(2 ** attempt + random.uniform(0, 1), 60)

        print(f"Retrying in {delay:.1f}s (attempt {attempt + 1}/{max_retries})")
        time.sleep(delay)

    raise RuntimeError("Unreachable")

Best Practices

• ✅ Always respect Retry-After headers — they come from the provider who knows their limits
• ✅ Add jitter to backoff to prevent thundering herd when multiple clients retry simultaneously
• ✅ Log every retry with status code, delay, and attempt number for debugging
• ✅ Set a maximum total timeout to avoid hanging indefinitely
• ✅ Use a client-side rate limiter proactively rather than only reacting to 429s
• ✅ Retry state-changing requests only with a provider-documented idempotency mechanism and a stable key
• ❌ Don't retry 4xx client errors (except 408 and 429) — fix the request instead
• ❌ Don't use fixed delays — exponential backoff distributes load more evenly
• ❌ Don't retry without a cap — unbounded retries can amplify outages
• ❌ Don't ignore per-endpoint limits — some APIs have different quotas per route

Limitations

• This skill does not replace environment-specific validation, testing, or expert review.
• Token bucket is approximate for distributed systems — use Redis-backed rate limiting for multi-instance deployments (for example the upstash-ratelimit skill, or any shared-store limiter).
• Some APIs use non-standard rate limit headers; check provider documentation.
• The elapsed-time cap shown here bounds retry waits, not a single hung network call; combine it with an AbortSignal or client timeout.

Common Pitfalls

• Problem: Retrying too aggressively during an outage amplifies the problem. Solution: Use exponential backoff with jitter and a circuit breaker for sustained failures.

• Problem: Multiple instances of your app all retry at the same time (thundering herd). Solution: Add randomized jitter (Math.random() * 0.3 * delay) to decorrelate retries.

• Problem: Retry-After header contains an HTTP-date instead of seconds. Solution: Parse both formats — check if the value is numeric first, then try Date parsing.

• Problem: Client-side limiter doesn't account for concurrent requests already in-flight. Solution: Serialize acquisition within one process, decrement before send, and use a shared distributed limiter across instances.

Related Skills

• @poka-yoke - Mistake-proofing APIs so invalid requests never reach the retry path
• @circuit-breaker - When to stop retrying entirely and fail fast