Groq Rate Limits

Groq Rate Limits

Overview

Handle Groq rate limits using the retry-after header, exponential backoff, and request queuing. Groq enforces limits at the organization level with both RPM (requests/minute) and TPM (tokens/minute) constraints -- hitting either one triggers a 429.

The workflow builds up in five composable layers: parse the rate-limit headers, wrap calls in retry-with-backoff, gate concurrency through a queue, monitor remaining capacity proactively, and fall back across models when one pool is exhausted. Read SKILL.md for the high-level flow, then drill into the full implementation for every code block and the reference tables + worked examples for header definitions and composed clients.

Prerequisites

• A Groq API key (GROQ_API_KEY) — get one at console.groq.com.
• groq-sdk installed: npm install groq-sdk.
• For queuing (Step 3): p-queue installed: npm install p-queue.
• Node.js 18+ (for native fetch and the SDK).
• Know your plan's limits — check console.groq.com/settings/limits.

Rate Limits at a Glance

Groq applies RPM, RPD, TPM, and TPD limits simultaneously — you must stay under every one, and either RPM or TPM can trip a 429. Every response (even a success) carries x-ratelimit-* headers describing remaining capacity and reset timing; 429 responses add a retry-after header. Full header and constraint tables: reference.md.

Instructions

Compose these five steps into one client wrapper (queue → monitor → retry). Each step's complete, copy-pasteable code is in implementation.md.

Step 1: Parse Rate Limit Headers

Read the x-ratelimit-* headers off every response into a typed RateLimitInfo so downstream logic can reason about remaining capacity. Groq reports reset times as strings like "1.2s" or "120ms" — normalize them to milliseconds.

Step 2: Exponential Backoff with Retry-After

Wrap each API call in a retry loop. Prefer Groq's retry-after header when present; otherwise back off exponentially with jitter, capped at maxDelayMs. Retry only 429 and 5xx — other 4xx errors are not retryable.

Step 3: Request Queue with Concurrency Control

Gate all requests through a p-queue sized to your plan's RPM (intervalCap over a 60s interval) so bursts never exceed the limit in the first place.

Step 4: Proactive Rate Limit Monitor

Track remaining requests/tokens from response headers and pause before hitting zero (shouldThrottle() → waitIfNeeded()), instead of reacting to 429s after the fact.

Step 5: Model-Aware Rate Limit Strategy

Different models draw from different limit pools. When the preferred model is throttled, fall back to another model to keep making progress without waiting for a reset.

Output

Applying this skill produces:

• A withRateLimitRetry() wrapper that transparently retries 429/5xx with retry-after-aware backoff.
• A RateLimitMonitor that surfaces live status (getStatus() → "Requests: N remaining | Tokens: M remaining") and throttles proactively.
• A p-queue-backed client that caps throughput to your RPM so 100 fan-out calls complete without tripping the limit.
• Console diagnostics on every backoff/throttle event (e.g. Rate limited (attempt 2/5). Waiting 1.4s...).

The observable end state: sustained request volume that stays under RPM/TPM with zero unhandled 429s.

Error Handling

Examples

Start from this minimal 429 handler, then graduate to the composed client (queue + monitor + retry) in reference.md:

try {
  await groq.chat.completions.create({ model, messages });
} catch (err) {
  if (err instanceof Groq.APIError && err.status === 429) {
    const retryAfter = parseInt(err.headers?.["retry-after"] || "0");
    console.log(`Rate limited. retry-after says wait ${retryAfter}s.`);
    // -> feed retryAfter into withRateLimitRetry (Step 2)
  }
}

• Full five-step implementation (every code block, verbatim): implementation.md
• Composed client + header/limit tables + a 100-request fan-out: reference.md

Resources

• Groq Rate Limits Documentation
• Groq Pricing / Plans
• p-queue on npm
• Full implementation: references/implementation.md
• Reference tables & worked examples: references/reference.md

Next Steps

For security configuration, see the groq-security-basics skill in this pack, which covers API key storage, rotation, and request signing to complement the throughput handling above.