Apify Rate Limits

Apify Rate Limits

Overview

The Apify API enforces rate limits per resource. The apify-client library auto-retries 429s (up to 8 times with exponential backoff), so most workloads never notice a limit. You reach for this skill when bulk operations, custom API calls, or large fan-outs push past what the built-in retry can absorb — you then batch, queue, stagger, and monitor to stay under the ceiling.

Full runnable code for every step is in implementation.md; combined scenarios are in examples.md.

Apify rate limit rules

"Per resource" means: calls to dataset A and dataset B each get 60 req/sec independently. Every response carries X-RateLimit-Limit, X-RateLimit-Remaining, and X-RateLimit-Reset (epoch seconds) headers.

Prerequisites

• An Apify account with API access and APIFY_TOKEN set in the environment.
• The apify-client package installed (npm install apify-client).
• For custom queuing: p-queue (npm install p-queue); crawlee for sleep and crawler-level concurrency.

Instructions

The workflow is five steps. Each is summarized here with its core lever; the full runnable code for every step is in implementation.md.

1. Understand built-in retries — apify-client already retries 429/500+ with exponential backoff. Tune maxRetries / minDelayBetweenRetriesMillis only when the defaults are wrong for your endpoint:

   import { ApifyClient } from 'apify-client';
   const client = new ApifyClient({
     token: process.env.APIFY_TOKEN,
     maxRetries: 5,                      // Default: 8
     minDelayBetweenRetriesMillis: 500,  // Default: 500
   });
   

2. Batch operations (biggest lever) — collapse per-item loops into one batched call (up to 9 MB), chunking only for very large datasets:

   await client.dataset(dsId).pushItems(items);   // 1 call, not N
   

3. Queue custom calls — gate raw API calls through p-queue (concurrency + intervalCap) so fan-out reads never exceed 60 req/sec. See implementation.md § Step 3.

4. Stagger Actor starts — insert a ~200 ms delay between start() calls so the runs endpoint never 429s, then waitForFinish() in parallel. See implementation.md § Step 4.

5. Monitor headers — feed X-RateLimit-* into a small monitor that warns before the wall and pauses exactly until reset. See implementation.md § Step 5.

Target-website throttling is a separate ceiling from the platform API — cap it with Crawlee's maxConcurrency / maxRequestsPerMinute (implementation.md § Crawlee-level concurrency).

Output

Applying this skill produces a rate-aware Apify integration:

• A configured ApifyClient with an explicit retry envelope.
• Batched/chunked dataset writes that cut API-call count by orders of magnitude.
• A p-queue-gated call path that holds requests under 60 req/sec per resource.
• Staggered Actor starts and, optionally, a header-driven monitor that pauses before exhaustion — the net effect being zero (or transparently retried) 429s under load.

Error Handling

Examples

Worked end-to-end scenarios live in examples.md:

• Bulk dataset push without 429s — 50,000 rows in ~50 calls via chunked batching.
• Fan-out reads through a queue — 500 Actor reads held under 50 req/sec.
• Launch 100 runs safely — staggered starts, then parallel wait-for-finish.
• Pause on header-driven exhaustion — sleep exactly until the limit resets.

Resources

• Apify API Rate Limits
• p-queue Documentation
• Crawlee Auto-scaling

For security configuration, see apify-security-basics.