Apify Rate Limits
Overview
The Apify API enforces rate limits per resource. The apify-client library
auto-retries 429s (up to 8 times with exponential backoff), so most workloads never
notice a limit. You reach for this skill when bulk operations, custom API calls, or
large fan-outs push past what the built-in retry can absorb — you then batch, queue,
stagger, and monitor to stay under the ceiling.
Full runnable code for every step is in
implementation.md; combined scenarios are in
examples.md.
Apify rate limit rules
| Scope | Limit | Notes |
|---|
| Per resource (default) | 60 req/sec | Applies to each Actor, dataset, KV store independently |
| Dataset push | 60 req/sec per dataset | Batch items to reduce call count |
| Actor runs | 60 req/sec per Actor | Start runs in sequence or with delays |
| Platform-wide | Higher limit | Aggregate across all resources |
"Per resource" means: calls to dataset A and dataset B each get 60 req/sec
independently. Every response carries X-RateLimit-Limit,
X-RateLimit-Remaining, and X-RateLimit-Reset (epoch seconds) headers.
Prerequisites
- An Apify account with API access and
APIFY_TOKEN set in the environment.
- The
apify-client package installed (npm install apify-client).
- For custom queuing:
p-queue (npm install p-queue); crawlee for sleep and
crawler-level concurrency.
Instructions
The workflow is five steps. Each is summarized here with its core lever; the full
runnable code for every step is in
implementation.md.
-
Understand built-in retries — apify-client already retries 429/500+ with
exponential backoff. Tune maxRetries / minDelayBetweenRetriesMillis only when
the defaults are wrong for your endpoint:
import { ApifyClient } from 'apify-client';
const client = new ApifyClient({
token: process.env.APIFY_TOKEN,
maxRetries: 5,
minDelayBetweenRetriesMillis: 500,
});
-
Batch operations (biggest lever) — collapse per-item loops into one batched
call (up to 9 MB), chunking only for very large datasets:
await client.dataset(dsId).pushItems(items);
-
Queue custom calls — gate raw API calls through p-queue
(concurrency + intervalCap) so fan-out reads never exceed 60 req/sec. See
implementation.md § Step 3.
-
Stagger Actor starts — insert a ~200 ms delay between start() calls so the
runs endpoint never 429s, then waitForFinish() in parallel. See
implementation.md § Step 4.
-
Monitor headers — feed X-RateLimit-* into a small monitor that warns before
the wall and pauses exactly until reset. See
implementation.md § Step 5.
Target-website throttling is a separate ceiling from the platform API — cap it with
Crawlee's maxConcurrency / maxRequestsPerMinute
(implementation.md § Crawlee-level concurrency).
Output
Applying this skill produces a rate-aware Apify integration:
- A configured
ApifyClient with an explicit retry envelope.
- Batched/chunked dataset writes that cut API-call count by orders of magnitude.
- A
p-queue-gated call path that holds requests under 60 req/sec per resource.
- Staggered Actor starts and, optionally, a header-driven monitor that pauses before
exhaustion — the net effect being zero (or transparently retried) 429s under load.
Error Handling
| Scenario | Detection | Response |
|---|
| API 429 | apify-client auto-retries | Usually transparent; increase delays if persistent |
| Target site 429 | statusCode === 429 in handler | Reduce maxConcurrency, add proxy rotation |
| Burst of starts | Starting 100+ runs at once | Stagger with 200ms delays |
| Large data push | Single 50MB dataset push | Chunk into 9MB batches |
Examples
Worked end-to-end scenarios live in examples.md:
- Bulk dataset push without 429s — 50,000 rows in ~50 calls via chunked batching.
- Fan-out reads through a queue — 500 Actor reads held under 50 req/sec.
- Launch 100 runs safely — staggered starts, then parallel wait-for-finish.
- Pause on header-driven exhaustion — sleep exactly until the limit resets.
Resources
For security configuration, see apify-security-basics.
