| name | apify-performance-tuning |
| description | Optimize Apify Actor performance: crawl speed, memory usage, concurrency, and proxy rotation.
Use when Actors are slow, consuming too much memory, or being blocked by target sites, or when a crawl is too expensive per run.
Trigger with "apify performance", "optimize apify actor", "apify slow", "crawlee concurrency", "apify memory tuning", "scraper performance".
|
| allowed-tools | Read, Write, Edit |
| version | 1.5.0 |
| license | MIT |
| author | Jeremy Longshore <jeremy@intentsolutions.io> |
| tags | ["saas","scraping","automation","apify"] |
| compatibility | Designed for Claude Code |
Apify Performance Tuning
Overview
Optimize Apify Actors for speed, cost, and reliability. Covers Crawlee concurrency settings, memory profiling, proxy rotation strategies, request batching, and crawler selection for different workloads.
The workflow is a repeatable loop: measure a baseline, apply one lever, re-measure. The single highest-impact lever is usually crawler choice — swapping a browser crawler for CheerioCrawler on non-JS pages is a 5-10x speedup on its own. The full six-step walkthrough, with every code block, lives in references/implementation.md.
Prerequisites
- Existing Actor with measurable baseline performance
- Understanding of
apify-sdk-patterns
- Access to Actor run stats in Apify Console
APIFY_TOKEN in the environment for reading run stats via ApifyClient
Instructions
Work the levers in order. Each step is expanded — with copy-paste code — in the reference file linked below.
- Measure a baseline. Pull
runTimeSecs, requestsFinished, memAvgBytes, and usageTotalUsd from the run stats before changing anything. You cannot judge an optimization without a before number.
- Choose the right crawler.
HttpCrawler/CheerioCrawler for static HTML or JSON (low memory, fast); PlaywrightCrawler/PuppeteerCrawler only when the page genuinely needs JavaScript rendering.
- Tune concurrency. Raise
maxConcurrency for Cheerio (up to ~50); keep it low (~3-5) for browser crawlers because each page costs ~200MB. Let autoscaledPoolOptions adjust within the band.
- Optimize memory. Push data immediately instead of accumulating arrays; for browser crawlers, block images/CSS/fonts in
preNavigationHooks and cap concurrent browsers.
- Right-size the memory allocation. Compute units bill on
memory x duration — start low (512 MB for Cheerio) and only raise it if the Actor is memory-starved.
- Rotate proxies and tune requests. Start on datacenter proxies, fall back to residential on 403/blocked; use a session pool for IP rotation and ban detection.
