Optimize Apify Actor performance: crawl speed, memory usage, concurrency, and proxy rotation.
Use when Actors are slow, consuming too much memory, or being blocked by target sites.
Trigger: "apify performance", "optimize apify actor", "apify slow",
"crawlee concurrency", "apify memory tuning", "scraper performance".
Optimize Apify Actor performance: crawl speed, memory usage, concurrency, and proxy rotation.
Use when Actors are slow, consuming too much memory, or being blocked by target sites.
Trigger: "apify performance", "optimize apify actor", "apify slow",
"crawlee concurrency", "apify memory tuning", "scraper performance".
allowed-tools
Read, Write, Edit
version
1.0.0
license
MIT
author
Jeremy Longshore <jeremy@intentsolutions.io>
tags
["saas","scraping","automation","apify"]
compatible-with
claude-code
Apify Performance Tuning
Overview
Optimize Apify Actors for speed, cost, and reliability. Covers Crawlee concurrency settings, memory profiling, proxy rotation strategies, request batching, and crawler selection for different workloads.
Prerequisites
Existing Actor with measurable baseline performance
Understanding of apify-sdk-patterns
Access to Actor run stats in Apify Console
Performance Baseline
Measure before optimizing. Key metrics from run stats:
// Switch from Playwright to Cheerio for 5-10x speed improvement// (if pages don't require JavaScript rendering)import { CheerioCrawler } from'crawlee';
const crawler = newCheerioCrawler({
// Cheerio parses HTML without launching a browserrequestHandler: async ({ $, request }) => {
const title = $('title').text();
awaitActor.pushData({ url: request.url, title });
},
});
Step 2: Tune Concurrency
const crawler = newCheerioCrawler({
// --- Concurrency controls ---minConcurrency: 1, // Start with 1 parallel requestmaxConcurrency: 50, // Scale up to 50 (CheerioCrawler can handle more)// For PlaywrightCrawler, use lower values (each page = ~200MB)// maxConcurrency: 5,// Auto-scaling pool adjusts between min and max based on system loadautoscaledPoolOptions: {
desiredConcurrency: 10,
scaleUpStepRatio: 0.05, // Increase concurrency 5% at a timescaleDownStepRatio: 0.05,
maybeRunIntervalSecs: 5,
},
// Rate limiting (protect target site)maxRequestsPerMinute: 300, // Hard cap
});
Step 3: Optimize Memory
// CheerioCrawler memory optimizationconst crawler = newCheerioCrawler({
// Don't keep full HTML in memoryrequestHandlerTimeoutSecs: 30,
// Process and discard — don't accumulaterequestHandler: async ({ $, request }) => {
// Extract only what you needconst data = {
url: request.url,
title: $('title').text().trim(),
price: parseFloat($('.price').text().replace(/[^0-9.]/g, '')),
};
// Push immediately (don't collect in array)awaitActor.pushData(data);
},
});
// PlaywrightCrawler memory optimizationconst playwrightCrawler = newPlaywrightCrawler({
maxConcurrency: 3, // Key: fewer concurrent browserslaunchContext: {
launchOptions: {
headless: true,
args: [
'--disable-gpu',
'--disable-dev-shm-usage',
'--no-sandbox',
'--disable-extensions',
],
},
},
preNavigationHooks: [
async ({ page }) => {
// Block heavy resources to save memory and bandwidthawait page.route('**/*.{png,jpg,jpeg,gif,svg,webp,ico}', route => route.abort());
await page.route('**/*.{css,woff,woff2,ttf}', route => route.abort());
await page.route('**/analytics*', route => route.abort());
await page.route('**/tracking*', route => route.abort());
},
],
postNavigationHooks: [
async ({ page }) => {
// Close unnecessary page resourcesawait page.evaluate(() => {
window.stop(); // Stop loading remaining resources
});
},
],
});
Step 4: Memory Allocation Strategy
Actor memory affects both performance and cost:
CU = (Memory in GB) x (Duration in hours)
CU cost = $0.25 - $0.30 per CU (plan-dependent)
Actor Type
Recommended Memory
Reasoning
CheerioCrawler (simple)
256-512 MB
HTML parsing is lightweight
CheerioCrawler (complex)
512-1024 MB
Large pages, many concurrent
PlaywrightCrawler
2048-4096 MB
Each browser page ~200MB
Data processing
1024-2048 MB
In-memory transforms
// Start low, let the platform auto-scale if neededconst run = await client.actor('user/actor').call(input, {
memory: 512, // Start here for Cheeriotimeout: 3600, // 1 hour max
});
Step 5: Proxy Rotation for Speed and Reliability
import { Actor } from'apify';
// Datacenter proxy (fast, cheap, may be blocked)const dcProxy = awaitActor.createProxyConfiguration({
groups: ['BUYPROXIES94952'],
});
// Residential proxy (slower, expensive, higher success rate)const resProxy = awaitActor.createProxyConfiguration({
groups: ['RESIDENTIAL'],
countryCode: 'US',
});
// Smart rotation: try datacenter first, fall back to residentialconst crawler = newCheerioCrawler({
proxyConfiguration: dcProxy, // Start with fast proxyasyncfailedRequestHandler({ request }, error) {
if (error.message.includes('403') || error.message.includes('blocked')) {
// Re-enqueue with residential proxy
request.userData.useResidential = true;
await crawler.requestQueue.addRequest(request, { forefront: true });
}
},
asyncrequestHandler({ request, session, ...ctx }) {
if (request.userData.useResidential) {
// Switch proxy for this request
session?.retire(); // Force new IP
}
// ... extraction logic
},
});
Step 6: Request-Level Optimizations
const crawler = newCheerioCrawler({
// Retry configurationmaxRequestRetries: 3, // Default: 3requestHandlerTimeoutSecs: 30, // Kill slow pages// Navigation settings (CheerioCrawler-specific)additionalMimeTypes: ['application/json'], // Accept JSON responsessuggestResponseEncoding: 'utf-8',
// Session pool (IP rotation and ban detection)useSessionPool: true,
sessionPoolOptions: {
maxPoolSize: 100, // Sessions in poolsessionOptions: {
maxUsageCount: 50, // Requests per sessionmaxErrorScore: 3, // Errors before retiring session
},
},
// Pre-navigation hooks for request modificationpreNavigationHooks: [
async ({ request }) => {
// Add headers that help avoid blocks
request.headers = {
...request.headers,
'Accept-Language': 'en-US,en;q=0.9',
'Accept': 'text/html,application/xhtml+xml',
};
},
],
});
