Skip to main content
firecrawl-load-scale Load test and scale Firecrawl scraping pipelines with concurrency control and batching.
Use when testing scraping throughput, planning capacity for large crawl jobs,
or optimizing concurrent scrape performance.
Trigger with phrases like "firecrawl load test", "firecrawl scale",
"firecrawl throughput", "firecrawl capacity", "firecrawl concurrent".
インストールへ移動 Skills Marketplace コミュニティが作成したAIスキルを発見・探索
Codex または Claude でインストール この Prompt をコピーして Codex、Claude、または他のアシスタントに貼り付けると、Skill ページを確認してインストールできます。
直接コマンドでは確認用 Prompt が省略されます。実行前にソースを確認してください。
npx skills add https://github.com/jeremylongshore/claude-code-plugins-plus-skills --skill firecrawl-load-scaleコマンドは1行のまま表示されます。コピー前に横へスクロールして全体を確認してください。
ローカルで確認しますか？SkillsMP が現在取得できるファイルをダウンロードできます。
Zipをダウンロード ダウンロード中... このリポジトリの他の Skills Implement user sign-up and sign-in flows with Clerk.
Use when building authentication UI, customizing sign-in experience,
or implementing OAuth social login.
Trigger with phrases like "clerk sign-in", "clerk sign-up",
"clerk login flow", "clerk OAuth", "clerk social login".
Implement session management and middleware with Clerk.
Use when managing user sessions, configuring route protection,
or implementing token refresh and custom JWT templates.
Trigger with phrases like "clerk session", "clerk middleware",
"clerk route protection", "clerk token", "clerk JWT".
Configure enterprise SSO, role-based access control, and organization management.
Use when implementing SSO integration, configuring role-based permissions,
or setting up organization-level controls.
Trigger with phrases like "clerk SSO", "clerk RBAC",
"clerk enterprise", "clerk roles", "clerk permissions", "clerk organizations".
jeremylongshore
jeremylongshore/claude-code-plugins-plus-skills
GitHub リポジトリを開く name firecrawl-load-scale description Load test and scale Firecrawl scraping pipelines with concurrency control and batching.
Use when testing scraping throughput, planning capacity for large crawl jobs,
or optimizing concurrent scrape performance.
Trigger with phrases like "firecrawl load test", "firecrawl scale",
"firecrawl throughput", "firecrawl capacity", "firecrawl concurrent".
allowed-tools Read, Write, Edit, Bash(node:*), Bash(npm:*) version 1.11.0 license MIT author Jeremy Longshore <jeremy@intentsolutions.io> tags ["saas","firecrawl","testing","performance","scaling"] compatibility Designed for Claude Code, also compatible with Codex and OpenClaw
Firecrawl Load & Scale
Overview
Load test and scale Firecrawl scraping pipelines. Firecrawl's rate limits are per-plan (RPM and concurrent connections), so scaling means maximizing throughput within those limits using batch scraping, async crawls, and queue-based request management.
Rate Limits by Plan
Plan Scrape RPM Concurrent Crawls Max Batch Size Free 10 2 10 Hobby 20 3 50 Standard 50 5 100 Growth 100 10 100 Scale 500+ 50+ 100
Instructions
Step 1: Measure Baseline Throughput
import FirecrawlApp from "@mendable/firecrawl-js" ;
const firecrawl = new FirecrawlApp ({
apiKey : process.env .FIRECRAWL_API_KEY !,
});
async function measureThroughput (urls : string [], concurrency : number const start = Date .now ();
const results : Array <{ url : string ; durationMs : number ; success : boolean ; chars : number }> = [];
for
let
0
length
const
slice
const
await
Promise
all
map
async
const
Date
now
try
const
await
scrapeUrl
formats
"markdown"
return
durationMs
Date
now
success
true
chars
markdown
length
0
catch
return
durationMs
Date
now
success
false
chars
0
push
const
Date
now
const
filter
r =>
success
length
console
log
`=== Throughput Report ===`
console
log
`URLs: ${urls.length} , Concurrency: ${concurrency} `
console
log
`Total time: ${totalMs} ms`
console
log
`Success: ${succeeded} /${urls.length} `
console
log
`Throughput: ${(urls.length / (totalMs / 1000 )).toFixed(1 )} pages/sec`
console
log
`Avg latency: ${(results.reduce((s, r) => s + r.durationMs, 0 ) / results.length).toFixed(0 )} ms`
return
Step 2: Use Batch Scrape for Maximum Efficiency
async function scaledBatchScrape (urls : string [], batchSize = 50 const allResults : any [] = [];
for (let i = 0 ; i < urls.length ; i += batchSize) {
const batch = urls.slice (i, i + batchSize);
console .log (`Batch ${i / batchSize + 1 } : scraping ${batch.length} URLs...` );
const result = await firecrawl.batchScrapeUrls (batch, {
formats : ["markdown" ],
onlyMainContent : true ,
});
allResults.push (...(result.data || []));
console .log (` Done: ${result.data?.length} pages scraped` );
}
return allResults;
}
Step 3: Queue-Based Scraping with p-queue import PQueue from "p-queue" ;
function createScrapeQueue (config : {
concurrency: number ;
requestsPerSecond: number ;
}const queue = new PQueue ({
concurrency : config.concurrency ,
interval : 1000 ,
intervalCap : config.requestsPerSecond ,
});
async function scrape (url : string return queue.add (async () => {
const result = await firecrawl.scrapeUrl (url, {
formats : ["markdown" ],
onlyMainContent : true ,
});
return { url, markdown : result.markdown , title : result.metadata ?.title };
});
}
return { scrape, queue };
}
const { scrape, queue } = createScrapeQueue ({
concurrency : 5 ,
requestsPerSecond : 10 ,
});
const urls = ["https://a.com" , "https://b.com" , ];
const results = await Promise .all (urls.map (scrape));
console .log (`Queue: ${queue.pending} pending, ${queue.size} queued` );
Step 4: Scale Async Crawls
async function parallelCrawls (targets : Array <{ url: string ; limit: number }>const jobs = await Promise .all (
targets.map (async t => {
const job = await firecrawl.asyncCrawlUrl (t.url , {
limit : t.limit ,
scrapeOptions : { formats : ["markdown" ] },
});
return { ...t, jobId : job.id };
})
);
console .log (`Started ${jobs.length} crawl jobs` );
const results : any [] = [];
const pending = new Set (jobs.map (j =>jobId ));
while (pending.size > 0 ) {
for (const jobId of [...pending]) {
const status = await firecrawl.checkCrawlStatus (jobId);
if (status.status === "completed" ) {
results.push ({ jobId, pages : status.data ?.length });
pending.delete (jobId);
console .log (`Job ${jobId} complete: ${status.data?.length} pages (${pending.size} remaining)` );
} else if (status.status === "failed" ) {
pending.delete (jobId);
console .error (`Job ${jobId} failed: ${status.error} ` );
}
}
if (pending.size > 0 ) {
await new Promise (r =>setTimeout (r, 5000 ));
}
}
return results;
}
Step 5: Capacity Planning function estimateCapacity (plan : {
rpm: number ;
concurrentCrawls: number ;
credits: number ;
}const pagesPerMinute = plan.rpm ;
const pagesPerHour = pagesPerMinute * 60 ;
const pagesPerDay = pagesPerHour * 24 ;
const daysOfCredits = plan.credits / (pagesPerDay * 0.5 );
console .log (`=== Capacity Estimate ===` );
console .log (`Max throughput: ${pagesPerMinute} pages/min` );
console .log (`Daily capacity: ${pagesPerDay.toLocaleString()} pages/day` );
console .log (`Credit runway: ${daysOfCredits.toFixed(0 )} days at 50% utilization` );
console .log (`Concurrent crawl jobs: ${plan.concurrentCrawls} ` );
}
estimateCapacity ({ rpm : 50 , concurrentCrawls : 5 , credits : 50000 });
Error Handling Issue Cause Solution 429 errors under load Exceeding RPM limit Reduce concurrency, use p-queue Batch scrape timeout Too many URLs Split into chunks of 50 Crawl jobs queued Hit concurrent crawl limit Stagger start times Diminishing returns Network bottleneck Increase plan tier, not concurrency
Examples
Quick Load Test const testUrls = Array .from ({ length : 20 }, (_, i ) =>
`https://docs.firecrawl.dev/features/${["scrape" , "crawl" , "map" , "extract" ][i % 4 ]} `
);
await measureThroughput (testUrls, 5 );
Resources
Next Steps For reliability patterns, see firecrawl-reliability-patterns.
