Firecrawl Performance Tuning
Overview
Optimize Firecrawl API performance by choosing efficient scraping modes, caching results, using batch endpoints, and minimizing unnecessary rendering. Key levers: format selection (markdown vs HTML vs screenshot), waitFor tuning, onlyMainContent, and batch vs individual scraping.
Latency Benchmarks
| Operation | Typical | With JS Wait | With Screenshot |
|---|
| scrapeUrl (markdown) | 2-5s | 5-10s | 8-15s |
| scrapeUrl (extract) | 3-8s | 8-15s | N/A |
| crawlUrl (10 pages) | 20-40s | 40-80s | N/A |
| mapUrl | 1-3s | N/A | N/A |
| batchScrapeUrls (10) | 10-20s | 20-40s | N/A |
Instructions
Step 1: Minimize Formats (Biggest Win)
import FirecrawlApp from "@mendable/firecrawl-js";
const firecrawl = new FirecrawlApp({
apiKey: process.env.FIRECRAWL_API_KEY!,
});
const slow = await firecrawl.scrapeUrl(url, {
formats: ["markdown", "html", "links", "screenshot"],
});
const fast = await firecrawl.scrapeUrl(url, {
formats: ["markdown"],
onlyMainContent: true,
});
Step 2: Tune waitFor for JS-Heavy Pages
const staticResult = await firecrawl.scrapeUrl("https://docs.example.com", {
formats: ["markdown"],
});
const spaResult = await firecrawl.scrapeUrl("https://app.example.com", {
formats: ["markdown"],
waitFor: 3000,
onlyMainContent: true,
});
const heavyResult = await firecrawl.scrapeUrl("https://dashboard.example.com", {
formats: ["markdown"],
actions: [
{ type: "wait", selector: ".data-table" },
{ type: "scroll", direction: "down" },
],
});
Step 3: Cache Scraped Content
import { LRUCache } from "lru-cache";
import { createHash } from "crypto";
const scrapeCache = new LRUCache<string, any>({
max: 500,
ttl: 3600000,
});
async function cachedScrape(url: string) {
const key = createHash("md5").update(url).digest("hex");
const cached = scrapeCache.get(key);
if (cached) {
console.log(`Cache hit: ${url}`);
return cached;
}
const result = await firecrawl.scrapeUrl(url, {
formats: ["markdown"],
onlyMainContent: true,
});
if (result.success) {
scrapeCache.set(key, result);
}
return result;
}
Step 4: Use Batch Scrape for Multiple URLs
const urls = ["https://a.com", "https://b.com", "https://c.com"];
for (const url of urls) {
await firecrawl.scrapeUrl(url, { formats: ["markdown"] });
}
const batchResult = await firecrawl.batchScrapeUrls(urls, {
formats: ["markdown"],
onlyMainContent: true,
});
Step 5: Map Before Crawl (Save Credits)
await firecrawl.crawlUrl("https://docs.example.com", { limit: 1000 });
const map = await firecrawl.mapUrl("https://docs.example.com");
const apiDocs = (map.links || []).filter(url =>
url.includes("/api/") || url.includes("/reference/")
);
console.log(`Map: ${map.links?.length} total, ${apiDocs.length} relevant`);
const result = await firecrawl.batchScrapeUrls(apiDocs.slice(0, 50), {
formats: ["markdown"],
});
Step 6: Measure Scrape Performance
async function timedScrape(url: string) {
const start = Date.now();
const result = await firecrawl.scrapeUrl(url, { formats: ["markdown"] });
const duration = Date.now() - start;
console.log({
url,
durationMs: duration,
contentLength: result.markdown?.length || 0,
success: result.success,
charsPerSecond: Math.round((result.markdown?.length || 0) / (duration / 1000)),
});
return result;
}
Error Handling
| Issue | Cause | Solution |
|---|
| Scrape > 10s | Screenshot or full HTML requested | Use formats: ["markdown"] only |
| Empty content | waitFor too short for SPA | Increase or use actions with selector |
| High credit burn | Scraping same URLs repeatedly | Implement URL-based caching |
| Batch timeout | Too many URLs in one batch | Split into chunks of 50 |
| Cache stale data | TTL too long | Reduce TTL or add cache invalidation |
Examples
Performance Comparison Script
const url = "https://docs.firecrawl.dev";
for (const formats of [["markdown"], ["markdown", "html"], ["markdown", "html", "screenshot"]]) {
const start = Date.now();
await firecrawl.scrapeUrl(url, { formats: formats as any, onlyMainContent: true });
console.log(`${formats.join(",")}: ${Date.now() - start}ms`);
}
Resources
Next Steps
For cost optimization, see firecrawl-cost-tuning.
