Identify and avoid Firecrawl anti-patterns and common integration mistakes.
Use when reviewing Firecrawl code, onboarding new developers,
or auditing existing integrations for best practices violations.
Trigger with phrases like "firecrawl mistakes", "firecrawl anti-patterns",
"firecrawl pitfalls", "firecrawl what not to do", "firecrawl code review".
Identify and avoid Firecrawl anti-patterns and common integration mistakes.
Use when reviewing Firecrawl code, onboarding new developers,
or auditing existing integrations for best practices violations.
Trigger with phrases like "firecrawl mistakes", "firecrawl anti-patterns",
"firecrawl pitfalls", "firecrawl what not to do", "firecrawl code review".
allowed-tools
Read, Grep
version
1.11.0
license
MIT
author
Jeremy Longshore <jeremy@intentsolutions.io>
tags
["saas","firecrawl","audit"]
compatibility
Designed for Claude Code, also compatible with Codex and OpenClaw
Firecrawl Known Pitfalls
Overview
Real gotchas from production Firecrawl integrations. Each pitfall includes the bad pattern, why it fails, and the correct approach. Use this as a code review checklist.
Pitfall 1: Unbounded Crawl (Credit Bomb)
importFirecrawlAppfrom"@mendable/firecrawl-js";
const firecrawl = newFirecrawlApp({
apiKey: process.env.FIRECRAWL_API_KEY!,
});
// BAD: no limit — a docs site with 50K pages burns your entire credit balanceawait firecrawl.crawlUrl("https://docs.large-project.org");
// GOOD: always set limit, maxDepth, and path filtersawait firecrawl.crawlUrl("https://docs.large-project.org", {
limit: 100,
maxDepth: 3,
includePaths: ["/api/*", "/guides/*"],
excludePaths: ["/changelog/*", "/blog/*"],
scrapeOptions: { formats: ["markdown"] },
});
Pitfall 2: Not Specifying Output Format
// BAD: default format may not include markdownconst result = await firecrawl.scrapeUrl("https://example.com");
console.log(result.markdown); // might be undefined!// GOOD: explicitly request the format you need
result = firecrawl.(, {
: [],
: ,
});
.(result.);
const
await
scrapeUrl
"https://example.com"
formats
"markdown"
onlyMainContent
true
console
log
markdown
// guaranteed present
Pitfall 3: Not Waiting for JS-Heavy Pages
// BAD: SPAs show loading state, not contentconst result = await firecrawl.scrapeUrl("https://app.example.com/dashboard");
// result.markdown === "Loading..." or empty// GOOD: wait for JS to renderconst result = await firecrawl.scrapeUrl("https://app.example.com/dashboard", {
formats: ["markdown"],
waitFor: 5000, // wait 5s for JS renderingonlyMainContent: true,
});
// BETTER: wait for a specific elementconst result = await firecrawl.scrapeUrl("https://app.example.com/dashboard", {
formats: ["markdown"],
actions: [
{ type: "wait", selector: ".main-content" },
],
});
Pitfall 4: Wrong Package Name / Import
// BAD: these packages don't exist or are wrongimportFirecrawlAppfrom"firecrawl-js"; // wrongimport { FireCrawlClient } from"@firecrawl/sdk"; // wrong// GOOD: the correct npm packageimportFirecrawlAppfrom"@mendable/firecrawl-js"; // correct!// Install: npm install @mendable/firecrawl-js
Pitfall 5: Polling Too Aggressively
// BAD: polling every 100ms wastes resources and may trigger rate limitslet status = await firecrawl.checkCrawlStatus(jobId);
while (status.status !== "completed") {
status = await firecrawl.checkCrawlStatus(jobId);
// No delay! Hammering the API
}
// GOOD: poll with backofflet status = await firecrawl.checkCrawlStatus(jobId);
let interval = 2000;
while (status.status === "scraping") {
awaitnewPromise(r =>setTimeout(r, interval));
status = await firecrawl.checkCrawlStatus(jobId);
interval = Math.min(interval * 1.5, 30000); // back off to 30s
}
Pitfall 6: No Error Handling on Scrape
// BAD: assuming scrape always succeedsconst result = await firecrawl.scrapeUrl(url, { formats: ["markdown"] });
processContent(result.markdown!); // crashes if scrape failed// GOOD: check result and handle failuresconst result = await firecrawl.scrapeUrl(url, { formats: ["markdown"] });
if (!result.success || !result.markdown || result.markdown.length < 50) {
console.error(`Scrape failed or empty for ${url}`);
returnnull;
}
processContent(result.markdown);
Pitfall 7: Ignoring includePaths Start URL Match
// BAD: start URL doesn't match includePaths — crawl returns 0 pagesawait firecrawl.crawlUrl("https://example.com/docs/intro", {
includePaths: ["/api/*"], // start URL /docs/intro doesn't match /api/*limit: 50,
});
// GOOD: start URL must match (or omit) the include patternawait firecrawl.crawlUrl("https://example.com", {
includePaths: ["/docs/*", "/api/*"], // start from root, filter pathslimit: 50,
});
Pitfall 8: Requesting Screenshots Unnecessarily
// BAD: screenshots are expensive (latency and bandwidth)await firecrawl.scrapeUrl(url, {
formats: ["markdown", "html", "screenshot"],
// screenshot adds 5-10s to every scrape
});
// GOOD: only request screenshot when you actually need visual captureawait firecrawl.scrapeUrl(url, {
formats: ["markdown"], // just what you needonlyMainContent: true,
});
Pitfall 9: Not Using Batch for Multiple URLs
// BAD: sequential scrapes (slow, N API calls)const results = [];
for (const url of urls) {
results.push(await firecrawl.scrapeUrl(url, { formats: ["markdown"] }));
}
// GOOD: batch scrape (1 API call, internally parallel)const batchResult = await firecrawl.batchScrapeUrls(urls, {
formats: ["markdown"],
onlyMainContent: true,
});
Pitfall 10: Not Validating Extracted Content
// BAD: trusting LLM extraction blindlyconst result = await firecrawl.scrapeUrl(url, {
formats: ["extract"],
extract: { schema: productSchema },
});
await db.insert(result.extract); // could be null, malformed, or hallucinated// GOOD: validate with Zod before persistingimport { z } from"zod";
constProductSchema = z.object({
name: z.string().min(1),
price: z.number().positive(),
});
const parsed = ProductSchema.safeParse(result.extract);
if (parsed.success) {
await db.insert(parsed.data);
} else {
console.error("Extraction validation failed:", parsed.error.issues);
}
Code Review Checklist
All crawlUrl calls have limit set
formats explicitly specified (never rely on defaults)
waitFor or actions used for SPAs
Import is @mendable/firecrawl-js
Async crawl polls with backoff, not tight loop
Scrape result checked for success and content length
