Firecrawl Known Pitfalls
Overview
Real gotchas from production Firecrawl integrations. Each pitfall includes the bad pattern, why it fails, and the correct approach. Use this as a code review checklist.
Pitfall 1: Unbounded Crawl (Credit Bomb)
import FirecrawlApp from "@mendable/firecrawl-js";
const firecrawl = new FirecrawlApp({
apiKey: process.env.FIRECRAWL_API_KEY!,
});
await firecrawl.crawlUrl("https://docs.large-project.org");
await firecrawl.crawlUrl("https://docs.large-project.org", {
limit: 100,
maxDepth: 3,
includePaths: ["/api/*", "/guides/*"],
excludePaths: ["/changelog/*", "/blog/*"],
scrapeOptions: { formats: ["markdown"] },
});
Pitfall 2: Not Specifying Output Format
const result = await firecrawl.scrapeUrl("https://example.com");
console.log(result.markdown);
const result = await firecrawl.scrapeUrl("https://example.com", {
formats: ["markdown"],
onlyMainContent: true,
});
console.log(result.markdown);
Pitfall 3: Not Waiting for JS-Heavy Pages
const result = await firecrawl.scrapeUrl("https://app.example.com/dashboard");
const result = await firecrawl.scrapeUrl("https://app.example.com/dashboard", {
formats: ["markdown"],
waitFor: 5000,
onlyMainContent: true,
});
const result = await firecrawl.scrapeUrl("https://app.example.com/dashboard", {
formats: ["markdown"],
actions: [
{ type: "wait", selector: ".main-content" },
],
});
Pitfall 4: Wrong Package Name / Import
import FirecrawlApp from "firecrawl-js";
import { FireCrawlClient } from "@firecrawl/sdk";
import FirecrawlApp from "@mendable/firecrawl-js";
Pitfall 5: Polling Too Aggressively
let status = await firecrawl.checkCrawlStatus(jobId);
while (status.status !== "completed") {
status = await firecrawl.checkCrawlStatus(jobId);
}
let status = await firecrawl.checkCrawlStatus(jobId);
let interval = 2000;
while (status.status === "scraping") {
await new Promise(r => setTimeout(r, interval));
status = await firecrawl.checkCrawlStatus(jobId);
interval = Math.min(interval * 1.5, 30000);
}
Pitfall 6: No Error Handling on Scrape
const result = await firecrawl.scrapeUrl(url, { formats: ["markdown"] });
processContent(result.markdown!);
const result = await firecrawl.scrapeUrl(url, { formats: ["markdown"] });
if (!result.success || !result.markdown || result.markdown.length < 50) {
console.error(`Scrape failed or empty for ${url}`);
return null;
}
processContent(result.markdown);
Pitfall 7: Ignoring includePaths Start URL Match
await firecrawl.crawlUrl("https://example.com/docs/intro", {
includePaths: ["/api/*"],
limit: 50,
});
await firecrawl.crawlUrl("https://example.com", {
includePaths: ["/docs/*", "/api/*"],
limit: 50,
});
Pitfall 8: Requesting Screenshots Unnecessarily
await firecrawl.scrapeUrl(url, {
formats: ["markdown", "html", "screenshot"],
});
await firecrawl.scrapeUrl(url, {
formats: ["markdown"],
onlyMainContent: true,
});
Pitfall 9: Not Using Batch for Multiple URLs
const results = [];
for (const url of urls) {
results.push(await firecrawl.scrapeUrl(url, { formats: ["markdown"] }));
}
const batchResult = await firecrawl.batchScrapeUrls(urls, {
formats: ["markdown"],
onlyMainContent: true,
});
Pitfall 10: Not Validating Extracted Content
const result = await firecrawl.scrapeUrl(url, {
formats: ["extract"],
extract: { schema: productSchema },
});
await db.insert(result.extract);
import { z } from "zod";
const ProductSchema = z.object({
name: z.string().min(1),
price: z.number().positive(),
});
const parsed = ProductSchema.safeParse(result.extract);
if (parsed.success) {
await db.insert(parsed.data);
} else {
console.error("Extraction validation failed:", parsed.error.issues);
}
Code Review Checklist
Resources
Next Steps
For reference architecture, see firecrawl-reference-architecture.
