Firecrawl Advanced Troubleshooting
Overview
Deep debugging techniques for complex Firecrawl issues: empty scrapes on certain domains, crawl jobs that never complete, inconsistent extraction results, and webhook delivery failures. Uses systematic layer-by-layer isolation.
Instructions
Step 1: Minimal Reproduction
import FirecrawlApp from "@mendable/firecrawl-js";
async function minimalRepro() {
const firecrawl = new FirecrawlApp({
apiKey: process.env.FIRECRAWL_API_KEY!,
});
console.log("Test 1: Basic scrape");
const basic = await firecrawl.scrapeUrl("https://example.com", {
formats: ["markdown"],
});
console.log(` Success: ${basic.success}, Length: ${basic.markdown?.length}`);
console.log("Test 2: Target URL");
const target = await firecrawl.scrapeUrl("https://YOUR-FAILING-URL.com", {
formats: ["markdown"],
});
console.log(` Success: ${target.success}, Length: ${target.markdown?.length}`);
console.log("Test 3: With JS wait");
const withWait = await firecrawl.scrapeUrl("https://YOUR-FAILING-URL.com", {
formats: ["markdown"],
waitFor: 10000,
onlyMainContent: true,
});
console.log(` Success: ${withWait.success}, Length: ${withWait.markdown?.length}`);
console.log("Test 4: With actions");
const withActions = await firecrawl.scrapeUrl("https://YOUR-FAILING-URL.com", {
formats: ["markdown", "screenshot"],
actions: [
{ type: "wait", milliseconds: 3000 },
{ type: "scroll", direction: "down" },
{ type: "wait", milliseconds: 2000 },
],
});
console.log(` Success: ${withActions.success}, Length: ${withActions.markdown?.length}`);
}
Step 2: Layer-by-Layer Isolation
async function diagnose(url: string) {
const firecrawl = new FirecrawlApp({ apiKey: process.env.FIRECRAWL_API_KEY! });
const results: Array<{ test: string; pass: boolean; detail: string }> = [];
try {
await firecrawl.scrapeUrl("https://example.com", { formats: ["markdown"] });
results.push({ test: "API connectivity", pass: true, detail: "OK" });
} catch (e: any) {
results.push({ test: "API connectivity", pass: false, detail: `${e.statusCode}: ${e.message}` });
return results;
}
try {
const result = await firecrawl.(url, { : [] });
hasContent = (result.?. || ) > ;
results.({
: ,
: result. && hasContent,
: ,
});
} (: ) {
results.({ : , : , : e. });
}
{
result = firecrawl.(url, {
: [, ],
: ,
: ,
});
md = result. || ;
isErrorPage = .(md);
results.({
: ,
: md. > && !isErrorPage,
: ,
});
} (: ) {
results.({ : , : , : e. });
}
{
map = firecrawl.(url);
results.({
: ,
: (map.?. || ) > ,
: ,
});
} (: ) {
results.({ : , : , : e. });
}
results;
}
results = ();
.(results);
Step 3: Debug Empty Scrapes
async function debugEmptyScrape(url: string) {
const firecrawl = new FirecrawlApp({ apiKey: process.env.FIRECRAWL_API_KEY! });
const result = await firecrawl.scrapeUrl(url, {
formats: ["markdown", "html", "screenshot"],
waitFor: 10000,
});
console.log("=== Scrape Debug ===");
console.log(`URL: ${result.metadata?.sourceURL}`);
console.log(`Status: ${result.metadata?.statusCode}`);
console.log(`Markdown length: ${result.markdown?.length || 0}`);
console.log(`HTML length: ${result.html?.length || 0}`);
console.log(`Title: ${result.metadata?.title}`);
((result.?. || ) > && (result.?. || ) < ) {
.();
.();
}
(.(result. || )) {
.();
.();
}
result;
}
Step 4: Debug Stuck Crawl Jobs
async function debugCrawlJob(jobId: string) {
const firecrawl = new FirecrawlApp({ apiKey: process.env.FIRECRAWL_API_KEY! });
const status = await firecrawl.checkCrawlStatus(jobId);
console.log("=== Crawl Job Debug ===");
console.log(`Status: ${status.status}`);
console.log(`Completed: ${status.completed}/${status.total}`);
console.log(`Error: ${status.error || "none"}`);
if (status.status === "scraping" && status.completed === status.total) {
console.log("DIAGNOSIS: All pages scraped but job not marked complete");
console.log("FIX: This is a Firecrawl backend issue. Wait or start a new crawl.");
}
if (status.completed === 0 && status.status === "scraping") {
console.();
.();
}
}
Step 5: Timing Analysis
async function timeScrape(url: string, iterations = 5) {
const firecrawl = new FirecrawlApp({ apiKey: process.env.FIRECRAWL_API_KEY! });
const times: number[] = [];
for (let i = 0; i < iterations; i++) {
const start = Date.now();
await firecrawl.scrapeUrl(url, { formats: ["markdown"] });
times.push(Date.now() - start);
}
times.sort((a, b) => a - b);
console.log(`p50: ${times[Math.floor(times.length * 0.5)]}ms`);
console.log(`p95: ${times[Math.floor(times.length * 0.95)]}ms`);
console.log(`min: ${times[0]}ms, max: ${times[times.length - 1]}ms`);
}
Error Handling
| Issue | Cause | Solution |
|---|
| Empty markdown, HTML exists | Shadow DOM or iframes | Use actions to interact with page |
| Scrape returns CAPTCHA | Bot detection | Try with mobile: true, contact Firecrawl |
| Crawl stuck at 0 pages | Start URL blocked | Verify URL loads in browser first |
| Inconsistent results | JS rendering timing | Increase waitFor, use selector-based wait |
| Webhook never fires | URL unreachable | Test with curl to your endpoint first |
Support Escalation Template
Subject: [P1/P2/P3] [Brief description]
URL: [failing URL]
API Key prefix: fc-xxx (first 6 chars)
Timestamp: [ISO 8601]
Expected: [what should happen]
Actual: [what happens]
Diagnostic output: [paste from diagnose() above]
Screenshot: [if available from screenshot format]
Workarounds tried:
1. [what you tried] — result: [outcome]
Resources
Next Steps
For load testing, see firecrawl-load-scale.
