Skip to main content
firecrawl-advanced-troubleshooting Debug hard-to-diagnose Firecrawl issues with systematic isolation and evidence collection.
Use when standard troubleshooting fails, investigating why scrapes return empty content,
crawl jobs hang, or webhooks don't fire.
Trigger with phrases like "firecrawl hard bug", "firecrawl mystery error",
"firecrawl impossible to debug", "firecrawl deep debug", "firecrawl not scraping".
Jump to install Skills Marketplace Discover and explore AI skills built by the community.
Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.
Show prompt details A direct command skips the review prompt. Inspect the source before running it.
npx skills add https://github.com/jeremylongshore/claude-code-plugins-plus-skills --skill firecrawl-advanced-troubleshootingThe command stays on one line. Scroll horizontally to inspect it before copying.
Prefer a local copy? Download the files currently available to SkillsMP.
Download Zip Downloading... More from this repository Implement user sign-up and sign-in flows with Clerk.
Use when building authentication UI, customizing sign-in experience,
or implementing OAuth social login.
Trigger with phrases like "clerk sign-in", "clerk sign-up",
"clerk login flow", "clerk OAuth", "clerk social login".
Implement session management and middleware with Clerk.
Use when managing user sessions, configuring route protection,
or implementing token refresh and custom JWT templates.
Trigger with phrases like "clerk session", "clerk middleware",
"clerk route protection", "clerk token", "clerk JWT".
Configure enterprise SSO, role-based access control, and organization management.
Use when implementing SSO integration, configuring role-based permissions,
or setting up organization-level controls.
Trigger with phrases like "clerk SSO", "clerk RBAC",
"clerk enterprise", "clerk roles", "clerk permissions", "clerk organizations".
Related occupations SOC
Based on SOC occupation classification
name firecrawl-advanced-troubleshooting description Debug hard-to-diagnose Firecrawl issues with systematic isolation and evidence collection.
Use when standard troubleshooting fails, investigating why scrapes return empty content,
crawl jobs hang, or webhooks don't fire.
Trigger with phrases like "firecrawl hard bug", "firecrawl mystery error",
"firecrawl impossible to debug", "firecrawl deep debug", "firecrawl not scraping".
allowed-tools Read, Grep, Bash(curl:*), Bash(node:*) version 1.11.0 license MIT author Jeremy Longshore <jeremy@intentsolutions.io> tags ["saas","firecrawl","debugging","scaling"] compatibility Designed for Claude Code, also compatible with Codex and OpenClaw
Firecrawl Advanced Troubleshooting
Overview
Deep debugging techniques for complex Firecrawl issues: empty scrapes on certain domains, crawl jobs that never complete, inconsistent extraction results, and webhook delivery failures. Uses systematic layer-by-layer isolation.
Instructions
Step 1: Minimal Reproduction
import FirecrawlApp from "@mendable/firecrawl-js" ;
async function minimalRepro (const firecrawl = new FirecrawlApp ({
apiKey : process.env .FIRECRAWL_API_KEY !,
});
console .log ("Test 1: Basic scrape" );
const basic = await firecrawl.scrapeUrl ("https://example.com" , {
formats : ["markdown" ],
});
console .log (` Success: ${basic.success} , Length: ${basic.markdown?.length} ` );
console .log ("Test 2: Target URL" );
const target = await firecrawl.scrapeUrl ("https://YOUR-FAILING-URL.com" , {
formats : ["markdown" ],
});
console .log (` Success: ${target.success} , Length: ${target.markdown?.length} `
console
log
"Test 3: With JS wait"
const
await
scrapeUrl
"https://YOUR-FAILING-URL.com"
formats
"markdown"
waitFor
10000
onlyMainContent
true
console
log
` Success: ${withWait.success} , Length: ${withWait.markdown?.length} `
console
log
"Test 4: With actions"
const
await
scrapeUrl
"https://YOUR-FAILING-URL.com"
formats
"markdown"
"screenshot"
actions
type
"wait"
milliseconds
3000
type
"scroll"
direction
"down"
type
"wait"
milliseconds
2000
console
log
` Success: ${withActions.success} , Length: ${withActions.markdown?.length} `
Step 2: Layer-by-Layer Isolation async function diagnose (url : string const firecrawl = new FirecrawlApp ({ apiKey : process.env .FIRECRAWL_API_KEY ! });
const results : Array <{ test : string ; pass : boolean ; detail : string }> = [];
try {
await firecrawl.scrapeUrl ("https://example.com" , { formats : ["markdown" ] });
results.push ({ test : "API connectivity" , pass : true , detail : "OK" });
} catch (e : any ) {
results.push ({ test : "API connectivity" , pass : false , detail : `${e.statusCode} : ${e.message} ` });
return results;
}
try {
const result = await firecrawl.scrapeUrl (url, { formats : ["markdown" ] });
const hasContent = (result.markdown ?.length || 0 ) > 50 ;
results.push ({
test : "Target scrape" ,
pass : result.success && hasContent,
detail : `Success: ${result.success} , Chars: ${result.markdown?.length} , Status: ${result.metadata?.statusCode} ` ,
});
} catch (e : any ) {
results.push ({ test : "Target scrape" , pass : false , detail : e.message });
}
try {
const result = await firecrawl.scrapeUrl (url, {
formats : ["markdown" , "html" ],
onlyMainContent : true ,
waitFor : 5000 ,
});
const md = result.markdown || "" ;
const isErrorPage = /404|403|access denied|captcha|blocked/i .test (md);
results.push ({
test : "Content quality" ,
pass : md.length > 100 && !isErrorPage,
detail : `Chars: ${md.length} , Error page: ${isErrorPage} , Has headings: ${/^#{1 ,3 }\s/m.test(md)} ` ,
});
} catch (e : any ) {
results.push ({ test : "Content quality" , pass : false , detail : e.message });
}
try {
const map = await firecrawl.mapUrl (url);
results.push ({
test : "Map endpoint" ,
pass : (map.links ?.length || 0 ) > 0 ,
detail : `Found ${map.links?.length} URLs` ,
});
} catch (e : any ) {
results.push ({ test : "Map endpoint" , pass : false , detail : e.message });
}
return results;
}
const results = await diagnose ("https://YOUR-URL.com" );
console .table (results);
Step 3: Debug Empty Scrapes
async function debugEmptyScrape (url : string const firecrawl = new FirecrawlApp ({ apiKey : process.env .FIRECRAWL_API_KEY ! });
const result = await firecrawl.scrapeUrl (url, {
formats : ["markdown" , "html" , "screenshot" ],
waitFor : 10000 ,
});
console .log ("=== Scrape Debug ===" );
console .log (`URL: ${result.metadata?.sourceURL} ` );
console .log (`Status: ${result.metadata?.statusCode} ` );
console .log (`Markdown length: ${result.markdown?.length || 0 } ` );
console .log (`HTML length: ${result.html?.length || 0 } ` );
console .log (`Title: ${result.metadata?.title} ` );
if ((result.html ?.length || 0 ) > 1000 && (result.markdown ?.length || 0 ) < 100 ) {
console .log ("DIAGNOSIS: HTML has content but markdown extraction failed" );
console .log ("FIX: Content may be in iframes or shadow DOM. Try with actions." );
}
if (/captcha|cloudflare|access denied|please verify/i .test (result.html || "" )) {
console .log ("DIAGNOSIS: Bot detection / CAPTCHA detected" );
console .log ("FIX: Site blocks automated scraping. Contact Firecrawl support." );
}
return result;
}
Step 4: Debug Stuck Crawl Jobs async function debugCrawlJob (jobId : string const firecrawl = new FirecrawlApp ({ apiKey : process.env .FIRECRAWL_API_KEY ! });
const status = await firecrawl.checkCrawlStatus (jobId);
console .log ("=== Crawl Job Debug ===" );
console .log (`Status: ${status.status} ` );
console .log (`Completed: ${status.completed} /${status.total} ` );
console .log (`Error: ${status.error || "none" } ` );
if (status.status === "scraping" && status.completed === status.total ) {
console .log ("DIAGNOSIS: All pages scraped but job not marked complete" );
console .log ("FIX: This is a Firecrawl backend issue. Wait or start a new crawl." );
}
if (status.completed === 0 && status.status === "scraping" ) {
console .log ("DIAGNOSIS: Crawl started but no pages scraped" );
console .log ("FIX: Check if start URL returns content. Try scrapeUrl first." );
}
}
Step 5: Timing Analysis async function timeScrape (url : string , iterations = 5 const firecrawl = new FirecrawlApp ({ apiKey : process.env .FIRECRAWL_API_KEY ! });
const times : number [] = [];
for (let i = 0 ; i < iterations; i++) {
const start = Date .now ();
await firecrawl.scrapeUrl (url, { formats : ["markdown" ] });
times.push (Date .now () - start);
}
times.sort ((a, b ) => a - b);
console .log (`p50: ${times[Math .floor(times.length * 0.5 )]} ms` );
console .log (`p95: ${times[Math .floor(times.length * 0.95 )]} ms` );
console .log (`min: ${times[0 ]} ms, max: ${times[times.length - 1 ]} ms` );
}
Error Handling Issue Cause Solution Empty markdown, HTML exists Shadow DOM or iframes Use actions to interact with page Scrape returns CAPTCHA Bot detection Try with mobile: true, contact Firecrawl Crawl stuck at 0 pages Start URL blocked Verify URL loads in browser first Inconsistent results JS rendering timing Increase waitFor, use selector-based wait Webhook never fires URL unreachable Test with curl to your endpoint first
Support Escalation Template Subject: [P1/P2/P3] [Brief description]
URL: [failing URL]
API Key prefix: fc-xxx (first 6 chars)
Timestamp: [ISO 8601]
Expected: [what should happen]
Actual: [what happens]
Diagnostic output: [paste from diagnose() above]
Screenshot: [if available from screenshot format]
Workarounds tried:
1. [what you tried] — result: [outcome]
Resources
Next Steps For load testing, see firecrawl-load-scale.
