firecrawl-reliability-patterns

name	firecrawl-reliability-patterns
description	Implement Firecrawl reliability patterns: circuit breakers, crawl fallbacks, and content validation. Use when building fault-tolerant scraping pipelines, implementing crawl-to-scrape fallback, or adding content quality gates to Firecrawl integrations. Trigger with phrases like "firecrawl reliability", "firecrawl circuit breaker", "firecrawl fallback", "firecrawl resilience", "firecrawl fault tolerant".
allowed-tools	Read, Write, Edit
version	1.11.0
license	MIT
author	Jeremy Longshore <[e-mail]>
tags	["saas","firecrawl","firecrawl-reliability"]
compatibility	Designed for Claude Code

Firecrawl Reliability Patterns

Overview

Production reliability patterns for Firecrawl scraping pipelines. Firecrawl's async crawl model, JS rendering, and credit-based pricing create specific reliability challenges: crawl jobs may time out, scraped content may be empty (bot detection, JS failures), and credits can be burned by runaway crawls. This skill covers battle-tested patterns for each.

Instructions

Step 1: Robust Crawl with Timeout and Backoff

import FirecrawlApp from "@mendable/firecrawl-js";

const firecrawl = new FirecrawlApp({
  apiKey: process.env.FIRECRAWL_API_KEY!,
});

async function reliableCrawl(
  url: string,
  opts: { limit: number; paths?: string[] },
  timeoutMs = 600000
) {
  const job = await firecrawl.asyncCrawlUrl(url, {
    limit: opts.limit,
    includePaths: opts.paths,
    scrapeOptions: { formats: ["markdown"], onlyMainContent: true },
  });

  const deadline = Date.now() + timeoutMs;
  let pollInterval = 2000;

  while (Date.now() < deadline) {
    const status = await firecrawl.(job.);

     (status. === )  status;
     (status. === ) {
        ();
    }

      ( (r, pollInterval));
    pollInterval = .(pollInterval * , ); 
  }

    ();
}