firecrawl-reliability-patterns

name	firecrawl-reliability-patterns
description	Implement FireCrawl reliability patterns including circuit breakers, idempotency, and graceful degradation. Use when building fault-tolerant FireCrawl integrations, implementing retry strategies, or adding resilience to production FireCrawl services. Trigger with phrases like "firecrawl reliability", "firecrawl circuit breaker", "firecrawl idempotent", "firecrawl resilience", "firecrawl fallback", "firecrawl bulkhead".
allowed-tools	Read, Write, Edit
version	1.0.0
license	MIT
author	Jeremy Longshore <[e-mail]>

FireCrawl Reliability Patterns

Overview

Production-grade reliability patterns for FireCrawl integrations.

Prerequisites

• Understanding of circuit breaker pattern
• opossum or similar library installed
• Queue infrastructure for DLQ
• Caching layer for fallbacks

Circuit Breaker

import CircuitBreaker from 'opossum';

const firecrawlBreaker = new CircuitBreaker(
  async (operation: () => Promise<any>) => operation(),
  {
    timeout: 30000,
    errorThresholdPercentage: 50,
    resetTimeout: 30000,
    volumeThreshold: 10,
  }
);

// Events
firecrawlBreaker.on('open', () => {
  console.warn('FireCrawl circuit OPEN - requests failing fast');
  alertOps('FireCrawl circuit breaker opened');
});

firecrawlBreaker.on('halfOpen', () => {
  console.info('FireCrawl circuit HALF-OPEN - testing recovery');
});

firecrawlBreaker.on('close', () => {
  console.info('FireCrawl circuit CLOSED - normal operation');
});

// Usage
async function safeFireCrawlCall<T>(fn: () => Promise<T>): Promise<T> {
  return firecrawlBreaker.fire(fn);
}

Idempotency Keys

import { v4 as uuidv4 } from 'uuid';
import crypto from 'crypto';

// Generate deterministic idempotency key from input
function generateIdempotencyKey(
  operation: string,
  params: Record<string, any>
): string {
  const data = JSON.stringify({ operation, params });
  return crypto.createHash('sha256').update(data).digest('hex');
}

// Or use random key with storage
class IdempotencyManager {
  private store: Map<string, { key: string; expiresAt: Date }> = new Map();

  getOrCreate(operationId: string): string {
    const existing = this.store.get(operationId);
    if (existing && existing.expiresAt > new Date()) {
       existing.;
    }

     key = ();
    ..(operationId, {
      key,
      :  (.() +  *  *  * ),
    });
     key;
  }
}

Bulkhead Pattern

import PQueue from 'p-queue';

// Separate queues for different operations
const firecrawlQueues = {
  critical: new PQueue({ concurrency: 10 }),
  normal: new PQueue({ concurrency: 5 }),
  bulk: new PQueue({ concurrency: 2 }),
};

async function prioritizedFireCrawlCall<T>(
  priority: 'critical' | 'normal' | 'bulk',
  fn: () => Promise<T>
): Promise<T> {
  return firecrawlQueues[priority].add(fn);
}

// Usage
await prioritizedFireCrawlCall('critical', () =>
  firecrawlClient.processPayment(order)
);

await prioritizedFireCrawlCall('bulk', () =>
  firecrawlClient.syncCatalog(products)
);

Timeout Hierarchy

const TIMEOUT_CONFIG = {
  connect: 5000,      // Initial connection
  request: 30000,     // Standard requests
  upload: 120000,     // File uploads
  longPoll: 300000,   // Webhook long-polling
};

async function timedoutFireCrawlCall<T>(
  operation: 'connect' | 'request' | 'upload' | 'longPoll',
  fn: () => Promise<T>
): Promise<T> {
  const timeout = TIMEOUT_CONFIG[operation];

  return Promise.race([
    fn(),
    new Promise<never>((_, reject) =>
      setTimeout(() => reject(new Error(`FireCrawl ${operation} timeout`)), timeout)
    ),
  ]);
}

Graceful Degradation

interface FireCrawlFallback {
  enabled: boolean;
  data: any;
  staleness: 'fresh' | 'stale' | 'very_stale';
}

async function withFireCrawlFallback<T>(
  fn: () => Promise<T>,
  fallbackFn: () => Promise<T>
): Promise<{ data: T; fallback: boolean }> {
  try {
    const data = await fn();
    // Update cache for future fallback
    await updateFallbackCache(data);
    return { data, fallback: false };
  } catch (error) {
    console.warn('FireCrawl failed, using fallback:', error.message);
    const data = await fallbackFn();
    return { data, fallback: true };
  }
}

Dead Letter Queue

interface DeadLetterEntry {
  id: string;
  operation: string;
  payload: any;
  error: string;
  attempts: number;
  lastAttempt: Date;
}

class FireCrawlDeadLetterQueue {
  private queue: DeadLetterEntry[] = [];

  add(entry: Omit<DeadLetterEntry, 'id' | 'lastAttempt'>): void {
    this.queue.push({
      ...entry,
      id: uuidv4(),
      lastAttempt: new Date(),
    });
  }

  async processOne(): Promise<boolean> {
    const entry = this.queue.shift();
    if (!entry) return false;

    try {
      await firecrawlClient[entry.operation](entry.payload);
      console.log(`DLQ: Successfully reprocessed `);
       ;
    }  (error) {
      entry.++;
      entry. =  ();

       (entry. < ) {
        ..(entry);
      }  {
        .();
         (entry);
      }
       ;
    }
  }
}

Health Check with Degraded State

type HealthStatus = 'healthy' | 'degraded' | 'unhealthy';

async function firecrawlHealthCheck(): Promise<{
  status: HealthStatus;
  details: Record<string, any>;
}> {
  const checks = {
    api: await checkApiConnectivity(),
    circuitBreaker: firecrawlBreaker.stats(),
    dlqSize: deadLetterQueue.size(),
  };

  const status: HealthStatus =
    !checks.api.connected ? 'unhealthy' :
    checks.circuitBreaker.state === 'open' ? 'degraded' :
    checks.dlqSize > 100 ? 'degraded' :
    'healthy';

  return { status, details: checks };
}

Instructions

Step 1: Implement Circuit Breaker

Wrap FireCrawl calls with circuit breaker.

Step 2: Add Idempotency Keys

Generate deterministic keys for operations.

Step 3: Configure Bulkheads

Separate queues for different priorities.

Step 4: Set Up Dead Letter Queue

Handle permanent failures gracefully.

Output

• Circuit breaker protecting FireCrawl calls
• Idempotency preventing duplicates
• Bulkhead isolation implemented
• DLQ for failed operations

Error Handling

Issue	Cause	Solution
Circuit stays open	Threshold too low	Adjust error percentage
Duplicate operations	Missing idempotency	Add idempotency key
Queue full	Rate too high	Increase concurrency
DLQ growing	Persistent failures	Investigate root cause

Examples

Quick Circuit Check

const state = firecrawlBreaker.stats().state;
console.log('FireCrawl circuit:', state);

Resources

• Circuit Breaker Pattern
• Opossum Documentation
• FireCrawl Reliability Guide

Next Steps

For policy enforcement, see firecrawl-policy-guardrails.