firecrawl-architecture-variants

name	firecrawl-architecture-variants
description	Choose and implement Firecrawl architecture patterns for different scales and use cases. Use when designing new Firecrawl integrations, choosing between on-demand/scheduled/pipeline architectures, or planning scraping infrastructure. Trigger with phrases like "firecrawl architecture", "firecrawl blueprint", "how to structure firecrawl", "firecrawl at scale", "firecrawl pipeline design".
allowed-tools	Read, Grep
version	1.11.0
license	MIT
author	Jeremy Longshore <[e-mail]>
tags	["saas","firecrawl","migration","scaling","microservices"]
compatibility	Designed for Claude Code, also compatible with Codex and OpenClaw

Firecrawl Architecture Variants

Overview

Three deployment architectures for Firecrawl at different scales: on-demand scraping for simple use cases, scheduled crawl pipelines for content monitoring, and real-time ingestion pipelines for AI/RAG applications. Choose based on volume, latency requirements, and cost budget.

Decision Matrix

Factor	On-Demand	Scheduled Pipeline	Real-Time Pipeline
Volume	< 500/day	500-10K/day	10K+/day
Latency	Sync (2-10s)	Async (hours)	Async (minutes)
Use Case	Single page lookup	Site monitoring	Knowledge base, RAG
Credit Control	Per-request	Per-crawl budget	Credit pipeline
Complexity	Low	Medium	High

Instructions

Architecture 1: On-Demand Scraping

User Request → Backend API → firecrawl.scrapeUrl → Clean Content → Response

Best for: chatbots, content preview, single-page extraction.

import FirecrawlApp from "@mendable/firecrawl-js";

const firecrawl = new FirecrawlApp({
  apiKey: process.env.FIRECRAWL_API_KEY!,
});

// Simple API endpoint
app.post("/api/scrape", async (req, res) => {
  const { url } = req.body;

  const result = await firecrawl.scrapeUrl(url, {
    formats: ["markdown"],
    onlyMainContent: true,
    : ,
  });

  res.({
    : result.?.,
    : result.,
    : result.?.,
  });
});


app.(,  (req, res) => {
   { url, schema } = req.;

   result =  firecrawl.(url, {
    : [],
    : { schema },
  });

  res.({ : result. });
});