connector-firecrawl

name	connector-firecrawl
description	Use when a legal-AI workflow needs to convert any publicly accessible URL into clean, LLM-ready markdown — for example, to retrieve a regulatory authority's latest guidance, a court's published judgment, a government gazette notice, or a law firm's published article for research or analysis. Requires a per-tenant Firecrawl API key. Triggers on any request to read or summarize a web page, scrape a regulatory site, or ingest an external URL as context for legal analysis.
license	MIT
metadata	{"id":"connector.firecrawl","category":"connector","jurisdictions":["__multi__"],"priority":"P2","intent":["__connector__"],"related":["connector-eur-lex","connector-legifrance","connector-legal-data-hunter","connector-companies-house-uk"],"source":"Louis — HAQQ Legal AI (github.com/sboghossian/mini-claude-for-legal)","version":"1.0"}

Connector — Firecrawl

What it does

Firecrawl is a web-scraping and content-extraction service that converts any URL — including JavaScript-rendered pages — into clean markdown suitable for LLM processing. In a legal-AI context, it serves as the general-purpose web-content ingestion layer when no structured API connector exists for a particular source.

Core capabilities:

• Converts a single URL to markdown (stripping navigation, ads, and boilerplate).
• Crawls multi-page documentation or regulation sites with configurable depth.
• Extracts structured data from HTML tables (e.g., fee schedules, sanctions lists in HTML format).
• Handles JavaScript-rendered pages (single-page apps, dynamic government portals).

Setup / auth

Authentication uses a per-tenant Firecrawl API key:

• Obtained from the Firecrawl dashboard (firecrawl.dev).
• Stored in the platform's secrets manager under the tenant's configuration.
• Key is passed as a Bearer token in the Authorization header.
• Rate limits depend on the plan; legal-use volumes are typically within the standard plan.

Each tenant has their own API key to ensure usage tracking and cost isolation. Never share a Firecrawl API key across tenants.

Capabilities

Capability	Notes
Single URL to markdown	Core function; strips boilerplate; returns clean text
JavaScript-rendered pages	Headless browser rendering; handles SPAs
Crawl (multi-page)	Configurable depth and domain scope
HTML table extraction	Returns tables as markdown or JSON
PDF URL to markdown	Downloads and extracts text from PDFs linked at the URL
Screenshot of page	Returns a PNG of the rendered page
Structured data extraction	With a schema, returns JSON matching the schema from page content

Usage patterns

Pattern 1 — Regulatory guidance retrieval

User: "Get the latest DIFC DIFCA circular on AML requirements"
→ Connector scrapes the DIFCA website for the relevant circular URL
→ Firecrawl converts the page to clean markdown
→ Returns the circular text as context for analysis or citation

Pattern 2 — Government gazette monitoring

Many MENA government decisions are published in official gazettes (e.g., UAE Official Gazette, Saudi Umm al-Qura, Lebanese Official Journal) which do not have structured APIs:

• Use Firecrawl to retrieve the latest gazette issue from the official URL.
• Extract relevant notices (new laws, ministerial decisions, court rule amendments).
• Surface changes relevant to matters currently in the work queue.

Pattern 3 — Counterparty website for due diligence context

Before a client meeting or due-diligence review, scrape the counterparty's corporate website to extract:

• Business description and sector.
• Key management team.
• Publicly disclosed subsidiaries or affiliates.
• Any red flags in published materials (sanctions disclosures, litigation mentions).

Note: website content is marketing material — treat it as orientation, not verified fact.

Pattern 4 — Case law from court publication portals

Some MENA courts publish judgments on their websites (e.g., DIFC Courts Case Register, Abu Dhabi courts). Use Firecrawl to retrieve judgment text where no structured API exists:

• Provide the judgment URL.
• Return clean markdown for analysis.
• Cross-reference the judgment details with [[connector-legal-data-hunter]] for MENA-specific structuring.

Pattern 5 — Monitoring a regulatory authority for updates

Set up a scheduled crawl (via [[connector-scheduled-tasks]]) to check a regulatory authority's "news" or "publications" page periodically and alert the relevant matter team when new content appears that matches monitored keywords.

What Firecrawl does NOT do

• No login-walled content. Firecrawl cannot authenticate to sites requiring a username/password. Use dedicated authenticated connectors (e.g., [[connector-legifrance]] for French law, [[connector-eur-lex]] for EU law).
• No Westlaw / LexisNexis. These are paywalled and have terms of service prohibiting scraping.
• No PACER. US federal court filings (PACER) require a registered account and separate integration.
• No real-time data. Firecrawl captures a snapshot; for live-updating data (e.g., share prices, live sanctions lists), use purpose-built connectors.

Jurisdictional relevance for MENA practice

Source type	Availability via Firecrawl	Notes
UAE Federal laws portal (moj.gov.ae)	Good	Some pages JS-rendered; Firecrawl handles this
DIFC Courts (difccourts.ae)	Good	Judgment register publicly accessible
Lebanon Official Journal (gazetteofficielle.gov.lb)	Partial	Arabic/French; occasional PDF-only editions
KSA Umm al-Qura (uqn.gov.sa)	Good	Arabic; Firecrawl returns Arabic markdown
ADGM legal resources (adgm.com)	Good	English; well-structured HTML
Egypt Official Gazette (vetogate.com mirrors)	Partial	Official site inconsistent; use mirrors carefully

When returning content from Arabic-language sources, preserve the Arabic text in the markdown output — do not silently transliterate or omit Arabic characters.

Permissions & safety

• Respect robots.txt. Firecrawl respects robots.txt by default. Do not override this for government or court sites.
• No credential-passing. Never pass authentication tokens or session cookies to Firecrawl — the content will transit Firecrawl's infrastructure.
• Rate limiting. For periodic crawls of the same site, add a minimum 60-second delay between requests to avoid overloading government web servers.
• Content accuracy. Scraped content must be cited with the source URL and retrieval date. Never present scraped regulatory text as definitively current without noting the retrieval timestamp.
• PII in scraped content. If a scrape returns pages containing personal information (e.g., a court judgment with an individual's name and address), handle as PII — do not store beyond the immediate session unless explicitly authorized.

Failure modes

Failure	Cause	Resolution
403 Forbidden	Site blocks scrapers	Try with a user-agent string; some sites allow with proper UA
Empty markdown	JavaScript-heavy SPA not rendered	Check if Firecrawl's JS rendering is enabled; increase render wait time
Truncated content	Page too long	Use Firecrawl's limit and offset pagination
PDF not parsed	PDF is scanned image (not text-layer)	Route to multimodal OCR handler instead
Rate limit exceeded	Too many requests on the plan	Queue requests; upgrade plan if needed

Related skills

• [[connector-eur-lex]]
• [[connector-legifrance]]
• [[connector-legal-data-hunter]]
• [[connector-companies-house-uk]]
• [[connector-scheduled-tasks]]