firecrawl-policy-guardrails

name	firecrawl-policy-guardrails
description	Implement Firecrawl scraping policy enforcement: domain blocklists, credit budgets, content filtering, and robots.txt compliance guardrails. Use when setting up scraping policies, enforcing crawl limits, or preventing accidental scraping of prohibited domains. Trigger with phrases like "firecrawl policy", "firecrawl guardrails", "firecrawl domain blocklist", "firecrawl scraping rules", "firecrawl compliance".
allowed-tools	Read, Write, Edit, Bash(npx:*)
version	1.11.0
license	MIT
author	Jeremy Longshore <[e-mail]>
tags	["saas","firecrawl","firecrawl-policy"]
compatibility	Designed for Claude Code, also compatible with Codex and OpenClaw

Firecrawl Policy Guardrails

Overview

Automated guardrails for Firecrawl scraping pipelines. Web scraping carries legal (robots.txt, ToS), ethical (rate limiting, attribution), and cost (credit burn) risks. This skill implements domain blocklists, credit budgets, content quality gates, and per-domain rate limits as enforceable policies.

Instructions

Step 1: Domain Policy Enforcement

import FirecrawlApp from "@mendable/firecrawl-js";

const firecrawl = new FirecrawlApp({
  apiKey: process.env.FIRECRAWL_API_KEY!,
});

class ScrapePolicy {
  // Domains that explicitly prohibit scraping in their ToS
  static BLOCKED_DOMAINS = [
    "facebook.com", "instagram.com",  // Meta ToS
    "linkedin.com",                    // LinkedIn ToS
    "twitter.com", "x.com",           // X/Twitter ToS
  ];

  // Domains with sensitive/regulated content
  static SENSITIVE_DOMAINS = [
    "*.gov", "*.mil",                 // Government
    "*.edu",                          // Educational (FERPA)
  ];

  static validateUrl(url: string): void {
    const hostname = new URL(url).hostname;

     ( blocked  .) {
       (hostname === blocked || hostname.()) {
          ();
      }
    }

     ( pattern  .) {
       regex =  ( + pattern.(, ) + );
       (regex.(hostname)) {
        .();
      }
    }
  }
}

    {
  () {
    (message);
    . = ;
  }
}