tool-firecrawl-scraper

Stars8

Forks0

UpdatedApril 26, 2026 at 20:03

Convert websites into LLM-ready data with Firecrawl API. Scrape, crawl, map, search, extract, agent, batch, change tracking, and branding extraction. Handles JavaScript rendering, anti-bot bypass, PDF/DOCX parsing. Requires FIRECRAWL_API_KEY in .env. Used by other skills (sci-communication, sci-trending-research) as a scraping backend when default web tools fail. Use when: scraping websites, crawling sites, web search + scrape, autonomous data gathering, monitoring content changes, extracting structured data. Does NOT trigger for general browsing or simple URL fetching that WebFetch can handle.

Installation

Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.

Source

krmdel

krmdel/organon

View GitHub Repository View Creator Repositories

Download

Related occupationsSOC

Based on SOC occupation classification

Software DevelopersComputer and Mathematical Occupations·SOC 15-1252

File Explorer

3 files

SKILL.md

readonly

name

tool-firecrawl-scraper

description

Firecrawl Web Scraper

API Version: v2 | Python SDK: firecrawl-py 4.13.0+ | JS SDK: @mendable/firecrawl-js 4.11.1+

Outcome

Website content converted into clean markdown, structured JSON, or extracted data. This skill is both standalone (user asks to scrape) and a backend for other skills (e.g., sci-communication for URL-based content).

Context Needs

File	Load level	Purpose
`context/learnings.md`	`## tool-firecrawl-scraper` section	Known issues, cost patterns

No research_context files needed — this is a utility skill.

Step 1: Check API Key

Requires FIRECRAWL_API_KEY in .env (get one at https://www.firecrawl.dev/app).

Before any Firecrawl call, check .env for the key. If missing, follow the External Services rules in CLAUDE.md:

Explain what Firecrawl would do for this specific task
Tell them how to get it: "Add FIRECRAWL_API_KEY=fc-your-key to your .env file. Free tier at firecrawl.dev gives 500 credits/month."
Offer the fallback: what can be done without it (WebFetch, manual paste, etc.)
Don't block — proceed with the fallback if they don't add the key

Step 2: Choose the Right Endpoint

Endpoint	Purpose	Credits	When to use
`scrape`	Single page → markdown/HTML/JSON	1 (5 stealth)	Articles, product pages, about pages
`crawl`	Full site	1/page	Docs sites, archives
`map`	URL discovery (no content)	1	Sitemap discovery, crawl planning
`search`	Web search + optional scrape	2/10 results	Research with live data
`extract`	Structured data with schema	5/page	Prices, contacts, specs
`agent`	Autonomous AI gathering	Dynamic	No URLs known, complex tasks
`batch-scrape`	Multiple URLs	1/page	Bulk processing
`branding`	Brand identity extraction	1	Colors, fonts, logos, brand traits

For detailed API usage, code examples, and parameters, read references/api-guide.md.

Step 3: Execute the Request

Always prefer the cheapest approach:

Try WebFetch first (free) — only fall back to Firecrawl if it fails
Use cache defaults (2-day) for static content; max_age=0 only for real-time data
Use map before crawl to scope the job and avoid over-crawling
Batch URLs together with batch-scrape instead of individual scrapes

For branding extraction (key capability for brand-related skills):

doc = app.scrape(url="https://example.com", formats=["branding"])

Read references/api-guide.md § Branding Extraction for full details.

Step 4: Validate and Return

Before processing results:

Use auto mode (default) — only charges stealth credits if basic fails
Validate content isn't an error page before processing
Check success field in response, not just HTTP status

For the full list of 10 known issues and their solutions, read references/known-issues.md.

Read references/known-issues.md § Stealth Mode Pricing for credit cost details.

Rules

Updated automatically when the user flags issues. Read before every run.

Self-Update

If the user flags an issue — wrong endpoint, bad code, cost surprise — update the ## Rules section immediately with the correction and today's date.

Troubleshooting

WebFetch works fine: Don't use Firecrawl. It's for JS-heavy sites, bot protection, and structured extraction.
Stealth mode surprise: Auto mode tries basic first, only charges stealth credits if basic fails. Read references/known-issues.md for details.
Empty content returned: The page may be behind authentication or geo-restricted. Check the success field and error message.
High credit usage on crawl: Always run map first to see the site structure and scope the crawl.

More from this repository

same repository

sci-writing

krmdel/organon

Draft manuscript sections, format citations, and get AI peer review. Natural language invocation -- describe what you need and the skill routes to drafting, citation formatting, or peer review. Reads research profile, data analysis outputs, and literature search results for context-aware writing. After drafting, offers to generate supporting figures via viz skills. Triggers on: "draft introduction", "write methods", "format citations", "APA style", "bibliography", "peer review", "review my draft", "write abstract". Does NOT trigger for: literature search (use sci-literature-research), data analysis (use sci-data-analysis), image generation (use viz-nano-banana), repurpose/blog/lay summary (use sci-communication).

2026-07-028

viz-figure-mirror

krmdel/organon

Transfer the visual style of a reference paper figure onto the user's own data. Takes a screenshot of a figure from a paper plus tabular data and produces an editable matplotlib script in that figure's register, gated by a mechanical quality floor and a Drawer/Reviewer refinement loop. Output is a self-contained .py + PNG + Type-42 PDF. Triggers on: "match this figure style", "figure in the style of", "make my plot look like this paper", "publication figure from reference", "style transfer figure", "reproduce this figure with my data", "NeurIPS-quality version of this chart". Does NOT trigger for: plain data plots with no reference image (use sci-data-analysis), AI-generated illustrations (use viz-nano-banana), flowcharts/architecture/sequence diagrams (use viz-diagram-code), hand-drawn sketches (use viz-excalidraw-diagram).

2026-07-028

tool-substack

krmdel/organon

Publish and edit markdown blog posts on Substack as drafts — converts markdown to Substack's ProseMirror schema, uploads local images to Substack's CDN, pre-renders mermaid diagrams to PNG, and creates or updates drafts via Substack's private API. Draft-only by design — publish is always a human click. Triggers on: "push to substack", "substack draft", "publish to substack", "create substack draft", "send to substack", "substack this post", "update substack", "edit substack draft", "list substack drafts". Does NOT trigger for: reading existing Substack posts, subscriber management, newsletter scheduling. Does NOT publish — only creates/updates drafts you review and publish by hand.

2026-05-128

meta-skill-creator

krmdel/organon

Create new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy.

2026-04-268

meta-wrap-up

krmdel/organon

End-of-session checklist that reviews deliverables, collects feedback, fixes skills, updates learnings, and commits work. Use when the user says "wrap up", "close session", "end session", "wrap things up", "we're done", "that's it for today", "session done", or invokes /wrap-up. Run at the end of any working session or after completing a major deliverable. Does NOT trigger for content writing, voice extraction, positioning, or audience research.

2026-04-268

ops-cron

krmdel/organon

Schedule recurring tasks that run automatically via a system-level watchdog, even when Claude Code is closed. Job definitions persist in cron/jobs/ as the registry. The watchdog (launchd on Mac, Task Scheduler on Windows) checks every hour and runs due jobs headlessly via claude -p. Triggers on: "schedule a job", "cron job", "run this every morning", "automate daily", "recurring task", "set up a scheduled job", "check scheduled jobs", "list scheduled jobs", "install watchdog", "uninstall watchdog". Does NOT trigger for one-off tasks or in-session reminders.

2026-04-268