菜单
Web scraping and structured data extraction. Activate when CC needs to pull content from competitor sites, extract pricing/contacts/listings, harvest data for research, scrape pages that don't have an API, or operate a site under CC's logged-in account.
用 Codex 或 Claude 帮你安装 复制这段 Prompt，粘贴到 Codex、Claude 或其他助手里，让它检查 Skill 页面并帮你完成安装。
基于 SOC 职业分类
| name | web-scraping |
| description | Web scraping and structured data extraction. Activate when CC needs to pull content from competitor sites, extract pricing/contacts/listings, harvest data for research, scrape pages that don't have an API, or operate a site under CC's logged-in account. |
| tags | ["skill","scraping","data-extraction","research","browser-harness"] |
| triggers | ["scrape a website","extract competitor pricing","pull data from this page","crawl this site for content","harvest leads from web"] |
Three tools, three jobs. Firecrawl extracts public data. Playwright automates throwaway-browser interactions. Browser Harness drives CC's real, logged-in Chrome for actions that require authenticated sessions.
| Scenario | Tool | Why |
|---|---|---|
| Extract text/pricing/contacts from a public page | Firecrawl | Returns clean markdown, handles JS-rendered pages, no browser overhead |
| Crawl an entire site for content harvesting | Firecrawl crawl | Follows links automatically up to the limit |
| Extract structured data (pricing tables, job listings, profiles) | Firecrawl extract | LLM-powered schema extraction — gets exactly the fields you specify |
| Get all URLs on a domain for analysis | Firecrawl map | Purpose-built for site mapping |
| Search for pages and extract their content in one step | Firecrawl search | Search + scrape in a single call |
| Automate a generic web workflow on a site you don't need to be logged into | Playwright MCP | Throwaway browser, deterministic, doesn't require user sessions |
| Take screenshots for visual reference / E2E test | Playwright MCP | Firecrawl doesn't capture visuals; Playwright is purpose-built for it |
| Act inside CC's logged-in account — Skool community, Stripe dashboard, Supabase, Vercel, LinkedIn profile views, anywhere "log in fresh" isn't an option | Browser Harness | Attaches to CC's already-running Chrome at port 9222. The session, cookies, MFA, and reputation are all already there. |
| Scrape a site that aggressively blocks bot browsers (Cloudflare, PerimeterX, Datadome) when you must read it as CC | Browser Harness | Anti-bot systems can't easily distinguish it from a real human's browser — because it IS one |
| Read pages that ONLY render under a specific extension/persona Chrome has installed | Browser Harness | Inherits whatever extensions and prefs CC's Chrome has |
Rule of thumb (in priority order):
Never use Browser Harness for what Firecrawl can do — it's heavier and depends on Chrome being open. Never use Playwright for what Browser Harness can do — fresh-session bot detection will block you on protected sites.
# Scrape a single page → clean markdown
python scripts/firecrawl_tool.py scrape https://example.com
# Crawl a site (follows links, max 10 pages)
python scripts/firecrawl_tool.py crawl https://example.com --limit 10
# Search query → scrape top results
python scripts/firecrawl_tool.py search "AI automation agencies Ontario"
# Extract structured data with a schema
python scripts/firecrawl_tool.py extract https://example.com/pricing \
--schema '{"type":"object","properties":{"plans":{"type":"array"}}}'
# Get all URLs on a domain
python scripts/firecrawl_tool.py map https://example.com
# Machine-readable JSON (for agent pipelines)
python scripts/firecrawl_tool.py scrape https://example.com --json
# Get their full pricing page
python scripts/firecrawl_tool.py scrape https://competitor.com/pricing
# Map what pages exist on their site
python scripts/firecrawl_tool.py map https://competitor.com
# Extract pricing structured data
python scripts/firecrawl_tool.py extract https://competitor.com/pricing \
--schema '{"type":"object","properties":{"plans":{"type":"array","items":{"type":"object","properties":{"name":{"type":"string"},"price":{"type":"string"},"features":{"type":"array","items":{"type":"string"}}}}}}}'
# Understand what a prospect's business does before the call
python scripts/firecrawl_tool.py scrape https://prospect.com
# Crawl their full site for comprehensive understanding
python scripts/firecrawl_tool.py crawl https://prospect.com --limit 15
# Search for industry articles to inform content strategy
python scripts/firecrawl_tool.py search "AI automation for HVAC businesses 2025"
# Crawl a resource site for research
python scripts/firecrawl_tool.py crawl https://industry-blog.com --limit 25
# Extract job listing data for hiring research
python scripts/firecrawl_tool.py extract https://jobs.example.com \
--schema '{"type":"object","properties":{"jobs":{"type":"array","items":{"type":"object","properties":{"title":{"type":"string"},"salary":{"type":"string"},"location":{"type":"string"}}}}}}'
# Search for competitor pricing intelligence
python scripts/firecrawl_tool.py search "HVAC AI software pricing 2025" --json
Set in .env.agents:
FIRECRAWL_API_KEY=fc-xxxxxxxxxxxxx
Get your key at firecrawl.dev. The free tier covers most agent research tasks.
| Feature | Firecrawl | Playwright MCP | Browser Harness |
|---|---|---|---|
| JS-rendered pages | Yes (cloud browser) | Yes (local browser) | Yes (CC's real Chrome) |
| Clean markdown output | Yes (built-in) | No (raw HTML/snapshot) | No (raw page state) |
| Login/auth sessions | No | Throwaway only — fresh login each run | Yes — CC's real Chrome session |
| Form submission | No | Yes | Yes |
| Anti-bot detection risk | Cloud-side (handled) | High on protected sites (Cloudflare etc.) | Lowest — it IS a real browser |
| Structured extraction | Yes (LLM schema) | No (manual parsing) | No (manual parsing) |
| Batch crawling | Yes (crawl) | No (manual loop) | No (manual loop) |
| Site mapping | Yes (map) | No | No |
| Screenshots | No | Yes | Yes |
| Works without API key | No | Yes | Yes (but needs Chrome already running) |
Survives bravo doctor-quality automation | Yes — stateless | Yes — stateless | Requires the harness daemon up + Chrome attached |
| Setup overhead per session | None | None | One-time bravo browser setup per machine, then leave Chrome open |
Browser Harness shines when the URL you need fails on the other two:
Run bravo browser doctor to confirm the daemon is attached to Chrome
before you start. If it isn't, run bravo browser setup once.
scrape output into scripts/competitive_intel.py addcrawl output into scripts/lead_engine.py addsearch results into content ideation/ingest scraped content into knowledge/raw/scripts/firecrawl_tool.py | scripts/browser_harness_doctor.py | [[browser/README]] | [[browser/SAFETY]]Maven scrapes: Meta Ad Library (competitor creative), Google Ads Transparency Center, LinkedIn ad library, TikTok Top Ads, AdSpy/Anstrex for paid intel, and public company landing pages for positioning analysis. Scraping respects robots.txt where legally relevant and never bypasses paywalls. Output lands in brain/intel/<vertical>/ for canon citation.
Higgsfield image prompt director covering Banana Pro, Soul Cinema, and GPT-2. Three asset types in strict order — single-image character outfit on white seamless studio (Banana Pro or Soul Cinema two-step), 6-panel multi-angle character sheet off that base, and scene plates with/without characters. Plus GPT-2 for detail face/chest-up portraits. Reads reference images for hair, makeup, wardrobe, jewelry, identity. Outputs photorealistic prompts with locked hyperreal stack — skin pores, subsurface scattering, strand-by-strand hair, fabric weave, Kodak Vision3 film. Use for any character outfit reference, character sheet, model sheet, ref sheet, multi-angle sheet, scene plate, environment plate, face detail, or photorealistic still — even without saying 'Banana Pro,' 'Nano Banana,' or 'Soul Cinema.'
Universal cinema worldbuilding director for Seedance video prompts. Five cinema modes (Narrative, Studio, Action, Performance, Atmospheric) with locked ARRI + Panavision/Cooke camera specs and diegetic-only audio rules. Reads reference images for wardrobe, hair, makeup, identity, and environment. Outputs production-ready Seedance prompts. Use whenever CC asks for a Seedance video prompt, mentions Seedance, asks for cinematic scene breakdowns, uploads reference images for a scene, describes a scene for video generation, or asks for shot prompts for music videos, action sequences, performance scenes, narrative shorts, fashion films, or atmospheric environment plates — even if 'cinematic' or a mode name isn't said.
Complete automated video production system for CC's content pipeline. Use whenever CC provides raw video footage and says "make this a post", "edit this", or asks for a cinematic output. Replaces a human video editor entirely. Covers hook engineering, pacing, caption styling, color grading, audio mastering, sound design, motion graphics, and multi-platform export.
Read-only social + web listening across public platforms (YouTube, Twitter/X, LinkedIn, Reddit, the open web, GitHub, Exa search) for competitive intelligence, content research, transcript mining, and checking public post results. Wraps Agent Reach via scripts/social_reach_tool.py. Does NOT read Instagram, TikTok, or Facebook.
Build high-converting, education-first advertorial / "infomercial" landing pages for ecommerce products — long-form scrollable HTML that warms cold paid traffic and clicks through to the product page. Covers angle selection, direct-response copy doctrine, AI image generation with a product reference, a premium anti-slop design system, and a portable build/QA convention. Use for advertorials, pre-sell pages, native/listicle ad landers, VSL pages, or any "read-an-article-then-buy" funnel page.
Automated video editing pipeline for CC's personal brand content. Use whenever CC drops raw footage and needs it edited, captioned, graded, and sent for review. The primary tool is video_editor.py which handles the full 8-step pipeline.