brightdata

name	BrightData
description	4-tier progressive scraping with automatic escalation: Tier 1 WebFetch, Tier 2 curl with Chrome headers, Tier 3 agent-browser (headless JS rendering), Tier 4 Bright Data MCP proxy (CAPTCHA, advanced bot detection, residential proxies). Two workflows: FourTierScrape for single URLs, Crawl for multi-page site mapping (light crawl up to 50 pages, Crawl API for full sites). Starts at Tier 1, escalates only when blocked — Tier 4 has usage costs. Output is markdown. Playwright is banned across LifeOS. USE WHEN Bright Data, scrape URL, web scraping, bot detection, crawl site, CAPTCHA, can't access, site blocking, extract page content, scrape whole site, spider domain, convert URL to markdown, getting blocked. NOT FOR simple public content (use WebFetch directly), social platform scraping with named actors (use Apify), parallel headless automation with persistent auth profiles (use Browser), or real-Chrome bot bypass with logged-in sessions and zero CDP fingerprint (use Interceptor).
effort	medium

Customization

Before executing, check for user customizations at: ~/.claude/LIFEOS/USER/CUSTOMIZATIONS/SKILLS/BrightData/

If this directory exists, load and apply any PREFERENCES.md, configurations, or resources found there. These override default behavior. If the directory does not exist, proceed with skill defaults.

🚨 MANDATORY: Voice Notification (REQUIRED BEFORE ANY ACTION)

You MUST send this notification BEFORE doing anything else when this skill is invoked.

1. Send voice notification:

   curl -s -X POST http://localhost:31337/notify \
     -H "Content-Type: application/json" \
     -d '{"message": "Running the WORKFLOWNAME workflow in the BrightData skill to ACTION"}' \
     > /dev/null 2>&1 &
   

2. Output text notification:

   Running the **WorkflowName** workflow in the **BrightData** skill to ACTION...
   

This is not optional. Execute this curl command immediately upon skill invocation.

BrightData

What It Does

Scrapes a URL or crawls a whole site, escalating through four tiers only as far as it needs to. Tier 1 is WebFetch, Tier 2 is curl with Chrome headers, Tier 3 is the agent-browser headless daemon for JavaScript-heavy pages, and Tier 4 is the Bright Data MCP proxy for CAPTCHA, advanced bot detection, and residential proxies. Two workflows: FourTierScrape for a single URL, Crawl for multi-page site mapping. Output is always markdown.

The Problem

A lot of pages won't give up their content to a simple fetch — some need JavaScript to render, some check headers, some throw a CAPTCHA at anything that looks automated. Reaching straight for the heavy proxy every time wastes money, since Tier 4 has usage costs and most pages don't need it. But guessing which tier a page needs is its own time sink. Starting cheap and escalating only when blocked gets the content with the least cost and the least latency.

How It Works

Progressive escalation, always starting at Tier 1 and stepping up only on failure:

1. Tier 1: WebFetch — fast, built-in.
2. Tier 2: curl with Chrome headers — bypasses basic user-agent bot detection.
3. Tier 3: agent-browser — headless browser via the agent-browser Rust CLI daemon for JavaScript rendering. Playwright is banned across LifeOS.
4. Tier 4: Bright Data MCP — proxy service that handles CAPTCHA and advanced bot detection.

Content is preserved in markdown at every tier. The Crawl workflow extends this to multiple pages — a light crawl loops the MCP batch scraper plus link extraction up to 50 pages, a full crawl uses the Bright Data Crawl API for entire sites.

Workflow Routing

When executing a workflow, output this notification directly:

Running the **WorkflowName** workflow in the **BrightData** skill to ACTION...

Workflow	Trigger	File
FourTierScrape	"scrape this URL", "fetch this page", "get content from [URL]", "pull content from this site", "retrieve [URL]", "can't access this site", "this site is blocking me", "use Bright Data to fetch"	Workflows/FourTierScrape.md
Crawl	"crawl this site", "crawl all pages under /docs", "spider this domain", "map this website", "get all pages from", "crawl [URL]", "scrape the whole site", "extract all pages"	Workflows/Crawl.md

FourTierScrape executes the progressive escalation (WebFetch → Curl → agent-browser → Bright Data MCP). Crawl runs Light Crawl for <50 pages or Full Crawl via the Bright Data Crawl API for larger sites.

---

When to Activate This Skill

Direct Scraping Requests (Categories 1-4)

• "scrape this URL", "scrape [URL]", "scrape this page"
• "fetch this URL", "fetch [URL]", "fetch this page", "fetch content from"
• "pull content from [URL]", "pull this page", "pull from this site"
• "get content from [URL]", "retrieve [URL]", "retrieve this page"
• "do scraping on [URL]", "run scraper on [URL]"
• "basic scrape", "quick scrape", "simple fetch"
• "comprehensive scrape", "deep scrape", "full content extraction"

Access & Bot Detection Issues (Categories 5-7)

• "can't access this site", "site is blocking me", "getting blocked"
• "bot detection", "CAPTCHA", "access denied", "403 error"
• "need to bypass bot detection", "get around blocking"
• "this URL won't load", "can't fetch this page"
• "use Bright Data", "use the scraper", "use advanced scraping"

Result-Oriented Requests (Category 8)

• "get me the content from [URL]"
• "extract text from [URL]"
• "download this page content"
• "convert [URL] to markdown"
• "need the HTML from this site"

Crawling Requests (Categories 9-11)

• "crawl this site", "crawl [URL]", "spider this domain"
• "map this website", "get all pages from [URL]", "scrape the whole site"
• "crawl all pages under /docs", "extract all pages from", "site crawl"
• "get every page on this site", "full site extraction"
• "crawl depth 3", "crawl up to 50 pages"

Use Case Indicators

• User needs web content for research or analysis
• Standard methods (WebFetch) are failing
• Site has bot detection or rate limiting
• Need reliable content extraction
• Converting web pages to structured format (markdown)
• User needs multiple pages from a site, not just one
• User wants to map a site's structure or extract a section

---

Core Capabilities

Progressive Escalation Strategy:

1. Tier 1: WebFetch - Fast, simple, built-in Claude Code tool
2. Tier 2: Customized Curl - Chrome-like browser headers to bypass basic bot detection
3. Tier 3: agent-browser - Headless browser automation via agent-browser Rust CLI daemon for JavaScript-heavy sites. Playwright is banned across LifeOS.
4. Tier 4: Bright Data MCP - Professional scraping service that handles CAPTCHA and advanced bot detection

Key Features:

• Automatic fallback between tiers
• Preserves content in markdown format
• Handles bot detection and CAPTCHA
• Works with any URL
• Efficient resource usage (only escalates when needed)

---

Workflow Overview

FourTierScrape.md - Complete URL content scraping with four-tier fallback strategy

• When to use: Any single URL content retrieval request
• Process: Start with WebFetch → If fails, use curl with Chrome headers → If fails, use Browser Automation → If fails, use Bright Data MCP
• Output: URL content in markdown format

Crawl.md - Multi-page crawling with link discovery and site mapping

• When to use: Crawling multiple pages from a site, mapping site structure, extracting a section
• Process: Light Crawl (MCP scrape_batch + link extraction loop, up to 50 pages) or Full Crawl (Bright Data Crawl API for entire sites)
• Output: Site map + page contents in markdown, with crawl stats and cost summary

---

Extended Context

Integration Points:

• WebFetch Tool - Built-in Claude Code tool for basic URL fetching
• Bash Tool - For executing curl commands with custom headers
• Browser Automation - agent-browser headless daemon for JavaScript rendering
• Bright Data MCP - mcp__Brightdata__scrape_as_markdown and scrape_batch for advanced scraping
• Bright Data Crawl API - HTTP POST to api.brightdata.com/datasets/v3/trigger for full-site crawls

When Each Tier Is Used:

• Tier 1 (WebFetch): Simple sites, public content, no bot detection
• Tier 2 (Curl): Sites with basic user-agent checking, simple bot detection
• Tier 3 (agent-browser): Sites requiring JavaScript execution, dynamic content loading
• Tier 4 (Bright Data): Sites with CAPTCHA, advanced bot detection, residential proxy requirements

Configuration: No configuration required - all tools are available by default in Claude Code

---

Examples

Example 1: Simple Public Website

User: "Scrape https://example.com"

Skill Response:

1. Routes to Workflows/FourTierScrape.md
2. Attempts Tier 1 (WebFetch)
3. Success → Returns content in markdown
4. Total time: <5 seconds

Example 2: Site with JavaScript Requirements

User: "Can't access this site https://dynamic-site.com"

Skill Response:

1. Routes to Workflows/FourTierScrape.md
2. Attempts Tier 1 (WebFetch) → Fails (blocked)
3. Attempts Tier 2 (Curl with Chrome headers) → Fails (JavaScript required)
4. Attempts Tier 3 (agent-browser) → Success
5. Returns content in markdown
6. Total time: ~15-20 seconds

Example 3: Site with Advanced Bot Detection

User: "Scrape https://protected-site.com"

Skill Response:

1. Routes to Workflows/FourTierScrape.md
2. Attempts Tier 1 (WebFetch) → Fails (blocked)
3. Attempts Tier 2 (Curl) → Fails (advanced detection)
4. Attempts Tier 3 (agent-browser) → Fails (CAPTCHA)
5. Attempts Tier 4 (Bright Data MCP) → Success
6. Returns content in markdown
7. Total time: ~30-40 seconds

Example 4: Explicit Bright Data Request

User: "Use Bright Data to fetch https://difficult-site.com"

Skill Response:

1. Routes to Workflows/FourTierScrape.md
2. User explicitly requested Bright Data
3. Goes directly to Tier 4 (Bright Data MCP) → Success
4. Returns content in markdown
5. Total time: ~5-10 seconds

---

Related Documentation:

• ~/.claude/LIFEOS/DOCUMENTATION/Skills/SkillSystem.md - Canonical structure guide
• ~/.claude/ - Overall LifeOS philosophy

Last Updated: 2026-02-22

Gotchas

• 4-tier escalation: WebFetch → curl → agent-browser → Bright Data proxy. Always start at Tier 1 and escalate only when blocked. Playwright is banned across LifeOS.
• Bright Data proxy has usage costs. Don't use Tier 4 for sites accessible via Tier 1-3.
• CAPTCHA-solving introduces latency. Allow extra time for Tier 4 responses.
• Credentials in ~/.claude/.env — BRIGHTDATA_API_KEY.

Execution Log

After completing any workflow, append a single JSONL entry:

echo '{"ts":"'$(date -u +%Y-%m-%dT%H:%M:%SZ)'","skill":"BrightData","workflow":"WORKFLOW_USED","input":"8_WORD_SUMMARY","status":"ok|error","duration_s":SECONDS}' >> ~/.claude/LIFEOS/MEMORY/SKILLS/execution.jsonl

Replace WORKFLOW_USED with the workflow executed, 8_WORD_SUMMARY with a brief input description, and SECONDS with approximate wall-clock time. Log status: "error" if the workflow failed.