| name | brightdata-reference-architecture |
| description | Implement Bright Data reference architecture with best-practice project layout.
Use when designing new Bright Data integrations, reviewing project structure,
or establishing architecture standards for Bright Data applications.
Trigger with phrases like "brightdata architecture", "brightdata best practices",
"brightdata project structure", "how to organize brightdata", "brightdata layout".
|
| allowed-tools | Read, Grep |
| version | 1.6.0 |
| license | MIT |
| author | Jeremy Longshore <jeremy@intentsolutions.io> |
| tags | ["saas","scraping","data","brightdata"] |
| compatibility | Designed for Claude Code |
Bright Data Reference Architecture
Overview
Production-ready architecture for Bright Data scraping systems. Covers project layout, data pipeline design, and integration patterns for Web Unlocker, Scraping Browser, SERP API, and Datasets API.
Prerequisites
- Understanding of layered architecture
- Node.js/TypeScript project setup
- Database for storing scraped data
Project Structure
my-scraper/
├── src/
│ ├── brightdata/
│ │ ├── proxy.ts # Proxy config helper (zone, country, session)
│ │ ├── client.ts # Axios client with proxy + retry
│ │ ├── browser.ts # Scraping Browser connection manager
│ │ ├── api.ts # REST API client (trigger, snapshot)
│ │ ├── cache.ts # Response cache (LRU + optional Redis)
│ │ └── types.ts # Shared TypeScript interfaces
│ ├── scrapers/
│ │ ├── product-scraper.ts # Domain-specific scraper
│ │ ├── serp-scraper.ts # Search result collector
│ │ └── parser.ts # HTML → structured data (cheerio)
│ ├── pipeline/
│ │ ├── scheduler.ts # Cron-based scraping scheduler
│ │ ├── processor.ts # Raw HTML → clean data
│ │ └── storage.ts # Database/file output
│ ├── webhooks/
│ │ └── brightdata.ts # Webhook delivery handler
│ └── api/
│ ├── health.ts # Health check endpoint
│ └── scrape.ts # On-demand scrape endpoint
├── tests/
│ ├── unit/ # Mocked tests (no proxy needed)
│ ├── integration/ # Live proxy tests
│ └── fixtures/ # Cached HTML for testing
├── config/
│ ├── zones.json # Zone configuration per environment
│ └── targets.json # Target URLs and scraping schedules
└── .env.example
Architecture Diagram
┌──────────────────────────────────────────────────────┐
│ API / Scheduler │
│ (On-demand scrape, cron jobs, webhooks) │
├──────────────────────────────────────────────────────┤
│ Scraper Layer │
│ (Product scraper, SERP scraper, custom parsers) │
├────────────┬─────────────────┬───────────────────────┤
│ Web │ Scraping │ SERP / Datasets │
│ Unlocker │ Browser │ API │
│ (Proxy) │ (WebSocket) │ (REST) │
├────────────┴─────────────────┴───────────────────────┤
│ Bright Data Infrastructure Layer │
│ (Proxy config, retry, cache, session management) │
├──────────────────────────────────────────────────────┤
│ Storage / Pipeline │
│ (Database, file output, webhook delivery) │
└──────────────────────────────────────────────────────┘
Key Components
Step 1: Multi-Product Client
