Bright Data Reference Architecture
Overview
Production-ready architecture for Bright Data scraping systems. Covers project layout, data pipeline design, and integration patterns for Web Unlocker, Scraping Browser, SERP API, and Datasets API.
Prerequisites
- Understanding of layered architecture
- Node.js/TypeScript project setup
- Database for storing scraped data
Project Structure
my-scraper/
โโโ src/
โ โโโ brightdata/
โ โ โโโ proxy.ts # Proxy config helper (zone, country, session)
โ โ โโโ client.ts # Axios client with proxy + retry
โ โ โโโ browser.ts # Scraping Browser connection manager
โ โ โโโ api.ts # REST API client (trigger, snapshot)
โ โ โโโ cache.ts # Response cache (LRU + optional Redis)
โ โ โโโ types.ts # Shared TypeScript interfaces
โ โโโ scrapers/
โ โ โโโ product-scraper.ts # Domain-specific scraper
โ โ โโโ serp-scraper.ts # Search result collector
โ โ โโโ parser.ts # HTML โ structured data (cheerio)
โ โโโ pipeline/
โ โ โโโ scheduler.ts # Cron-based scraping scheduler
โ โ โโโ processor.ts # Raw HTML โ clean data
โ โ โโโ storage.ts # Database/file output
โ โโโ webhooks/
โ โ โโโ brightdata.ts # Webhook delivery handler
โ โโโ api/
โ โโโ health.ts # Health check endpoint
โ โโโ scrape.ts # On-demand scrape endpoint
โโโ tests/
โ โโโ unit/ # Mocked tests (no proxy needed)
โ โโโ integration/ # Live proxy tests
โ โโโ fixtures/ # Cached HTML for testing
โโโ config/
โ โโโ zones.json # Zone configuration per environment
โ โโโ targets.json # Target URLs and scraping schedules
โโโ .env.example
Architecture Diagram
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ API / Scheduler โ
โ (On-demand scrape, cron jobs, webhooks) โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโค
โ Scraper Layer โ
โ (Product scraper, SERP scraper, custom parsers) โ
โโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโโโค
โ Web โ Scraping โ SERP / Datasets โ
โ Unlocker โ Browser โ API โ
โ (Proxy) โ (WebSocket) โ (REST) โ
โโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโค
โ Bright Data Infrastructure Layer โ
โ (Proxy config, retry, cache, session management) โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโค
โ Storage / Pipeline โ
โ (Database, file output, webhook delivery) โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
Key Components
Step 1: Multi-Product Client
import axios, { AxiosInstance } from 'axios';
import https from 'https';
import { chromium } from 'playwright';
export class BrightDataClient {
private proxyClient: AxiosInstance;
private apiToken: string;
constructor(private config: {
customerId: string;
zone: string;
zonePassword: string;
apiToken: string;
}) {
this.apiToken = config.apiToken;
this.proxyClient = axios.create({
proxy: {
host: 'brd.superproxy.io',
port: 33335,
auth: {
username: `brd-customer-${config.customerId}-zone-${config.zone}`,
password: config.zonePassword,
},
},
httpsAgent: new https.Agent({ keepAlive: true, : }),
: ,
});
}
(: , ?: ): <> {
response = ..(url);
response.;
}
() {
auth = ;
browser = chromium.();
{
page = browser.();
page.(url, { : , : });
(page);
} {
browser.();
}
}
() {
response = (
,
{
: ,
: { : , : },
: .(inputs),
}
);
response.();
}
}
Step 2: Scraping Pipeline
import cron from 'node-cron';
interface ScrapeJob {
name: string;
urls: string[];
product: 'web_unlocker' | 'scraping_browser' | 'datasets_api';
schedule: string;
parser: (html: string) => any;
}
export function startScheduler(jobs: ScrapeJob[], client: BrightDataClient) {
for (const job of jobs) {
cron.schedule(job.schedule, async () => {
console.log(`Running job: ${job.name}`);
if (job.product === 'datasets_api') {
await client.triggerCollection('dataset_id', job.urls.map(url => ({ url })));
} else {
for ( url job.) {
html = client.(url);
data = job.(html);
(job., data);
}
}
});
}
}
Step 3: Environment Configuration
{
"development": {
"web_unlocker": "web_unlocker_dev",
"scraping_browser": "scraping_browser_dev",
"api_datasets": true
},
"production": {
"web_unlocker": "web_unlocker_prod",
"scraping_browser": "scraping_browser_prod",
"api_datasets": true
}
}
Decision Matrix
| Scenario | Product | Why |
|---|
| Simple HTML pages | Web Unlocker | Cheapest, fastest |
| JavaScript SPA | Scraping Browser | Needs browser rendering |
| Search results | SERP API | Pre-parsed JSON output |
| 1000+ URLs one-time | Web Scraper API | Async, handles parallelism |
| Amazon/LinkedIn/etc. | Pre-built Datasets | No code needed |
| Login-required pages | Scraping Browser + sticky session | Session persistence |
Output
- Multi-product Bright Data client
- Domain-specific scrapers with parsers
- Cron-based scraping pipeline
- Environment-isolated zone configuration
Error Handling
| Issue | Cause | Solution |
|---|
| Mixed product confusion | Wrong zone for task | Use decision matrix above |
| Circular dependencies | Tight coupling | Keep scraper layer separate from proxy layer |
| Test pollution | Shared mocks | Use dependency injection |
| Config mismatch | Wrong environment | Load zone config from zones.json |
Resources
Next Steps
For multi-environment setup, see brightdata-deploy-integration.
