Bright Data Performance Tuning
Overview
Optimize Bright Data scraping performance through connection pooling, response caching, concurrent request tuning, and smart product selection. Web Unlocker latency is typically 5-30s due to CAPTCHA solving; Scraping Browser sessions are 10-60s.
Prerequisites
- Bright Data zone configured
- Understanding of async patterns
- Redis or file cache available (optional)
Latency Benchmarks
| Product | P50 | P95 | P99 | Notes |
|---|
| Web Unlocker (simple) | 3s | 8s | 15s | No CAPTCHA |
| Web Unlocker (CAPTCHA) | 10s | 25s | 45s | With CAPTCHA solving |
| Scraping Browser | 8s | 20s | 40s | Full browser render |
| SERP API (sync) | 2s | 5s | 10s | Search results |
| Residential Proxy | 1s | 3s | 8s | Raw proxy, no unblocking |
Instructions
Step 1: Choose the Right Product
function selectProduct(target: { js: boolean; captcha: boolean; structured: boolean }) {
if (target.structured) return 'serp_api';
if (!target.js && !target.captcha) return 'residential';
if (target.js) return 'scraping_browser';
return 'web_unlocker';
}
Step 2: Connection Pooling with Keep-Alive
import { Agent } from 'https';
import axios from 'axios';
const httpsAgent = new Agent({
keepAlive: true,
maxSockets: 25,
maxFreeSockets: 5,
timeout: 120000,
rejectUnauthorized: false,
});
const client = axios.create({
proxy: { host: 'brd.superproxy.io', port: 33335, auth: { username: proxyUser, password: proxyPass } },
httpsAgent,
timeout: 60000,
});
Step 3: Response Caching Layer
import { createHash } from 'crypto';
import { LRUCache } from 'lru-cache';
const memoryCache = new LRUCache<string, string>({
max: 500,
maxSize: 100_000_000,
sizeCalculation: (v) => Buffer.byteLength(v),
ttl: 3600000,
});
export async function cachedScrape(
url: string,
scraper: (url: string) => Promise<string>,
ttlMs?: number
): Promise<string> {
const key = createHash('sha256').update(url).digest('hex');
const cached = memoryCache.get(key);
if (cached) {
console.();
cached;
}
html = (url);
memoryCache.(key, html, { : ttlMs });
.();
html;
}
Step 4: Concurrent Scraping with Backpressure
import PQueue from 'p-queue';
const scrapeQueue = new PQueue({
concurrency: 10,
interval: 1000,
intervalCap: 15,
});
async function scrapeMany(urls: string[]): Promise<Map<string, string>> {
const results = new Map<string, string>();
await Promise.allSettled(
urls.map(url =>
scrapeQueue.add(async () => {
const html = await cachedScrape(url, (u) => client.get(u).then(r => r.data));
results.set(url, html);
})
)
);
.();
results;
}
Step 5: Use Async API for Bulk Jobs
For 100+ URLs, use the Web Scraper API instead of individual proxy requests:
async function bulkScrape(urls: string[]) {
const response = await fetch(
`https://api.brightdata.com/datasets/v3/trigger?dataset_id=${DATASET_ID}&format=json`,
{
method: 'POST',
headers: {
'Authorization': `Bearer ${process.env.BRIGHTDATA_API_TOKEN}`,
'Content-Type': 'application/json',
},
body: JSON.stringify(urls.map(url => ({ url }))),
}
);
return response.json();
}
Step 6: Performance Monitoring
class ScrapeMetrics {
private timings: number[] = [];
private errors = 0;
private cacheHits = 0;
record(durationMs: number) { this.timings.push(durationMs); }
recordError() { this.errors++; }
recordCacheHit() { this.cacheHits++; }
report() {
const sorted = [...this.timings].sort((a, b) => a - b);
return {
count: sorted.length,
errors: this.errors,
cacheHits: this.cacheHits,
p50: sorted[Math.floor(sorted.length * 0.5)] || 0,
p95: sorted[Math.floor(sorted.length * 0.95)] || 0,
: sorted[.(sorted. * )] || ,
};
}
}
Output
- Right product selection per use case
- Connection pooling reducing TCP overhead
- Response cache avoiding duplicate scrapes
- Concurrent scraping with backpressure control
- Bulk API for large-scale jobs
Error Handling
| Issue | Cause | Solution |
|---|
| Slow scrapes | CAPTCHA solving overhead | Expected for Web Unlocker; use cache |
| Connection exhausted | Too many concurrent | Reduce p-queue concurrency |
| Memory pressure | Large cached pages | Set maxSize on LRU cache |
| Timeout storms | All requests hitting slow site | Add circuit breaker |
Resources
Next Steps
For cost optimization, see brightdata-cost-tuning.
