Skip to main content
brightdata-performance-tuning Optimize Bright Data API performance with caching, batching, and connection pooling.
Use when experiencing slow API responses, implementing caching strategies,
or optimizing request throughput for Bright Data integrations.
Trigger with phrases like "brightdata performance", "optimize brightdata",
"brightdata latency", "brightdata caching", "brightdata slow", "brightdata batch".
Aller à l'installation Skills Marketplace Découvrez et explorez les compétences IA créées par la communauté.
Installer avec Codex ou Claude Copiez ce prompt, collez-le dans Codex, Claude ou un autre assistant, puis laissez-le vérifier la page du skill et l'installer pour vous.
Afficher les détails du prompt Une commande directe contourne le prompt de vérification. Examinez la source avant de l'exécuter.
npx skills add https://github.com/jeremylongshore/claude-code-plugins-plus-skills --skill brightdata-performance-tuningLa commande reste sur une seule ligne. Faites défiler horizontalement pour la vérifier avant de la copier.
Vous préférez une copie locale ? Téléchargez les fichiers actuellement disponibles dans SkillsMP.
Télécharger Zip Téléchargement... Plus depuis ce dépôt Implement user sign-up and sign-in flows with Clerk.
Use when building authentication UI, customizing sign-in experience,
or implementing OAuth social login.
Trigger with phrases like "clerk sign-in", "clerk sign-up",
"clerk login flow", "clerk OAuth", "clerk social login".
Implement session management and middleware with Clerk.
Use when managing user sessions, configuring route protection,
or implementing token refresh and custom JWT templates.
Trigger with phrases like "clerk session", "clerk middleware",
"clerk route protection", "clerk token", "clerk JWT".
Configure enterprise SSO, role-based access control, and organization management.
Use when implementing SSO integration, configuring role-based permissions,
or setting up organization-level controls.
Trigger with phrases like "clerk SSO", "clerk RBAC",
"clerk enterprise", "clerk roles", "clerk permissions", "clerk organizations".
Métiers associés SOC
Basé sur la classification professionnelle SOC
name brightdata-performance-tuning description Optimize Bright Data API performance with caching, batching, and connection pooling.
Use when experiencing slow API responses, implementing caching strategies,
or optimizing request throughput for Bright Data integrations.
Trigger with phrases like "brightdata performance", "optimize brightdata",
"brightdata latency", "brightdata caching", "brightdata slow", "brightdata batch".
allowed-tools Read, Write, Edit version 1.6.0 license MIT author Jeremy Longshore <jeremy@intentsolutions.io> tags ["saas","scraping","data","brightdata"] compatibility Designed for Claude Code
Bright Data Performance Tuning
Overview
Optimize Bright Data scraping performance through connection pooling, response caching, concurrent request tuning, and smart product selection. Web Unlocker latency is typically 5-30s due to CAPTCHA solving; Scraping Browser sessions are 10-60s.
Prerequisites
Bright Data zone configured
Understanding of async patterns
Redis or file cache available (optional)
Latency Benchmarks
Product P50 P95 P99 Notes Web Unlocker (simple) 3s 8s 15s No CAPTCHA Web Unlocker (CAPTCHA) 10s 25s 45s With CAPTCHA solving Scraping Browser 8s 20s 40s Full browser render SERP API (sync) 2s 5s 10s Search results Residential Proxy 1s 3s 8s Raw proxy, no unblocking
Instructions
Step 1: Choose the Right Product
function selectProduct (target : { js: boolean ; captcha: boolean ; structured: boolean }if (target.structured ) return 'serp_api' ;
if (!target.js && !target.captcha ) return 'residential' ;
if (target.js ) return 'scraping_browser' ;
return
'web_unlocker'
Step 2: Connection Pooling with Keep-Alive import { Agent } from 'https' ;
import axios from 'axios' ;
const httpsAgent = new Agent ({
keepAlive : true ,
maxSockets : 25 ,
maxFreeSockets : 5 ,
timeout : 120000 ,
rejectUnauthorized : false ,
});
const client = axios.create ({
proxy : { host : 'brd.superproxy.io' , port : 33335 , auth : { username : proxyUser, password : proxyPass } },
httpsAgent,
timeout : 60000 ,
});
Step 3: Response Caching Layer
import { createHash } from 'crypto' ;
import { LRUCache } from 'lru-cache' ;
const memoryCache = new LRUCache <string , string >({
max : 500 ,
maxSize : 100_000_000 ,
sizeCalculation : (v ) => Buffer .byteLength (v),
ttl : 3600000 ,
});
export async function cachedScrape (
url : string ,
scraper : (url: string ) => Promise <string >,
ttlMs ?: number
): Promise <string > {
const key = createHash ('sha256' ).update (url).digest ('hex' );
const cached = memoryCache.get (key);
if (cached) {
console .log (`Cache HIT: ${url} ` );
return cached;
}
const html = await scraper (url);
memoryCache.set (key, html, { ttl : ttlMs });
console .log (`Cache MISS: ${url} (${Buffer.byteLength(html)} bytes)` );
return html;
}
Step 4: Concurrent Scraping with Backpressure import PQueue from 'p-queue' ;
const scrapeQueue = new PQueue ({
concurrency : 10 ,
interval : 1000 ,
intervalCap : 15 ,
});
async function scrapeMany (urls : string []Promise <Map <string , string >> {
const results = new Map <string , string >();
await Promise .allSettled (
urls.map (url =>add (async () => {
const html = await cachedScrape (url, (u ) => client.get (u).then (r =>data ));
results.set (url, html);
})
)
);
console .log (`Scraped ${results.size} /${urls.length} successfully` );
return results;
}
Step 5: Use Async API for Bulk Jobs For 100+ URLs, use the Web Scraper API instead of individual proxy requests:
async function bulkScrape (urls : string []const response = await fetch (
`https://api.brightdata.com/datasets/v3/trigger?dataset_id=${DATASET_ID} &format=json` ,
{
method : 'POST' ,
headers : {
'Authorization' : `Bearer ${process.env.BRIGHTDATA_API_TOKEN} ` ,
'Content-Type' : 'application/json' ,
},
body : JSON .stringify (urls.map (url =>return response.json ();
}
Step 6: Performance Monitoring class ScrapeMetrics {
private timings : number [] = [];
private errors = 0 ;
private cacheHits = 0 ;
record (durationMs : number this .timings .push (durationMs); }
recordError (this .errors ++; }
recordCacheHit (this .cacheHits ++; }
report (const sorted = [...this .timings ].sort ((a, b ) => a - b);
return {
count : sorted.length ,
errors : this .errors ,
cacheHits : this .cacheHits ,
p50 : sorted[Math .floor (sorted.length * 0.5 )] || 0 ,
p95 : sorted[Math .floor (sorted.length * 0.95 )] || 0 ,
p99 : sorted[Math .floor (sorted.length * 0.99 )] || 0 ,
};
}
}
Output
Right product selection per use case
Connection pooling reducing TCP overhead
Response cache avoiding duplicate scrapes
Concurrent scraping with backpressure control
Bulk API for large-scale jobs
Error Handling Issue Cause Solution Slow scrapes CAPTCHA solving overhead Expected for Web Unlocker; use cache Connection exhausted Too many concurrent Reduce p-queue concurrency Memory pressure Large cached pages Set maxSize on LRU cache Timeout storms All requests hitting slow site Add circuit breaker
Resources
Next Steps For cost optimization, see brightdata-cost-tuning.
