Skip to main content 홈 크리에이터 jeremylongshore tons-of-skills-marketplace brightdata-performance-tuning
brightdata-performance-tuning Optimize Bright Data API performance with caching, batching, and connection pooling.
Use when experiencing slow API responses, implementing caching strategies,
or optimizing request throughput for Bright Data integrations.
Trigger with phrases like "brightdata performance", "optimize brightdata",
"brightdata latency", "brightdata caching", "brightdata slow", "brightdata batch".
설치로 이동 Skills Marketplace 커뮤니티가 만든 AI 스킬을 발견하고 탐색하세요.
Codex 또는 Claude로 설치 이 Prompt를 복사해 Codex, Claude 또는 다른 어시스턴트에 붙여 넣으면 Skill 페이지를 검토하고 설치를 진행할 수 있습니다.
직접 명령은 검토 Prompt를 거치지 않습니다. 실행하기 전에 소스를 확인하세요.
npx skills add https://github.com/jeremylongshore/tons-of-skills-marketplace --skill brightdata-performance-tuning명령은 한 줄로 유지됩니다. 복사하기 전에 가로로 스크롤해 전체 내용을 확인하세요.
로컬 사본을 원하시나요? SkillsMP에서 현재 제공할 수 있는 파일을 다운로드하세요.
Zip 다운로드 다운로드 중... 이 저장소의 다른 Skills langchain-deploy-integration Deploy a LangChain 1.0 / LangGraph 1.0 app to Cloud Run, Vercel, or LangServe correctly — with timeouts sized for chain length, cold-start mitigation, SSE anti-buffering headers, and Secret Manager over .env. Use when prepping a first production deploy, debugging a stream that hangs behind a proxy, or diagnosing p99 latency spikes. Trigger with "langchain deploy", "langchain cloud run", "langchain vercel python", "langchain langserve", or "langchain docker".
langchain-langgraph-agents Build a correct LangGraph 1.0 ReAct agent with create_react_agent — typed tools, error propagation, recursion caps, and stop conditions that actually stop. Use when writing a first tool-calling agent, migrating from AgentExecutor or initialize_agent, or diagnosing an agent that loops on vague prompts. Trigger with "langgraph agent", "create_react_agent", "langgraph tool calling", "AgentExecutor migration", or "agent loop cost".
langchain-langgraph-human-in-loop Build LangGraph 1.0 human-in-the-loop approval flows with interrupt_before /
interrupt_after and Command(resume=...) — JSON-serializable state, clean
resume semantics, and UI wiring for approval decisions. Use when adding an
approval gate before an expensive tool call, wiring a Slack/web UI for agent
approvals, or debugging a graph that crashes on interrupt.
Trigger with "langgraph human in loop", "langgraph interrupt_before",
"langgraph approval flow", "Command resume", "langgraph HITL".
name brightdata-performance-tuning description Optimize Bright Data API performance with caching, batching, and connection pooling.
Use when experiencing slow API responses, implementing caching strategies,
or optimizing request throughput for Bright Data integrations.
Trigger with phrases like "brightdata performance", "optimize brightdata",
"brightdata latency", "brightdata caching", "brightdata slow", "brightdata batch".
allowed-tools Read, Write, Edit version 1.6.0 license MIT author Jeremy Longshore <jeremy@intentsolutions.io> tags ["saas","scraping","data","brightdata"] compatibility Designed for Claude Code
Bright Data Performance Tuning
Overview
Optimize Bright Data scraping performance through connection pooling, response caching, concurrent request tuning, and smart product selection. Web Unlocker latency is typically 5-30s due to CAPTCHA solving; Scraping Browser sessions are 10-60s.
Prerequisites
Bright Data zone configured
Understanding of async patterns
Redis or file cache available (optional)
Latency Benchmarks
Product P50 P95 P99 Notes Web Unlocker (simple) 3s 8s 15s No CAPTCHA Web Unlocker (CAPTCHA) 10s 25s 45s With CAPTCHA solving Scraping Browser 8s 20s 40s Full browser render SERP API (sync) 2s 5s 10s Search results Residential Proxy 1s 3s 8s Raw proxy, no unblocking
Instructions
Step 1: Choose the Right Product
function selectProduct (target : { js: boolean ; captcha: boolean ; structured: boolean }if (target.structured ) return 'serp_api' ;
if (!target.js && !target.captcha ) return 'residential' ;
if (target.js ) return 'scraping_browser' ;
return
'web_unlocker'
Step 2: Connection Pooling with Keep-Alive import { Agent } from 'https' ;
import axios from 'axios' ;
const httpsAgent = new Agent ({
keepAlive : true ,
maxSockets : 25 ,
maxFreeSockets : 5 ,
timeout : 120000 ,
rejectUnauthorized : false ,
});
const client = axios.create ({
proxy : { host : 'brd.superproxy.io' , port : 33335 , auth : { username : proxyUser, password : proxyPass } },
httpsAgent,
timeout : 60000 ,
});
Step 3: Response Caching Layer
import { createHash } from 'crypto' ;
import { LRUCache } from 'lru-cache' ;
const memoryCache = new LRUCache <string , string >({
max : 500 ,
maxSize : 100_000_000 ,
sizeCalculation : (v ) => Buffer .byteLength (v),
ttl : 3600000 ,
});
export async function cachedScrape (
url : string ,
scraper : (url: string ) => Promise <string >,
ttlMs ?: number
): Promise <string > {
const key = createHash ('sha256' ).update (url).digest ('hex' );
const cached = memoryCache.get (key);
if (cached) {
console .log (`Cache HIT: ${url} ` );
return cached;
}
const html = await scraper (url);
memoryCache.set (key, html, { ttl : ttlMs });
console .log (`Cache MISS: ${url} (${Buffer.byteLength(html)} bytes)` );
return html;
}
Step 4: Concurrent Scraping with Backpressure import PQueue from 'p-queue' ;
const scrapeQueue = new PQueue ({
concurrency : 10 ,
interval : 1000 ,
intervalCap : 15 ,
});
async function scrapeMany (urls : string []Promise <Map <string , string >> {
const results = new Map <string , string >();
await Promise .allSettled (
urls.map (url =>add (async () => {
const html = await cachedScrape (url, (u ) => client.get (u).then (r =>data ));
results.set (url, html);
})
)
);
console .log (`Scraped ${results.size} /${urls.length} successfully` );
return results;
}
Step 5: Use Async API for Bulk Jobs For 100+ URLs, use the Web Scraper API instead of individual proxy requests:
async function bulkScrape (urls : string []const response = await fetch (
`https://api.brightdata.com/datasets/v3/trigger?dataset_id=${DATASET_ID} &format=json` ,
{
method : 'POST' ,
headers : {
'Authorization' : `Bearer ${process.env.BRIGHTDATA_API_TOKEN} ` ,
'Content-Type' : 'application/json' ,
},
body : JSON .stringify (urls.map (url =>return response.json ();
}
Step 6: Performance Monitoring class ScrapeMetrics {
private timings : number [] = [];
private errors = 0 ;
private cacheHits = 0 ;
record (durationMs : number this .timings .push (durationMs); }
recordError (this .errors ++; }
recordCacheHit (this .cacheHits ++; }
report (const sorted = [...this .timings ].sort ((a, b ) => a - b);
return {
count : sorted.length ,
errors : this .errors ,
cacheHits : this .cacheHits ,
p50 : sorted[Math .floor (sorted.length * 0.5 )] || 0 ,
p95 : sorted[Math .floor (sorted.length * 0.95 )] || 0 ,
p99 : sorted[Math .floor (sorted.length * 0.99 )] || 0 ,
};
}
}
Output
Right product selection per use case
Connection pooling reducing TCP overhead
Response cache avoiding duplicate scrapes
Concurrent scraping with backpressure control
Bulk API for large-scale jobs
Error Handling Issue Cause Solution Slow scrapes CAPTCHA solving overhead Expected for Web Unlocker; use cache Connection exhausted Too many concurrent Reduce p-queue concurrency Memory pressure Large cached pages Set maxSize on LRU cache Timeout storms All requests hitting slow site Add circuit breaker
Resources
Next Steps For cost optimization, see brightdata-cost-tuning.
