| name | hasdata |
| description | Use HasData to scrape any public web page, run real-time Google/Bing/Google-AI-Mode search queries, pull structured data from e-commerce, real-estate, lodging, jobs, maps, travel, video, and social platforms, or run async bulk-scraping and crawling jobs without managing proxies, browsers, or captchas. Reach for this skill when the user mentions web scraping, SERP/search results, Google Maps/Trends/Flights/Images, YouTube videos/transcripts/channels, Amazon, Zillow, Redfin, Airbnb, Booking.com hotels, Yelp, Indeed, Glassdoor, Instagram, Shopify, scraper jobs, website crawling, RAG/LLM data ingestion, lead/contact enrichment, or HasData itself. |
HasData
Cloud platform for extracting public web data. One API key, three execution modes. All endpoints sit under https://api.hasdata.com and authenticate with x-api-key.
curl -G 'https://api.hasdata.com/scrape/google/serp' \
--data-urlencode 'q=coffee' \
-H 'x-api-key: <your-api-key>'
401 invalid key, 403 quota exhausted, 429 concurrency cap, 500 server error (retry).
Three execution modes
| Mode | Latency | When | Endpoint |
|---|
| Web Scraping API | seconds | Arbitrary URL — JS rendering, CSS/AI extraction, screenshots | POST /scrape/web |
| Scraper APIs (sync) | seconds | Pre-parsed JSON for known platforms (Google, Amazon, Zillow, …) | GET /scrape/<vertical>/<resource> |
| Scraper Jobs (async) | minutes–hours | Bulk extraction, recursive crawling, webhook fan-out | POST /scrapers/<slug>/jobs |
Decision rule. Default to a Scraper API when one exists for the platform (pre-parsed JSON, no selector maintenance). Use Web Scraping for arbitrary URLs not covered by an API. Reach for a Scraper Job only when no API equivalent exists — crawler, contacts, sec-edgar, amazon-bestsellers, amazon-product-reviews — or when async fan-out + webhooks save engineering time over a paginated client loop.
Always-true response shape
{ "requestMetadata": { "id": "…", "status": "ok", "url": "…" }, "...": "endpoint-specific" }
Treat data as valid only if requestMetadata.status === "ok". HTTP 200 alone isn't enough.
High-leverage patterns
- SERP-first enrichment. Google SERP is a free-form data lake.
q="<Person> <Company> linkedin" → organicResults[0].title + .snippet already carries role + location, no profile-page scrape needed. Same recipe with crunchbase, wikipedia, github, or quoted literals ("jane@x.com", "+1 555 …") for reverse lookup.
- AI Mode + verify.
/scrape/google/ai-mode for the answer + references → /scrape/web (markdown) on each reference URL → cited RAG context, no vector DB.
- Maps → leads.
/scrape/google-maps/search returns websites + phones; fan out to /scrape/web with extractEmails: true for full lead rows.
- Crawler → corpus.
crawler Scraper Job with outputFormat: ["markdown"] + includePaths: "/docs/.+" produces an LLM-ready corpus in one submission.
- Pre-extracted via SERP rich snippets.
knowledgeGraph, localResults, inlineShoppingResults, relatedQuestions carry pre-parsed facts that bypass anti-bot. Always check them before scraping.
When to call from code (the wiring)
- Auth:
x-api-key header on every request. Read from HASDATA_API_KEY env. Never hardcode, never log.
- Timeouts: set client timeout ≥ 300 s. HasData's own deadline is 300 s; shorter clients produce phantom failures while still being billed on completion.
- Retries:
429 and 5xx only — exponential backoff, jitter. Never retry 4xx (auth, validation).
- Concurrency: cap at your plan limit. The free tier is 1; anything higher just generates
429s.
- Async jobs: the submit response handle is
body.id (integer), not jobId. Persist it immediately. Poll GET /scrapers/jobs/<id> every 10–30 s with backoff; treat webhooks as best-effort and always pair with polling. On finished the status carries data: {csv, json, xlsx} short-lived URLs — download immediately.
See references/code-recipes.md for ready-to-paste Python and TypeScript clients with retry, backoff, bounded concurrency, and the full job lifecycle.
Common gotchas
- 300 s server deadline. Match client timeout.
- Disable
jsRendering first, enable only if the page needs it — most static pages parse fine without a headless browser.
- No
cookies parameter — cookies go through headers["Cookie"].
includePaths regex is case-sensitive. /blog/.+ won't match /Blog/....- Scraper Job
data is double-wrapped. Each row is body.data[i].data; outer wraps with id, jobId, dataId, createdAt, updatedAt.
requestMetadata.status === "ok" is the only success signal. HTTP 200 alone isn't enough.- Webhooks are best-effort with 3 retries. Always have a polling fallback.
References
references/web-scraping.md — POST /scrape/web parameters, JS scenarios, AI extraction, cookie auth.references/search.md — Google SERP / Light / AI Mode / News / Shopping / Bing / Trends + pagination.references/ecommerce.md — Amazon (product, search, seller, seller-products) and Shopify.references/real-estate.md — Zillow, Redfin (bracketed filters).references/travel.md — Airbnb, Booking, Google Flights (occupancy rules, token pagination, IATA codes).references/local-business.md — Maps (search/place/reviews/photos/posts), Yelp, YellowPages.references/jobs.md — Indeed and Glassdoor.references/youtube.md — YouTube search / video / channel / transcript.references/scraper-jobs.md — async submit/poll/results, Crawler, Contacts, SEC EDGAR, webhook receiver.references/code-recipes.md — Python / TypeScript clients with retry, backoff, concurrency, polling.
Resources
