Calls HasData (api.hasdata.com, x-api-key) for Google/Bing SERP JSON, vertical scrapers (Amazon, Zillow, Maps), POST /scrape/web on arbitrary public URLs, and async Scraper Jobs with poll plus webhook. Use when the request is HasData, public SERP, or bulk public crawl. Not for authenticated first-party product APIs, CAPTCHA bypass, or Playwright/CDP sessions that never hit HasData.
Instrucciones de origen · Vista previa de solo lectura
name
hasdata
description
Calls HasData (api.hasdata.com, x-api-key) for Google/Bing SERP JSON, vertical scrapers (Amazon, Zillow, Maps), POST /scrape/web on arbitrary public URLs, and async Scraper Jobs with poll plus webhook. Use when the request is HasData, public SERP, or bulk public crawl. Not for authenticated first-party product APIs, CAPTCHA bypass, or Playwright/CDP sessions that never hit HasData.
Cloud platform for extracting public web data. One API key, three execution modes. All endpoints sit under https://api.hasdata.com and authenticate with the x-api-key header.
When to Use
Use this skill when:
The user needs web scraping of arbitrary URLs.
The user needs search engine results (Google, Bing, Trends).
The user needs structured data extraction (ecommerce, real estate, travel, jobs, local business, YouTube).
The user needs bulk/async crawling with webhook fan-out.
The user explicitly asks about HasData or api.hasdata.com.
Prerequisites
A valid HasData API key. Store it in the environment variable HASDATA_API_KEY. Never hardcode, never log.
On Windows (PowerShell), set it for the current session:
Decision rule. Default to a Scraper API when one exists for the target platform (pre-parsed JSON, no selector maintenance). Use Web Scraping for arbitrary URLs not covered by an API. Reach for a Scraper Job only when no API equivalent exists — , , , , — when async fan-out + webhooks save engineering time over a paginated client loop.
Expected: JSON with requestMetadata.status equal to "ok".
2. Choose the execution mode
Scraper API — if the platform has a dedicated endpoint (Google, Amazon, Zillow, Airbnb, Indeed, YouTube, etc.). Load the matching reference file (see Reference Loading Guide below) for parameters.
Web Scraping API — for arbitrary URLs. Load references/web-scraping.md.
Scraper Job — for bulk/async. Load references/scraper-jobs.md.
3. Web Scraping API (POST /scrape/web)
Load references/web-scraping.md before constructing the request.
The submit response handle is body.id (an integer), notjobId. Persist it immediately.
Poll GET /scrapers/jobs/<id> every 10–30 s with backoff.
Treat webhooks as best-effort (3 retries). Always pair with polling.
On finished, the status carries data: {csv, json, xlsx} short-lived URLs — download immediately.
6. High-leverage patterns
SERP-first enrichment. Use Google SERP to surface public snippets for company/professional lookup before attempting direct scraping. Treat personal email/phone lookup as allowed only with a legitimate purpose and user authorization.
AI Mode + verify./scrape/google/ai-mode for the answer + references → /scrape/web (markdown) on each reference URL → cited RAG context, no vector DB.
Maps → leads./scrape/google-maps/search returns business websites and phones. Apply opt-out, rate, and privacy-law constraints before any outreach use.
Crawler → corpus.crawler Scraper Job with outputFormat: ["markdown"] + includePaths: "/docs/.+" produces an LLM-ready corpus in one submission.
Pre-extracted via SERP rich snippets.knowledgeGraph, localResults, inlineShoppingResults, relatedQuestions carry pre-parsed public facts. Always check them before considering direct page access.
Reference Loading Guide
Load these files on demand — when the user's request maps to a specific vertical or pattern:
Reference file
Load when…
references/web-scraping.md
Using POST /scrape/web — parameters, JS scenarios, AI extraction, cookie auth.
references/search.md
Google SERP / Light / AI Mode / News / Shopping / Bing / Trends + pagination.
references/ecommerce.md
Amazon (product, search, seller, seller-products) and Shopify.
references/real-estate.md
Zillow, Redfin (bracketed filters).
references/travel.md
Airbnb, Booking, Google Flights (occupancy rules, token pagination, IATA codes).
Ready-to-paste Python and TypeScript clients with retry, backoff, bounded concurrency, and the full job lifecycle.
Wiring from Code
Auth:x-api-key header on every request. Read from HASDATA_API_KEY env. Never hardcode, never log.
Timeouts: Set client timeout ≥ 300 s. HasData's own deadline is 300 s; shorter clients produce phantom failures while still being billed on completion.
Retries:429 and 5xx only — exponential backoff with jitter. Never retry 4xx (auth, validation).
Concurrency: Cap at your plan limit. The free tier is 1; anything higher just generates 429s.
Async jobs: Handle is body.id (integer), not jobId. Poll GET /scrapers/jobs/<id> every 10–30 s. Webhooks are best-effort; always pair with polling. Download result URLs immediately on finished.
See references/code-recipes.md for complete Python and TypeScript client implementations.
Pitfalls
300 s server deadline. Match your client timeout to at least 300 s. Shorter timeouts cause phantom failures while still being billed.
requestMetadata.status === "ok" is the only success signal. HTTP 200 alone is not enough — always check the metadata status field.
Disable jsRendering first. Enable only if the page needs it — most static pages parse fine without a headless browser. Rendering costs more and is slower.
No cookies parameter. Cookies go through headers["Cookie"], not a top-level field.
includePaths regex is case-sensitive./blog/.+ will not match /Blog/....
Scraper Job data is double-wrapped. Each row is body.data[i].data; the outer wrapper carries id, jobId, dataId, createdAt, updatedAt.
Async job handle is body.id (integer), not jobId. Persist it immediately after submit.
Webhooks are best-effort with 3 retries. Always have a polling fallback.
Concurrency cap. Free tier is 1 concurrent request. Exceeding your plan limit generates 429s.
Never retry 4xx.401 = invalid key, 403 = quota exhausted, 429 = concurrency cap. Only 429 and 5xx are retryable.
Result URLs are short-lived. Download csv/json/xlsx URLs immediately on job finished.
Verification
API key validity — run the smoke test and confirm requestMetadata.status === "ok":
