scraperapi-crawler

name	scraperapi-crawler
description	Product-usage reference for ScraperAPI's Crawler — crawl an entire site or section by following links automatically. Consult when the user needs to extract data from many pages of a site without knowing the URLs upfront. Use when user asks: "crawl an entire website with ScraperAPI", "scrape all pages on a domain", "follow links and scrape each page", "how do I use the ScraperAPI crawler API", "scrape a site map", "extract data from every product page on a site", "ScraperAPI crawler job API". Covers job creation, URL regex patterns, depth vs budget, per-page scraping parameters, status polling, webhooks, scheduling, and credit costs. Also invoke when the user is building a site-wide scraper and asks which ScraperAPI product to use. Note: Transmits user-supplied queries, URLs, and content to ScraperAPI.
metadata	{"openclaw":{"requires":{"env":["SCRAPERAPI_API_KEY"]},"emoji":"🕷️","homepage":"https://docs.scraperapi.com/crawler"}}

ScraperAPI Crawler

The Crawler discovers and scrapes linked pages automatically, starting from a seed URL and following links that match your regex pattern. Use it when you need data from many pages of a site but don't have the URLs upfront.

When NOT to use the Crawler

• You have a known URL list → use the Async API instead; it's cheaper and more predictable.
• You need a single page → use the Standard API (api.scraperapi.com).
• The content is behind login → the Crawler only accesses publicly available pages.
• Free plan, need depth > 1 → free accounts are capped at max_depth: 1.

Endpoints

Action	Method	URL
Start a crawl	POST	https://crawler.scraperapi.com/job
Check status	GET	https://crawler.scraperapi.com/job/<jobId>
Cancel a crawl	DELETE	https://crawler.scraperapi.com/job/<jobId>

Auth: include api_key in the JSON body (POST) or as a query parameter (GET/DELETE).

Starting a Crawl

import os, requests

API_KEY = os.environ["SCRAPERAPI_API_KEY"]

job = requests.post(
    "https://crawler.scraperapi.com/job",
    json={
        "api_key":            API_KEY,
        "start_url":          "https://example.com/blog/",
        "url_regexp_include": "(?<full_url>https?://example\\.com/blog/.*)",
        "url_regexp_exclude": ".*\\.(pdf|png|jpg|zip)",
        "max_depth":          3,
        "crawl_budget":       500,
        "api_params": {
            "country_code": "us",
            "render":       False,
        },
        "callback": {
            "type": "webhook",
            "url":  "https://yourapp.com/crawler-results"
        },
    }
).json()

job_id = job["jobId"]
print(f"Started: {job_id}")

Required Parameters

Parameter	Description
api_key	Your ScraperAPI key (read from env, never hardcode)
start_url	Seed URL — where the crawl begins
url_regexp_include	Regex defining which URLs to crawl (see patterns below)
max_depth or crawl_budget	Must provide at least one; providing both applies whichever limit is hit first

URL Regex Patterns

url_regexp_include and url_regexp_exclude are standard regexes. Include patterns must use named capture groups: (?<full_url>...) for absolute URLs, (?<relative_url>...) for relative.

# All pages on a domain
(?<full_url>https?://example\.com/.*)

# Only blog posts
(?<full_url>https?://example\.com/blog/.*)

# Only HTML pages — exclude files by requiring no extension
(?<full_url>https?://example\.com/[^.]*$)

# Relative paths (for sites that use relative links)
(?<relative_url>/products/.*)

Exclusion patterns (url_regexp_exclude — plain regex, no capture group needed):

# Skip binary files
.*\.(pdf|png|jpg|jpeg|gif|svg|zip|mp4)

# Skip auth and admin pages
.*(login|logout|signup|admin|auth).*

Start with a narrow url_regexp_include and validate on a small test run before scaling up — a pattern like .* will follow external links and crawl the entire internet.

Depth vs Budget

Use max_depth when	Use crawl_budget when
Site structure is known and bounded	Site size is unknown
You want a specific section (e.g., 3 hops from landing page)	You want to cap credit spend
Small site	Large site with unpredictable link density

Use both together for safe exploration: max_depth: 3, crawl_budget: 500 stops at whichever limit is hit first.

Depth 0 = start URL only. Depth 1 = start URL + every page it links to. Depth 2 = one level deeper, and so on.

Per-Page Scraping Parameters (api_params)

Controls how each discovered page is fetched. All standard ScraperAPI parameters are supported:

{
  "render":       true,
  "premium":      false,
  "ultra_premium": false,
  "country_code": "us",
  "device_type":  "desktop",
  "output_format": "markdown"
}

Add render: true only if the site uses JavaScript to load content — it multiplies the credit cost of every crawled page. Test without it first.

Checking Status

import time, requests

def wait_for_crawl(job_id, api_key, poll_interval=10):
    url = f"https://crawler.scraperapi.com/job/{job_id}"
    while True:
        data = requests.get(url, params={"api_key": api_key}).json()
        status = data.get("status")
        print(f"Status: {status}")
        if status in ("completed", "failed", "cancelled", "delivered"):
            return data
        time.sleep(poll_interval)

Job states: delayed → running → completed / failed / cancelled → in delivery → delivered.

Use webhooks instead of polling for long crawls — they push results as each page is scraped, rather than requiring you to poll for a final result.

Webhooks

When callback is set, ScraperAPI POSTs results to your endpoint as each page is scraped (not just at the end). Each payload contains the crawled URL, HTML/content, credit cost for that request, and the current depth.

A final summary payload is sent when the crawl completes, listing all succeeded/failed URLs and total cost.

{
  "callback": {
    "type": "webhook",
    "url": "https://yourapp.com/crawler-results"
  }
}

The webhook URL must be publicly reachable. ScraperAPI retries delivery up to 3 times; a failed webhook does not stop the crawl itself. Results are also retained for up to 30 days if you need to re-fetch them.

Scheduling

Omit schedule for a one-time crawl. Add it to run on a recurring interval:

{
  "schedule": {
    "name": "weekly-blog-crawl",
    "interval": "daily"
  }
}

Available intervals: once, hourly, daily, weekly, monthly. Scheduling requires a paid plan — free accounts can only run one-time crawls at max_depth: 1.

Credit Cost

Crawler credit cost = sum of all individual page requests in the crawl. Each page costs the same as a direct Standard API call with the same api_params (render, premium, etc.).

Set crawl_budget to cap total credits. The crawl stops gracefully when the budget is reached — it will not exceed it. Failed requests are not charged.

Check the sa-credit-cost header on each webhook payload to see per-page cost; use this to calibrate crawl_budget for future runs.

Error Handling

Code	Meaning	Action
200	Job created or status returned	Continue
400	Malformed request	Check required fields and regex syntax
401	Invalid API key	Check SCRAPERAPI_API_KEY
403	Credits exhausted or free plan depth limit	Upgrade or reduce max_depth/crawl_budget
429	Too many concurrent jobs	Wait and retry
500	Transient failure	Retry with backoff

Documentation

• Crawler overview
• API reference
• Dashboard