| name | generate-scraper |
| description | Use this skill ONLY when the user explicitly asks to generate, build, create, or write a scraper/parser for SPECIFIC URLs they already have. Trigger phrases: "generate scraper", "build scraper", "make scraper", "new scraper", "create a scraper", "write a scraper", "I need a scraper", "generate a scraper for", "build a scraper for", "create scraper for". Do NOT trigger for generic requests like "fetch this page", "download HTML", "get the content of this URL", "scrape this" (without mentioning scraper/parser), or any request that is not specifically about generating a reusable scraper file. IMPORTANT: Do NOT trigger if the user mentions "crawler" anywhere in their request — use the generate-crawler-scraper skill instead. If the request mentions "crawler", "crawler + scraper", "crawler-scraper", "crawl", "search for X on site Y", "busca", "buscando", or describes discovering/collecting product URLs from a listing page, use generate-crawler-scraper, NOT this skill. |
ScrapeOps — Generate Scraper Skill
Generates a production-ready single-URL scraper using the ScrapeOps AI backend via the scrapeops MCP server. The generated script:
- Accepts a URL as a CLI argument at runtime.
- Fetches HTML in-memory through the ScrapeOps proxy (
https://proxy.scrapeops.io/v1/ for HTTP-client libs, or residential-proxy.scrapeops.io:8181 for browser-based libs).
- Runs the extractor and writes the result to a JSONL output file.
- Reads the API key from the
SCRAPEOPS_API_KEY environment variable.
- Ships with a
README.md that explains install, env setup, and usage.
No HTML is downloaded to disk at generation time. No execution happens at the end — the user runs the scraper manually against any URL they want.
Overview
- Collect inputs (URLs, language, library)
- Submit job via
scrapeops_submit_job with parser_only: false + include_api_key: false
- Poll until complete via
scrapeops_poll_status
- Retrieve the full scraper via
scrapeops_get_code
- Install dependencies (optional)
- Save the file verbatim, then apply two targeted Edits: (a) API-key env var, (b) hardcoded URL list → CLI arg
- Generate
<domain>_README.md
- Show the final summary — do NOT execute the scraper
API key handling: The MCP server reads SCRAPEOPS_API_KEY from ~/.claude/settings.json automatically. Do NOT check or validate it before making tool calls. If a MCP tool call fails with "SCRAPEOPS_API_KEY is not configured", handle it inline:
- Ask the user: "Enter your ScrapeOps API key (find it at https://scrapeops.io/app/dashboard):"
- Read
~/.claude/settings.json, add the key to env.SCRAPEOPS_API_KEY using Edit, and retry the failed tool call
- Do NOT tell the user to run
/scrapeops-setup or restart — just ask, save, and continue
Step 1 — Collect Inputs
| Slot | Required |
|---|
urls | Yes — 1 to 5 URLs from the same domain. They are used ONLY as generation-time samples to teach the backend the page structure. At runtime the generated script accepts a single URL via CLI. |
language | Yes — always ask if not explicitly provided |
library | Yes — always ask if not explicitly provided |
country | No — skip if not mentioned |
Rule: never assume language or library. Ask one at a time using AskUserQuestion. Only skip a question if the user already stated that value clearly.
🚨 MANDATORY BEFORE MOVING TO STEP 2: After collecting language + library + URLs, you MUST run Step 1.5 (search-page detection) below. Do NOT jump straight from Step 1.4 to Step 2 — Step 1.5 routes the user to Search Page Mode when their URL is a search/query page (e.g. ?q=..., /search?...). Skipping Step 1.5 produces a broken single-URL scraper for what should have been a keyword-loop search scraper. This applies even when the user already provided language and library up front — the routing decision is independent of Steps 1.1–1.4.
1.1 — Ask language (if not provided)
question: "Which language do you want the scraper in?"
header: "Language"
options:
- label: "Python"
- label: "JavaScript"
- label: "PHP"
- label: "Other"
description: "Ruby, Go, Rust, Java, C#"
1.2 — Ask library (if not provided)
Use the EXACT labels and descriptions below. Do NOT add any marker like "(default)" / "(recommended)" / "most common" / "default choice" / "popular", and do NOT reorder the options.
Python:
question: "Which Python library?"
header: "Library"
options:
- label: "BeautifulSoup"
description: "HTML parser used with the requests HTTP client."
- label: "Scrapy"
description: "Full framework with project structure, middlewares, and built-in concurrency."
- label: "Playwright"
description: "Real browser — use when JS rendering is required."
- label: "Selenium"
description: "Real browser (older) — use when JS rendering is required."
JavaScript:
question: "Which JavaScript library?"
header: "Library"
options:
- label: "Cheerio"
description: "Server-side HTML parser used with axios."
- label: "Crawlee"
description: "Full Node.js crawling framework — built on Cheerio/Playwright with request queue and proxy management."
- label: "Playwright"
description: "Real browser — use when JS rendering is required."
- label: "Puppeteer"
description: "Headless Chrome automation."
Scrapy and Crawlee are framework libraries — if the user picks one, take a completely
different path (Framework Mode). They impose their own project structure, request
lifecycle, and middleware system, so the standard monolithic template from the Go backend
doesn't fit. After Step 1.4, if the chosen library is Scrapy or Crawlee, skip Steps 2–10
below and go straight to the # Framework Mode (Scrapy / Crawlee) section at the end of
this skill. Steps 2–10 assume a single standalone script with backend-baked proxy.
PHP / Ruby / Go / Rust / Java / C#: use AskUserQuestion with the options from references/languages.md. Apply the same "no preference marker" rule — don't add (recommended) / (default) anywhere.
1.3 — URLs (if not provided)
If the user didn't include a URL, ask: "Paste 1–5 URLs from the target website (same domain). These are used only as generation-time samples — the generated script will accept any URL via CLI."
1.4 — Map to API values
Use references/languages.md to map language + library selections to the correct target_language and target_library API values.
1.5 — 🚨 MANDATORY: Detect search-page URLs (route to Search Page Mode if matched)
This step is REQUIRED on every run. Do NOT skip to Step 2 without executing the detection below — even if the user provided language/library up front and you're tempted to "just go to confirmation". Search-page URLs need a totally different scraper shape (keyword × filters × pagination loop, not single-URL fetch). Missing this step produces the wrong scraper.
After collecting URLs (and before confirming in Step 2), run a quick heuristic on each URL to detect search pages. A search page is a URL whose query string carries a search keyword (?k=, ?q=, ?query=, ?s=, ?search=, ?term=, ?kw=, ?keyword(s)=) OR whose path looks like a search route (/search, /s/, /busca, /find, /results).
Apply both regexes via Bash:
SEARCH_URL_REGEX='[?&](k|q|query|search|s|term|kw|keywords?)=[^&]+'
SEARCH_PATH_REGEX='/(search|s|busca|find|results)(/|\?|$)'
echo "<URL>" | grep -E "$SEARCH_URL_REGEX|$SEARCH_PATH_REGEX"
If any of the user's URLs matches either regex, ask via AskUserQuestion:
question: "I detected a search page. Build a search scraper (keyword × filters × pagination loop)?"
header: "Search mode"
options:
- label: "Yes — build a search scraper"
description: "Loops over keywords, filters, and pages. Output: cards from search results page."
- label: "No — single URL only"
description: "Just scrape this exact URL once (default generate-scraper behavior)."
Routing based on the answer:
If no URL matches the regex, skip this step entirely and continue to Step 2.
Step 2 — Confirmation
Show a summary and confirm before submitting. You MUST use this EXACT format every time — do not rephrase, reorder, or use a different layout:
Now let me confirm before submitting:
Here's what I'll generate:
| Field | Value |
|------------|--------------------------------|
| **URL(s)** | <url1>, <url2>, ... |
| **Language**| <language> |
| **Library** | <library> |
| **Country** | <country or "not set"> |
Then ask for confirmation using AskUserQuestion:
question: "Does everything look correct?"
header: "Confirm"
options:
- label: "Yes, generate it"
- label: "No, change something"
If "No, change something" → ask what to change and loop back.
Step 3 — Submit the Job
Call the scrapeops_submit_job MCP tool with:
urls: array of URLstarget_language: from Step 1.4target_library: from Step 1.4parser_only: false — request the full runnable scraper (with proxy fetcher), not just the extract_data snippetinclude_api_key: false — the backend will emit "YOUR-API-KEY" as a placeholder, which Step 6 replaces with an env-var read
Do NOT pass api_key or api_url — the MCP server reads them from the environment automatically.
The tool returns a JSON object. Read version_id from it. If missing, show the full response and stop:
Failed to submit scraper job. Please check your API key and backend URL.
Otherwise confirm:
Job submitted (ID: <version_id>). Waiting for generation...
Step 4 — Poll for Completion
Call scrapeops_poll_status MCP tool with:
The tool waits 30 seconds internally before making the request — no sleep needed on your side.
After each call:
- Show:
[Poll #N] <_summary field from response>
status = "completed" → proceed to Step 5status = "error" / "failed" / "wrong_page_type" / "cancelled" / "expired" → show error_message and stop- Any other status (
queued, processing, running, step_N, generating_*, refactoring, ai_fix_suggestions, agent_fix, re_execute_parser) → call scrapeops_poll_status again
CRITICAL — ONLY proceed when status === "completed". Never based on completed_at being set, never based on _summary text, never based on elapsed time. Only the literal string "completed" in the status field. scrapeops_poll_status intentionally does NOT return output_code / link_output_code — those are only available via scrapeops_get_code in Step 5.
Jobs typically take 5–15 minutes but can take longer when the backend runs validation/self-heal loops. Keep polling silently until a terminal status. Never surface a "Stuck job" question.
Capture install_command from the final response.
Step 5 — Retrieve the Generated Code
Call scrapeops_get_code MCP tool with:
On success it returns { code, language, library, install_command, version_id }. Use the code field for the Write in Step 7.
If the tool errors saying the job isn't completed yet, loop back to Step 4 and keep polling.
Step 6 — Install Dependencies
If install_command is non-empty, show the command and ask via AskUserQuestion:
question: "Install dependencies now?"
header: "Dependencies"
options:
- label: "Yes, run"
- label: "No, I'll install later"
For Python (language = python) — always use a venv. pip install directly on system Python fails on macOS/Homebrew, Ubuntu 23.04+, Debian 12+, Fedora 38+ and other modern distros because of PEP 668. Detect and skip only if the user is already inside a venv ($VIRTUAL_ENV is set) or inside a Docker/CI container ($CI, /.dockerenv).
Flow:
- Check env:
[ -n "$VIRTUAL_ENV" ] — if already in a venv, just run pip install <pkgs>.
- Otherwise, create a project-local venv:
python3 -m venv .venv
source .venv/bin/activate
pip install --upgrade pip
pip install <pkgs from install_command>
- Tell the user: "Created a venv at
.venv/. Before running the scraper later, activate it: source .venv/bin/activate."
- Include that activation step in the README (Step 8).
For JavaScript / Node.js — just run npm install <pkgs> in the current directory. Create package.json if missing: npm init -y first.
For other languages (PHP, Ruby, Go, Rust, Java, C#) — show the install command from install_command and ask the user to install manually; don't try to run language-specific package managers from the skill.
Step 7 — Save the Scraper File
Compute the domain slug from the first URL: books.toscrape.com → books_toscrape_com. Extension by language: .py, .js, .php, .rb, .go, .rs, .java, .cs.
Save <slug>_scraper.<ext> using the Write tool (never echo/heredoc).
CRITICAL — save VERBATIM. Write exactly what scrapeops_get_code returned in the code field. Do NOT refactor, reformat, rename variables, add type hints, add docstrings, remove comments, or "clean up" imports. The only modifications allowed are the two targeted Edits in Step 8.
Step 8 — Refactor the saved file for CLI use
The Go backend emits a template with (a) a hardcoded "YOUR-API-KEY" literal, and (b) a hardcoded list of URLs inside main() / if __name__ == "__main__" / equivalent. Two Edits fix both.
8.1 — Replace "YOUR-API-KEY" with an env-var read
Use Edit on the saved file:
| Language | Replace "YOUR-API-KEY" with |
|---|
| Python | os.environ.get("SCRAPEOPS_API_KEY", "") |
| JavaScript / Node | process.env.SCRAPEOPS_API_KEY || "" |
| PHP | getenv("SCRAPEOPS_API_KEY") ?: "" |
| Ruby | ENV["SCRAPEOPS_API_KEY"] || "" |
| Go | os.Getenv("SCRAPEOPS_API_KEY") |
| Rust | std::env::var("SCRAPEOPS_API_KEY").unwrap_or_default() |
| Java | System.getenv("SCRAPEOPS_API_KEY") |
| C# | Environment.GetEnvironmentVariable("SCRAPEOPS_API_KEY") |
The assignment in the template usually looks like API_KEY = "YOUR-API-KEY" (or the language equivalent). The Edit's old_string should include the quotes — e.g. old_string = "YOUR-API-KEY", new_string = os.environ.get("SCRAPEOPS_API_KEY", "") — so the surrounding assignment becomes API_KEY = os.environ.get("SCRAPEOPS_API_KEY", "").
For Python, ensure import os is at the top of the file. If missing, add it.
For Go, ensure "os" is in the import block.
For Rust, no extra import needed (std::env is stdlib).
For Java/C#, ensure System / Environment namespaces are accessible (usually yes by default).
8.2 — Replace the hardcoded URL list with a CLI arg read
The template iterates over a hardcoded list. The user runs the script with one URL at the CLI:
python3 <slug>_scraper.py <url>
node <slug>_scraper.js <url>
php <slug>_scraper.php <url>
ruby <slug>_scraper.rb <url>
go run <slug>_scraper.go <url>
cargo run -- <url> # Rust, inside a cargo project
java <ClassName> <url>
dotnet run -- <url> # C#
Find the hardcoded URL list in the saved file by running Bash with grep -n using a language-appropriate pattern:
| Language | grep pattern |
|---|
| Python | urls = \[ or URLS = \[ |
| JavaScript | const urls = \[ or let urls = \[ or urls: \[ |
| PHP | \$urls = \[ |
| Ruby | urls = \[ |
| Go | urls := \[\]string{ |
| Rust | let urls = vec!\[ or let urls: Vec<.*?> = vec!\[ |
| Java | String\[\] urls = { or List<String> urls = |
| C# | string\[\] urls = { or var urls = new .*?{ |
Use Read to see the full block (from the urls = [ start through the closing ]). Then Edit replacing the entire block with CLI arg parsing:
| Language | Replacement block |
|---|
| Python | python<br>import sys<br>if len(sys.argv) < 2:<br> sys.exit("Usage: python3 <script>.py <url>")<br>urls = [sys.argv[1]]<br> |
| JavaScript | js<br>if (process.argv.length < 3) {<br> console.error("Usage: node <script>.js <url>");<br> process.exit(1);<br>}<br>const urls = [process.argv[2]];<br> |
| PHP | php<br>if ($argc < 2) { fwrite(STDERR, "Usage: php <script>.php <url>\n"); exit(1); }<br>$urls = [$argv[1]];<br> |
| Ruby | ruby<br>abort("Usage: ruby <script>.rb <url>") if ARGV.empty?<br>urls = [ARGV[0]]<br> |
| Go | go<br>if len(os.Args) < 2 { fmt.Fprintln(os.Stderr, "Usage: go run <script>.go <url>"); os.Exit(1) }<br>urls := []string{os.Args[1]}<br> |
| Rust | rust<br>let url = std::env::args().nth(1).unwrap_or_else(|| { eprintln!("Usage: <bin> <url>"); std::process::exit(1); });<br>let urls = vec![url];<br> |
| Java | java<br>if (args.length < 1) { System.err.println("Usage: java <Class> <url>"); System.exit(1); }<br>String[] urls = { args[0] };<br> |
| C# | csharp<br>if (args.Length < 1) { Console.Error.WriteLine("Usage: dotnet run -- <url>"); Environment.Exit(1); }<br>string[] urls = { args[0] };<br> |
For Python, also ensure import sys is at the top (add if missing). For Go, ensure "fmt" and "os" are in the import block.
After Edit, verify with grep:
- The hardcoded URLs are gone:
grep -n '<original-URL>' <file> should return nothing.
- The CLI arg parsing is present:
grep -n 'argv\|ARGV\|Args\|os.Args\|std::env::args' <file> should show matches.
If the backend's template uses a different URL-list shape than the patterns above (can happen for PHP/Ruby/Go/Rust/Java/C# since those don't have dedicated templates in the backend and the LLM produces variable shapes), use judgment:
- Grep broadly for string literals that look like URLs.
- Read the surrounding 20-40 lines to understand the entry-point structure.
- Replace the initialization of whatever variable holds the URL list with the CLI arg read.
- Preserve any loop/worker structure that iterates over that variable.
Step 9 — Generate <slug>_README.md
Use the Write tool to create a README that explains:
- What the scraper does — one paragraph.
- Prerequisites — runtime version (Python 3.9+ / Node 18+ / PHP 8.1+ / etc.).
- Install — the
install_command from Step 5. For Python, show venv setup commands first:
python3 -m venv .venv
source .venv/bin/activate
pip install --upgrade pip
<install_command>
- Set the API key — export env var:
export SCRAPEOPS_API_KEY=<your-scrapeops-key>
Find the key at https://scrapeops.io/app/dashboard.
- Run — one line showing the CLI form for the chosen language:
python3 <slug>_scraper.py "<url>"
(or node …, php …, ruby …, go run …, cargo run -- …, java …, dotnet run -- …).
- Output — one paragraph on where the output goes (default JSONL filename from the template, one line per URL processed). For single-URL runs, the file will have a single JSON line.
- How it works under the hood — brief: fetches through
https://proxy.scrapeops.io/v1/?api_key=$SCRAPEOPS_API_KEY&url=<url> (or residential proxy for browser libs), runs the extractor, writes JSONL.
- Troubleshooting:
SCRAPEOPS_API_KEY not set → echo $SCRAPEOPS_API_KEY should be non-empty.- HTTP 401/403 → check the key at the dashboard.
- Empty output → the page may need JS rendering; regenerate with Playwright/Selenium/Puppeteer.
Keep the README concise (under 100 lines). Do NOT include examples of the expected output JSON — the user will see it when they run the script.
Step 10 — Final Summary
Show:
Scraper generated successfully!
Files saved:
<slug>_scraper.<ext>
<slug>_README.md
Language: <language> / <library>
Job ID: <version_id>
Before running:
export SCRAPEOPS_API_KEY=<your-key>
Run:
python3 <slug>_scraper.py "<any URL from the same domain>"
Do NOT execute the scraper. The user runs it manually with whatever URL they want. Never download HTML locally. Never invoke the parser at this step. The job is done.
Error Handling
| Situation | Response |
|---|
| Missing API key on MCP call | Ask user inline (see box at top), save to ~/.claude/settings.json, retry |
scrapeops_submit_job returns error | Show full response, stop |
scrapeops_poll_status returns an error status | Show error_message, stop |
scrapeops_get_code returns _error (job not completed) | Loop back to Step 4 and keep polling |
scrapeops_get_code returns _error: "neither output_code nor link_output_code is set" | Show warning + raw response, stop |
| Step 8.2 grep finds no URL list (non-Python/JS language with off-template LLM output) | Read the file, use judgment to locate the URL variable, Edit accordingly. If the structure is truly opaque, show the user the saved file and ask them to wire the CLI arg themselves — don't guess blindly. |
Important Notes
- The
scrapeops MCP server is started automatically by the plugin — no manual setup.
parser_only: false + include_api_key: false are the two flags that matter for the standard flow (Steps 2–10). Framework Mode (Scrapy/Crawlee) uses parser_only: true instead.- All URLs must be from the same domain — the backend validates this.
- Maximum 5 URLs per job — they are generation-time samples, not runtime inputs.
- The generated script runs on ONE URL at a time (from CLI). For batch use, the user calls the script multiple times or wraps it in a loop.
- No smoke test, no local HTML download, no parser execution at generation time. The standard skill ends at Step 10. Framework Mode runs a smoke test (FM-Step 7) because frameworks have non-trivial wiring that benefits from a single end-to-end check.
Framework Mode (Scrapy / Crawlee)
Active when the user picks Python + Scrapy or JavaScript + Crawlee in Step 1. The Go
backend doesn't return monolithic Scrapy projects or Crawlee projects — it returns
extractor functions in BeautifulSoup (Python) or Cheerio (JavaScript) and the skill builds
the framework project locally around them. ScrapeOps proxy is injected via the
framework's idiomatic mechanism (Scrapy DownloaderMiddleware, Crawlee
preNavigationHooks / ProxyConfiguration).
Reference files:
references/frameworks.md — templates, BS4→Scrapy and Cheerio→Crawlee cheatsheets, decision matrix.references/proxies.md — offline fallback for the two ScrapeOps proxy product docs.
FM-Step 1 — Confirmation (replaces Step 2)
Show a summary table identifying Framework Mode explicitly:
Now let me confirm before submitting (Framework Mode):
| Field | Value |
|--------------|--------------------------------|
| **URL** | <url> |
| **Language** | <language> |
| **Library** | <library> (framework mode) |
| **Country** | <country or "not set"> |
Then ask via AskUserQuestion:
question: "Does everything look correct?"
header: "Confirm"
options:
- label: "Yes, generate it"
- label: "No, change something"
FM-Step 2 — Fetch framework docs (mandatory)
Download the canonical framework docs to /tmp/. Use Bash curl with a browser
User-Agent, not WebFetch (Cloudflare blocks it on docs.scrapy.org; crawlee.dev is
usually fine via WebFetch but curl is the safe default).
UA="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
For Scrapy:
curl -sSL -A "$UA" "https://docs.scrapy.org/en/latest/intro/tutorial.html" -o /tmp/scrapy_tutorial.html
curl -sSL -A "$UA" "https://docs.scrapy.org/en/latest/topics/settings.html" -o /tmp/scrapy_settings.html
curl -sSL -A "$UA" "https://docs.scrapy.org/en/latest/topics/downloader-middleware.html" -o /tmp/scrapy_middleware.html
For Crawlee:
curl -sSL -A "$UA" "https://crawlee.dev/docs/quick-start" -o /tmp/crawlee_quickstart.html
curl -sSL -A "$UA" "https://crawlee.dev/docs/guides/proxy-management" -o /tmp/crawlee_proxy.html
curl -sSL -A "$UA" "https://crawlee.dev/api/cheerio-crawler/class/CheerioCrawler" -o /tmp/crawlee_cheerio.html
Verify each file is non-empty HTML (head -c 200 /tmp/<file> | grep -q "<html\|<!DOCTYPE").
If ALL fail, fall back to references/frameworks.md and warn the user.
FM-Step 3 — Fetch ScrapeOps proxy docs (MANDATORY every run)
This is non-negotiable. The plugin must consult both proxy product docs before writing
any proxy code, on every Framework Mode run. Order:
-
Try WebFetch for both URLs:
https://scrapeops.io/docs/web-scraping-proxy-api-aggregator/quickstart/https://scrapeops.io/docs/residential-mobile-proxy-aggregator/overview/
Use a prompt that extracts: endpoint URL, auth method, target-URL parameter, listed
optional parameters, Python and JS snippets. If both succeed and contain those
fields, proceed to FM-Step 4 with the live data.
-
If WebFetch fails (403, timeout, empty), fall back to Bash curl with the same UA
pattern as FM-Step 2:
curl -sSL -A "$UA" "https://scrapeops.io/docs/web-scraping-proxy-api-aggregator/quickstart/" -o /tmp/scrapeops_proxy_api.html
curl -sSL -A "$UA" "https://scrapeops.io/docs/residential-mobile-proxy-aggregator/overview/" -o /tmp/scrapeops_residential.html
Then Read + Grep to extract the same fields.
-
If both methods fail for both URLs, fall back to references/proxies.md and warn the
user explicitly:
scrapeops.io docs unreachable — using offline reference. The generated proxy code
reflects a 2026-04 snapshot; verify against the live docs before running in
production.
The skill does NOT proceed to FM-Step 5 until it has, for both proxy products: the
endpoint, the auth format, and at least the listed optional parameters.
FM-Step 4 — Submit the parser-only job
Call scrapeops_submit_job with the intermediate target_library from
references/languages.md:
| Framework | target_language | target_library (intermediate) | parser_only |
|---|
| Scrapy | python | beautifulsoup | true |
| Crawlee | javascript | cheerio | true |
Other parameters: urls (the user's input URLs), no scraper_type (let backend
auto-detect), no include_api_key (irrelevant when parser_only: true — there's no proxy
fetcher to inject the key into).
Poll with scrapeops_poll_status as in Step 4 — only proceed when
status === "completed". Then call scrapeops_get_code and save the returned code
in memory as extract_data_body (a Python BS4 function for Scrapy, or a JS Cheerio
function for Crawlee).
FM-Step 5 — Convert / adapt the extractor
Scrapy (BS4 → native Scrapy selectors)
The Go backend returns def extract_data(soup, base_url=None) -> dict. Convert to
def extract_data(response, base_url=None) -> dict using the cheatsheet in
references/frameworks.md (Section A.2). Save the result as extract_data_scrapy_body.
Validation grep — ABSOLUTE rule, must return zero matches:
grep -E 'BeautifulSoup|from bs4|import bs4|soup\.|\.get_text\(|\.prettify\(\)' <converted_file>
If after one rewrite pass any token remains, invoke the parser-fixer agent on the file
with explicit "do NOT keep BS4 tokens" instructions.
Crawlee (Cheerio adapter)
The Go backend returns function extract_data($, baseUrl) { ... } using Cheerio. Cheerio
is the native parser inside Crawlee's CheerioCrawler — no selector conversion
needed. Just save the body verbatim as extract_data_body for use in src/extractor.js.
Validation grep — must return zero matches in the final project:
grep -rE "require\(['\"]axios['\"]\)|from ['\"]axios['\"]|require\(['\"]https?['\"]\)" <slug>_crawlee/src/
FM-Step 6 — Choose the proxy product
Apply the decision matrix from references/frameworks.md Section C and references/proxies.md:
| Framework + transport | Proxy product | Why |
|---|
| Scrapy default downloader | Proxy API Aggregator | URL rewrite is clean in Scrapy middleware; pay-per-success cheapest |
Crawlee CheerioCrawler (default) | Proxy API Aggregator | URL rewrite via preNavigationHooks; pay-per-success |
Crawlee PlaywrightCrawler (if user explicitly asked for browser/JS render) | Residential & Mobile Proxy Aggregator | URL rewrite breaks browser navigation; need real proxy on the wire |
Print the choice and reason to the user (one line each):
Selected proxy: <product> — <one-line reason>.
For the first cut of Crawlee, default to CheerioCrawler + Proxy API Aggregator. Switch
to PlaywrightCrawler + Residential only if the user said they need JS rendering (you can
ask if unsure).
FM-Step 7 — Assemble the framework project
Use templates from references/frameworks.md filled with values gathered so far.
Critically — every Framework Mode project must include the framework's native machinery
(items + loaders, real pipelines, middlewares, sessions, routers, failed-request handlers,
autothrottle/concurrency). A thin wrapper around extract_data(...) is NOT acceptable.
The required-features list is in references/frameworks.md Section E (E.1 for Scrapy, E.2
for Crawlee, E.3 for the validation greps you must run before declaring the project done).
Compute:
slug = <domain-slug> (e.g. books_toscrape_com → books_toscrape_com_scrapy or
books_toscrape_com_crawlee).- For Scrapy:
project_pkg = <domain-slug> (no dashes, valid Python identifier).
- API key: read from env var
SCRAPEOPS_API_KEY at runtime — NEVER bake into source.
Scrapy single-URL project
Write the files documented in references/frameworks.md Section A.3, with all features
from Section E.1 wired in (selectors mixing CSS+XPath, ItemLoader, three real Item
Pipelines, autothrottle in settings, both ScrapeOpsProxyMiddleware and Scrapy's built-in
RetryMiddleware configured):
<slug>_scrapy/scrapy.cfg<slug>_scrapy/<project_pkg>/__init__.py (empty)<slug>_scrapy/<project_pkg>/settings.py — populated with AUTOTHROTTLE_ENABLED = True,
RETRY_TIMES, RETRY_HTTP_CODES, ITEM_PIPELINES registering all three real pipelines,
DOWNLOADER_MIDDLEWARES with ScrapeOpsProxyMiddleware.<slug>_scrapy/<project_pkg>/items.py — ProductItem(scrapy.Item) with one
scrapy.Field() per top-level key returned by extract_data, plus a
ProductLoader(scrapy.loader.ItemLoader) subclass with default_output_processor,
_in / _out processors per field (MapCompose(str.strip), TakeFirst(),
Identity() for lists, Join(" ") for joined text).<slug>_scrapy/<project_pkg>/pipelines.py — three real classes:
ValidationPipeline (drops items missing required keys), DuplicatesPipeline
(tracks url/productId in a set), NormalizationPipeline (whitespace + URL
resolution + numeric coercion). Each raises DropItem from scrapy.exceptions where
appropriate.<slug>_scrapy/<project_pkg>/middlewares.py — ScrapeOpsProxyMiddleware populated with
the endpoint and auth format from FM-Step 3 (live docs preferred).<slug>_scrapy/<project_pkg>/spiders/__init__.py (empty)<slug>_scrapy/<project_pkg>/spiders/scraper.py — pastes extract_data_scrapy_body
(which mixes response.css and response.xpath per Section A.2) and defines
ScraperSpider with -a url= CLI arg. The spider's parse_product populates
ProductLoader (NOT a raw dict) and yields loader.load_item().
Crawlee single-URL project
Write the files documented in references/frameworks.md Section B.3, with all features
from Section E.2 wired in (router with default handler, useSessionPool: true,
failedRequestHandler writing to a failed Dataset, maxConcurrency,
preNavigationHooks for proxy):
<slug>_crawlee/package.json<slug>_crawlee/src/extractor.js — export function extract_data($, baseUrl) { ... }
with the body from extract_data_body. Uses both CSS ($('selector')) and XPath where
appropriate (.find() traversal helpers, or for PlaywrightCrawler page.locator('xpath=//...')).<slug>_crawlee/src/router.js — createCheerioRouter() (or createPlaywrightRouter())
with a single default handler that calls extract_data($, request.userData.originalUrl)
and pushData(...). Even for one URL, route through the router.<slug>_crawlee/src/proxy.js — Proxy API Aggregator URL builder (or Residential
ProxyConfiguration if browser transport was chosen in FM-Step 6).<slug>_crawlee/src/main.js — CheerioCrawler (or PlaywrightCrawler) with:
requestHandler: router, preNavigationHooks rewriting URL through Proxy API,
useSessionPool: true, sessionPoolOptions: { maxPoolSize: 100 },
failedRequestHandler pushing to a failed Dataset, maxRequestsPerCrawl: 1 for the
single-URL skill, maxConcurrency: 1.
After writing, run both sets of validation greps:
- From FM-Step 5: zero
BeautifulSoup/from bs4/etc. (Scrapy) or
axios/require('http') (Crawlee) matches.
- From
references/frameworks.md Section E.3: each required native feature must be
detectable via grep. If any prints MISSING:, the project is broken — fix the
assembly before moving to FM-Step 8.
FM-Step 8 — Smoke test (1 URL) + local self-heal
Run the project against the user's first URL. The framework's wiring is non-trivial; a
smoke test catches assembly mistakes before the user sees them.
Scrapy
cd <slug>_scrapy
python3 -m venv .venv && source .venv/bin/activate
pip install --upgrade pip
pip install scrapy
export SCRAPEOPS_API_KEY=<value>
scrapy crawl scraper -a url="<first user URL>" -O smoke_out.jsonl
Validate smoke_out.jsonl: file exists, has ≥1 line, the line is valid JSON with at least
the top-level fields the schema implied (e.g. name, price, etc.).
If broken, fetch the URL's HTML via scrapeops_fetch_html, save to
/tmp/<slug>_sample.html, and invoke parser-fixer directly:
Agent(
description: "Fix Scrapy spider locally",
subagent_type: "parser-fixer",
prompt: "Fix the Scrapy spider at <slug>_scrapy/<project_pkg>/spiders/scraper.py.
Inputs:
- project_path: <slug>_scrapy
- parser_path: <slug>_scrapy/<project_pkg>/spiders/scraper.py (the extract_data function inside)
- html_path: /tmp/<slug>_sample.html
- language: python
- task: Smoke test failed — <stderr or empty output description>.
Run command: cd <slug>_scrapy && scrapy crawl scraper -a url=\"<URL>\" -O /tmp/smoke_test.jsonl
CRITICAL: keep the spider Scrapy-pure. Do NOT add `from bs4`, `BeautifulSoup`, or any BS4
usage. Only native response.css / response.xpath. Do NOT touch the
`SCRAPEOPS_API_KEY = os.environ.get(...)` line in settings.py. Verify with grep before
finishing."
)
Crawlee
cd <slug>_crawlee
npm install
export SCRAPEOPS_API_KEY=<value>
node src/main.js --url "<first user URL>"
Output goes to storage/datasets/default/*.json. Validate ≥1 file with valid JSON.
If broken, same pattern: fetch HTML, invoke parser-fixer with project_path
<slug>_crawlee and parser_path <slug>_crawlee/src/extractor.js. Critical guard:
"do NOT add axios or hand-rolled http imports — Crawlee owns transport".
FM-Step 9 — Generate <slug>_<framework>/README.md
Write a README with:
- What the project is — domain, language, framework, generated date.
- Prerequisites — Python 3.9+ / Node 18+. Show the venv setup commands explicitly for
Scrapy.
- API key —
export SCRAPEOPS_API_KEY=<your-key> is required. State that the project
reads from env at runtime; rotating the key needs no code edits.
- Run — exact CLI for the framework:
- Scrapy:
scrapy crawl scraper -a url="<target>" -O out.jsonl
- Crawlee:
node src/main.js --url "<target>"
- Output — Scrapy writes JSONL to the path passed via
-O. Crawlee writes JSON files
into ./storage/datasets/default/.
- Proxy — name the chosen ScrapeOps proxy product and how it's wired (one paragraph).
Mention how to disable for debugging (comment out middleware / hook).
- Customize — point at
extract_data for selectors, settings.py /
CheerioCrawler config for concurrency.
- Troubleshooting — same items as the standard README (Step 9), plus "verify proxy
middleware is enabled".
FM-Step 10 — Final Summary
Show the project tree, the run command, and a one-line note that proxy is on by default.
Do NOT execute anything else after the smoke test.
Scraper project generated successfully (Framework Mode).
Directory: ./<slug>_<framework>/
└── <tree of files written>
Language / library: <language> / <library>
Selected proxy: <Proxy product> — <one-line reason>
Before running:
export SCRAPEOPS_API_KEY=<your-key>
Run:
<runner command>
Framework Mode notes
- Always consult the proxy docs at runtime (FM-Step 3) — the offline fallback
references/proxies.md is insurance, not the source of truth.
include_api_key is irrelevant in Framework Mode. parser_only: true returns only
the extractor function — there's no API_KEY = "YOUR-API-KEY" literal to substitute.
The key is read from env at runtime by the framework project's settings/proxy config.- Backend
target_library is intermediate. Sending beautifulsoup for Scrapy and
cheerio for Crawlee is correct — that's what the backend can produce today, and the
skill converts/adapts locally.
- Smoke test is mandatory in Framework Mode. Frameworks have non-trivial wiring (proxy
middleware, project layout, item pipelines) that the standard skill doesn't have. A
smoke test catches assembly errors before the user runs it.
parser-fixer for self-heal must be told NOT to reintroduce BS4 (Scrapy) or axios
(Crawlee). Hard-code that constraint in the prompt; verify with grep after the agent
returns.
Search Page Mode (BS4 / Cheerio)
Active when Step 1.5 confirmed the URL is a search page AND the chosen library is
BeautifulSoup (Python) or Cheerio (JavaScript). The Go backend doesn't generate
keyword-driven scrapers natively — instead, it returns an extract_data($, base_url)
function for a single search-results page, and the skill wraps it locally with a URL
builder that loops over keywords × filters × pages.
The generated script supports three keyword input modes at runtime — --keywords
(comma-separated CLI), --keywords-file (JSON), or hardcoded defaults — plus one CLI
flag per filter the user chose to expose, plus --max-pages. Output is JSONL with one
line per product card found, annotated with keyword, page, and source_url.
Frameworks (Scrapy, Crawlee) for search pages are out of scope for this version. If
the user picked Scrapy/Crawlee + search URL, Step 1.5 already routed them to a switch
question.
SP-Step 1 — Confirmation (replaces Step 2)
Show a summary table identifying Search Page Mode explicitly:
Now let me confirm before submitting (Search Page Mode):
| Field | Value |
|---------------|--------------------------------|
| **URL** | <url> |
| **Language** | <language> |
| **Library** | <library> |
| **Mode** | Search page (keyword × filters × pagination) |
| **Country** | <country or "not set"> |
Then ask via AskUserQuestion:
question: "Does everything look correct?"
header: "Confirm"
options:
- label: "Yes, generate it"
- label: "No, change something"
SP-Step 2 — Discovery: parse the URL structure
Local URL parsing — no fetch, no network. Use Bash + Python urllib.parse (or node -e /
URL for JS users — Python is fine even for a JS project, just for parsing):
python3 - <<'PY'
from urllib.parse import urlsplit, parse_qsl
u = urlsplit("<URL>")
print("base:", f"{u.scheme}://{u.netloc}{u.path}")
print("params:", list(parse_qsl(u.query, keep_blank_values=True)))
PY
Identify:
KEYWORD_PARAM — the param matching the search regex (k, q, query, s,
search, term, kw, keyword, keywords).KEYWORD_VALUE_DEFAULT — the current value of that param (becomes one of the
hardcoded keywords in the script default).PAGINATION_PARAM — try page, p, offset, start, from. Note which (if any)
is present in the initial URL. If none, leave as None for now — SP-Step 3's DOM
inspection will try to find pagination via the page itself.PAGINATION_KIND — page-number if the value is small (1–50), offset-based if the
value looks like a row offset (multiples of 12/24/48). When in doubt, default to
page-number.OTHER_QUERY_PARAMS — every other key=value pair. These are filter candidates.SEARCH_URL_BASE — <scheme>://<netloc><path> (without any query string).
Save these as variables in your working memory for SP-Steps 5 / 7 / 8.
SP-Step 3 — Discovery: inspect the page DOM
Fetch the search page once via scrapeops_fetch_html to:
-
Confirm pagination structure (especially when SP-Step 2 found no pagination
param). Grep the saved HTML for:
grep -E '<a[^>]+rel="next"' <html_file>
grep -E 'pagination|paginator|page-link' <html_file> | head -10
grep -oE '\?[a-z]+=([0-9]+|next)[^"]*' <html_file> | sort -u | head -20
If a rel="next" link or numbered anchor exists, parse out the param name and adopt
it as PAGINATION_PARAM.
-
Identify filter candidates by scanning forms and sidebar/aside facets:
grep -E '<form[^>]*(role="search"|class="[^"]*(filter|facet|refine)[^"]*")' <html_file> | head -10
grep -oE '<input[^>]+name="[^"]+"' <html_file> | sort -u
grep -oE '<select[^>]+name="[^"]+"' <html_file> | sort -u
grep -oE 'data-(filter|facet|refine)-[a-z-]+="[^"]*"' <html_file> | sort -u
-
Combine the candidates: union of OTHER_QUERY_PARAMS (from SP-Step 2) and
name= attributes found in the DOM. Deduplicate. For each candidate, capture (when
visible): the human-readable label / placeholder / select options. SP-Step 4
auto-exposes all of them as CLI flags — no user filter-selection prompt.
The HTML may be saved at .scrapeops_cache/<hash>.html — scrapeops_fetch_html already
handles caching (see mcp-server/index.js linhas 261–316). Use that file path with
Read / Grep / Bash.
SP-Step 4 — Auto-expose all detected filters as CLI flags
Do NOT prompt the user. Every filter candidate from SP-Step 3 is automatically
exposed as a --<name> CLI flag in the generated script — no multi-select, no question.
The user already chose this trade-off at skill design time: more flags is fine, friction
during generation is not.
Build EXPOSED_FILTERS from the SP-Step 3 candidates directly:
EXPOSED_FILTERS = {
"<filter_name_1>": None,
"<filter_name_2>": None,
...
}
FIXED_FILTERS stays empty in the default path. If the original URL had query params that
DON'T look like filters (e.g. affid, tracking_id, _refineresult, session tokens —
parameters that should be preserved verbatim, not user-controlled), put those in
FIXED_FILTERS = {<name>: <original_value>} instead. Use judgment: tracking/session
params → fixed; user-meaningful params (price, category, sort, brand, rating) → exposed.
Print to the user a short, non-blocking summary so they know what was wired:
✓ Auto-exposed <N> filters as CLI flags: --min-price, --max-price, --sort, --cat-id
(Edit the script's EXPOSED_FILTERS / FIXED_FILTERS blocks if you want to move
any between exposed and fixed.)
No AskUserQuestion here. Continue straight to SP-Step 5.
SP-Step 5 — Keyword source choice
AskUserQuestion:
question: "How should the keyword list be provided?"
header: "Keywords"
options:
- label: "Inline now (comma-separated)"
description: "Type them now. They become the script's hardcoded default."
- label: "Sample with the URL keyword + 4 placeholders"
description: 'Default = ["<KEYWORD_VALUE_DEFAULT>", "TODO_keyword_2", "TODO_keyword_3", "TODO_keyword_4", "TODO_keyword_5"]. User edits later.'
- label: "From a JSON file at runtime only"
description: "No hardcoded default. Script always requires --keywords or --keywords-file."
Resolve to DEFAULT_KEYWORDS:
- Inline: ask a free-text follow-up
"Paste the keywords (comma-separated):". Split,
trim, drop empties.
- Sample:
[KEYWORD_VALUE_DEFAULT, "TODO_keyword_2", "TODO_keyword_3", "TODO_keyword_4", "TODO_keyword_5"].
- JSON-file-only:
[] (the script will exit with an error if invoked without
--keywords or --keywords-file).
Note which choice was made — SP-Step 9 conditionally writes a sample JSON file.
SP-Step 6 — Generate the search-page schema
Invoke the /generate-data-schema skill programmatically:
target_page_type: product_search_page
user_description: "Search results page for '<KEYWORD_VALUE_DEFAULT>' on <domain> — extract product cards with name, price, url, image, rating; plus pagination."
output_path: ./<slug>_search_schema.json
The skill writes the schema and returns the path. Read the file with the Read tool so
the JSON is in context for SP-Step 7.
SP-Step 7 — Submit the parser-only job
Call scrapeops_submit_job with:
| Param | Value |
|---|
urls | [<the original URL with KEYWORD_VALUE_DEFAULT>] (just one — backend uses it as a sample) |
target_language | python (BS4) or javascript (Cheerio) |
target_library | beautifulsoup or cheerio |
scraper_type | product_search_page |
parser_only | true |
schema_json | parsed content of <slug>_search_schema.json |
country | from Step 1 (if the user provided one) |
Do NOT pass include_api_key — parser_only: true returns just the extractor function,
no API_KEY = "..." literal exists.
Poll with scrapeops_poll_status as in Step 4 — only proceed when
status === "completed". Then call scrapeops_get_code and save the returned code
in memory as extract_data_body (a Python BS4 function or a JS Cheerio function).
SP-Step 8 — Render the local script
Compute slug from domain (books.toscrape.com → books_toscrape_com). Extension by
language: .py for Python+BS4, .js for JavaScript+Cheerio.
Render the appropriate template, substituting all discovery-time constants. Save as
<slug>_search.<ext> using Write.
Python (BS4) template
"""<domain> search scraper — keywords × filters × pagination.
Generated by ScrapeOps generate-scraper on <YYYY-MM-DD>. Output is JSONL: one line per
product card visible on a search results page, annotated with `keyword`, `page`, and
`source_url`.
"""
import argparse
import json
import os
import sys
from urllib.parse import urlencode, quote
import requests
from bs4 import BeautifulSoup
API_KEY = os.environ.get("SCRAPEOPS_API_KEY", "")
PROXY_BASE = "https://proxy.scrapeops.io/v1/"
SEARCH_URL_BASE = "<SEARCH_URL_BASE>"
KEYWORD_PARAM = "<KEYWORD_PARAM>"
PAGINATION_PARAM = "<PAGINATION_PARAM>"
PAGINATION_KIND = "<page-number|offset-based>"
PAGE_SIZE = <PAGE_SIZE_OR_NONE>
FIXED_FILTERS = <FIXED_FILTERS_DICT_LITERAL>
DEFAULT_KEYWORDS = <DEFAULT_KEYWORDS_LIST>
DEFAULT_MAX_PAGES = 1
EXPOSED_FILTERS = <EXPOSED_FILTERS_DICT>
<extract_data_body>
def build_search_url(keyword, filters, page):
params = {KEYWORD_PARAM: keyword}
params.update(FIXED_FILTERS)
params.update({k: v for k, v in filters.items() if v not in (None, "")})
if PAGINATION_PARAM:
if PAGINATION_KIND == "offset-based":
params[PAGINATION_PARAM] = (page - 1) * (PAGE_SIZE or 1)
else:
params[PAGINATION_PARAM] = page
return f"{SEARCH_URL_BASE}?{urlencode(params, doseq=True)}"
def fetch_via_proxy(url):
proxy_url = f"{PROXY_BASE}?api_key={API_KEY}&url={quote(url, safe='')}"
resp = requests.get(proxy_url, timeout=120)
resp.raise_for_status()
return resp.text
def load_keywords(args):
if args.keywords_file:
with open(args.keywords_file, "r", encoding="utf-8") as f:
data = json.load(f)
out = []
for item in data:
if isinstance(item, str):
out.append({"keyword": item, "filters": {}})
elif isinstance(item, dict) and item.get("keyword"):
out.append({"keyword": item["keyword"], "filters": item.get("filters", {}) or {}})
return out
if args.keywords:
return [{"keyword": k.strip(), "filters": {}} for k in args.keywords.split(",") if k.strip()]
return [{"keyword": k, "filters": {}} for k in DEFAULT_KEYWORDS]
def main():
ap = argparse.ArgumentParser(description="<domain> search scraper")
ap.add_argument("--keywords", help="Comma-separated keywords (overrides DEFAULT_KEYWORDS)")
ap.add_argument("--keywords-file", help="Path to JSON file with keyword list")
ap.add_argument("--max-pages", type=int, default=DEFAULT_MAX_PAGES,
help="Pages to crawl per keyword (stops early on empty page)")
ap.add_argument("--output", default="<slug>_search.jsonl",
help="Output JSONL path")
<FILTER_FLAG_DEFINITIONS>
args = ap.parse_args()
if not API_KEY:
sys.exit("SCRAPEOPS_API_KEY not set in environment. `export SCRAPEOPS_API_KEY=...` first.")
cli_filters = <FILTER_VALUES_FROM_ARGS>
keywords = load_keywords(args)
if not keywords:
sys.exit("No keywords provided. Use --keywords or --keywords-file.")
total_rows = 0
with open(args.output, "w", encoding="utf-8") as out:
for kw_entry in keywords:
kw = kw_entry["keyword"]
filters = {**cli_filters, **kw_entry.get("filters", {})}
for page in range(1, args.max_pages + 1):
url = build_search_url(kw, filters, page)
print(f"[{kw}] page={page} → {url}", file=sys.stderr)
try:
html = fetch_via_proxy(url)
except Exception as exc:
print(f" failed: {exc}", file=sys.stderr)
continue
soup = BeautifulSoup(html, "html.parser")
data = extract_data(soup, url) or {}
products = data.get("products") or []
if not products:
print(f" empty page — stopping pagination for keyword '{kw}'", file=sys.stderr)
break
for product in products:
out.write(json.dumps({
"keyword": kw,
"page": page,
"source_url": url,
**product,
}, ensure_ascii=False) + "\n")
total_rows += 1
print(f"\n✓ Wrote {total_rows} rows to {args.output}", file=sys.stderr)
if __name__ == "__main__":
main()
<FILTER_FLAG_DEFINITIONS> — one ap.add_argument("--<name>", default=EXPOSED_FILTERS["<name>"])
per exposed filter, joined with newlines+indent.
<FILTER_VALUES_FROM_ARGS> — {"<name>": getattr(args, "<name>".replace("-", "_"))} for
each exposed filter.
<FIXED_FILTERS_DICT_LITERAL> and <EXPOSED_FILTERS_DICT> — literal Python dict syntax,
not f-string interpolation. Empty dicts are fine ({}).
<extract_data_body> — paste verbatim from scrapeops_get_code. The function signature
expected by the template is def extract_data(soup, base_url=None). If the backend
returned a different signature, normalize before pasting.
JavaScript (Cheerio) template
Same shape, with axios + cheerio.load, and manual process.argv parsing:
#!/usr/bin/env node
import fs from "node:fs";
import axios from "axios";
import * as cheerio from "cheerio";
const API_KEY = process.env.SCRAPEOPS_API_KEY ?? "";
const PROXY_BASE = "https://proxy.scrapeops.io/v1/";
const SEARCH_URL_BASE = "<SEARCH_URL_BASE>";
const KEYWORD_PARAM = "<KEYWORD_PARAM>";
const PAGINATION_PARAM = <"PAGINATION_PARAM" | null>;
const PAGINATION_KIND = "<page-number|offset-based>";
const PAGE_SIZE = <PAGE_SIZE_OR_NULL>;
const FIXED_FILTERS = <FIXED_FILTERS_OBJECT_LITERAL>;
const DEFAULT_KEYWORDS = <DEFAULT_KEYWORDS_ARRAY>;
const DEFAULT_MAX_PAGES = 1;
const EXPOSED_FILTERS = <EXPOSED_FILTERS_OBJECT>;
<extract_data_body>
function parseArgs(argv) {
const args = {};
for (let i = 2; i < argv.length; i++) {
if (argv[i].startsWith("--")) {
const key = argv[i].slice(2);
args[key] = argv[i + 1];
i++;
}
}
return args;
}
function buildSearchUrl(keyword, filters, page) {
const params = new URLSearchParams();
params.set(KEYWORD_PARAM, keyword);
for (const [k, v] of Object.entries(FIXED_FILTERS)) params.set(k, String(v));
for (const [k, v] of Object.entries(filters)) {
if (v !== undefined && v !== null && v !== "") params.set(k, String(v));
}
if (PAGINATION_PARAM) {
const value = PAGINATION_KIND === "offset-based"
? String((page - 1) * (PAGE_SIZE ?? 1))
: String(page);
params.set(PAGINATION_PARAM, value);
}
return `${SEARCH_URL_BASE}?${params.toString()}`;
}
async function fetchViaProxy(url) {
const proxyUrl = `${PROXY_BASE}?api_key=${API_KEY}&url=${encodeURIComponent(url)}`;
const resp = await axios.get(proxyUrl, { timeout: 120000 });
return resp.data;
}
function loadKeywords(args) {
if (args["keywords-file"]) {
const data = JSON.parse(fs.readFileSync(args["keywords-file"], "utf-8"));
const out = [];
for (const item of data) {
if (typeof item === "string") out.push({ keyword: item, filters: {} });
else if (item && item.keyword) out.push({ keyword: item.keyword, filters: item.filters ?? {} });
}
return out;
}
if (args.keywords) {
return args.keywords.split(",").map(k => k.trim()).filter(Boolean)
.map(k => ({ keyword: k, filters: {} }));
}
return DEFAULT_KEYWORDS.map(k => ({ keyword: k, filters: {} }));
}
async function main() {
const args = parseArgs(process.argv);
if (!API_KEY) {
console.error("SCRAPEOPS_API_KEY not set in environment.");
process.exit(1);
}
const maxPages = parseInt(args["max-pages"] ?? String(DEFAULT_MAX_PAGES), 10);
const outputPath = args.output ?? "<slug>_search.jsonl";
const cliFilters = <CLI_FILTERS_FROM_ARGS>;
const keywords = loadKeywords(args);
if (keywords.length === 0) {
console.error("No keywords provided. Use --keywords or --keywords-file.");
process.exit(1);
}
const out = fs.createWriteStream(outputPath, { flags: "w" });
let totalRows = 0;
for (const entry of keywords) {
const kw = entry.keyword;
const filters = { ...cliFilters, ...(entry.filters ?? {}) };
for (let page = 1; page <= maxPages; page++) {
const url = buildSearchUrl(kw, filters, page);
console.error(`[${kw}] page=${page} → ${url}`);
let html;
try { html = await fetchViaProxy(url); }
catch (exc) { console.error(` failed: ${exc.message ?? exc}`); continue; }
const $ = cheerio.load(html);
const data = extract_data($, url) ?? {};
const products = data.products ?? [];
if (products.length === 0) {
console.error(` empty page — stopping pagination for keyword '${kw}'`);
break;
}
for (const product of products) {
out.write(JSON.stringify({ keyword: kw, page, source_url: url, ...product }) + "\n");
totalRows++;
}
}
}
out.end();
console.error(`\n✓ Wrote ${totalRows} rows to ${outputPath}`);
}
main();
<CLI_FILTERS_FROM_ARGS> — { <name>: args["<name>"] ?? EXPOSED_FILTERS["<name>"] } for
each exposed filter. Convert kebab-case (--min-price) ↔ camelCase or stick with the
original param name — match whatever the website expects.
After Write, do NOT run any of the pre-existing Step 8 grep substitutions (those apply
only to standard mode templates).
SP-Step 9 — Sample keywords JSON (only if D3 = "JSON file at runtime only")
If the user picked the "From a JSON file" option in SP-Step 5, also Write
<slug>_keywords.sample.json:
[
"<KEYWORD_VALUE_DEFAULT>",
"TODO_keyword_2",
"TODO_keyword_3",
"TODO_keyword_4",
"TODO_keyword_5"
]
Otherwise skip this step entirely.
SP-Step 10 — Smoke test (1 keyword × 1 page) + local self-heal
The script's wiring (URL builder, filter merging, pagination, extract_data integration)
is non-trivial — a one-shot smoke test catches assembly errors before the user runs it.
Install dependencies (use the same venv pattern from Step 6 of the standard mode for
Python; npm install for JS).
Then invoke:
Python:
export SCRAPEOPS_API_KEY=<value>
python3 <slug>_search.py --keywords "<KEYWORD_VALUE_DEFAULT>" --max-pages 1 \
--output /tmp/<slug>_smoke.jsonl
wc -l /tmp/<slug>_smoke.jsonl
head -1 /tmp/<slug>_smoke.jsonl
JavaScript:
export SCRAPEOPS_API_KEY=<value>
node <slug>_search.js --keywords "<KEYWORD_VALUE_DEFAULT>" --max-pages 1 \
--output /tmp/<slug>_smoke.jsonl
wc -l /tmp/<slug>_smoke.jsonl
head -1 /tmp/<slug>_smoke.jsonl
Validate /tmp/<slug>_smoke.jsonl:
- File exists, ≥1 line.
- First line is valid JSON with
keyword, page, source_url, and at least
name + url populated (matching the schema's critical priority fields).
If broken, fetch the URL's HTML via scrapeops_fetch_html, save to
/tmp/<slug>_search_sample.html, and invoke parser-fixer:
Agent(
description: "Fix search-page extractor",
subagent_type: "parser-fixer",
prompt: "Fix the extract_data() function inside <slug>_search.<ext>.
Inputs:
- parser_path: <slug>_search.<ext> (the extract_data function inside)
- html_path: /tmp/<slug>_search_sample.html
- language: <python|javascript>
- task: Smoke test failed — extract_data($, url) returned 0 products OR products are
missing required card fields (name, url, price, image). Expected at least name + url
populated per product card.
Run command (Python): python3 <slug>_search.py --keywords \"<KEYWORD_VALUE_DEFAULT>\" --max-pages 1 --output /tmp/<slug>_smoke.jsonl
Run command (JS): node <slug>_search.js --keywords \"<KEYWORD_VALUE_DEFAULT>\" --max-pages 1 --output /tmp/<slug>_smoke.jsonl
CRITICAL:
- Do NOT touch the URL builder (build_search_url / buildSearchUrl), the proxy fetch
function, the argparse/yargs CLI wiring, or the discovery-time constants block at the
top of the file. Only edit selectors inside extract_data.
- Do NOT touch the SCRAPEOPS_API_KEY env-var read.
- Verify the smoke command produces ≥1 valid JSON line before finishing."
)
After the agent returns, re-run the smoke command and re-validate. If still broken,
offer the user: regenerate / manual fix / stop.
SP-Step 11 — Generate <slug>_README.md
Use Write to create a README documenting:
- What the script does — domain, search scraper, output JSONL.
- Prerequisites — Python 3.9+ / Node 18+. Show venv setup for Python (per Step 6 of
standard mode).
- API key —
export SCRAPEOPS_API_KEY=<your-key>. State that the script reads from
env at runtime; rotating the key needs no edits.
- Three keyword input modes with one-line example each:
python3 <slug>_search.py
python3 <slug>_search.py --keywords "shirts,jeans,jackets"
python3 <slug>_search.py --keywords-file <slug>_keywords.json
- JSON file format — show both supported shapes (lista de strings vs lista de
objetos
{keyword, filters}).
- Filter flags — table listing each exposed
--<filter> flag, its current default,
and a one-line semantics note.
- Pagination —
--max-pages N. Mention the early-stop behavior on empty pages.
- Output shape — one JSONL line per product card, with
keyword, page,
source_url annotations + the schema fields.
- Customizing — point at the discovery-time constants block at the top of the
script, and at
extract_data for selector edits.
- Troubleshooting — same items as the standard README plus "empty results = either
selector broken or all keywords return zero hits".
Keep under 120 lines.
SP-Step 12 — Final summary
Show the file list, a one-line note on the discovered URL structure, and the run command.
Do NOT execute anything else.
Search scraper generated successfully (Search Page Mode).
Files saved:
<slug>_search.<ext>
<slug>_search_schema.json
<slug>_README.md
<slug>_keywords.sample.json # only if "JSON file" mode was chosen
Discovery:
Keyword param: <KEYWORD_PARAM>
Pagination: <PAGINATION_PARAM> (<PAGINATION_KIND>)
Exposed filters: <list of --flags>
Fixed filters: <list of fixed=value pairs>
Before running:
export SCRAPEOPS_API_KEY=<your-key>
Run examples:
python3 <slug>_search.py
python3 <slug>_search.py --keywords "term1,term2" --max-pages 3
python3 <slug>_search.py --keywords-file <slug>_keywords.sample.json
Search Page Mode notes
- Output is list-only. Search Page Mode never follows into individual product detail
pages. For that, use
/generate-crawler-scraper — it's the right tool when the user
needs full product objects per result.
include_api_key is irrelevant in Search Page Mode (same as Framework Mode).
parser_only: true returns just extract_data; the API key is read from env at
runtime by the local fetch function.- Empty-page early-stop is built into the loop. If a page returns zero products, the
script stops paginating that keyword and moves to the next. So
--max-pages 100 won't
burn 100 fetches if results run out at page 7.
parser-fixer for self-heal must be told NOT to touch the URL builder, fetch
function, CLI wiring, or discovery constants — only extract_data selectors. Hard-code
that in the prompt, like Framework Mode does for BS4/axios.- Backend
target_library is intermediate for the same reason as Framework Mode:
the backend can produce BS4 / Cheerio extractors today, and we wrap them locally.
