scrape-batch

# Batch scrape multiple URLs ## Trigger Scraping multiple web pages at once — comparing products, collecting articles, or gathering data from several sources. ## Workflow 1. Verify anakin-cli is authenticated by running `anakin status`. 2. Create the output directory if it doesn't exist: ```bash mkdir -p .anakin ``` 3. Run the batch scrape: ```bash anakin scrape-batch "$ARGUMENTS" -o .anakin/batch-results.json ``` 4. If the user provides more than 10 URLs, split into batches of 10: ```bash anakin scrape-batch "<url1>" ... "<url10>" -o .anakin/batch-1.json anakin scrape-batch "<url11>" ... "<url20>" -o .anakin/batch-2.json ``` 5. Read the output and present a summary of all scraped pages. ## Commands ```bash # Batch scrape multiple URLs anakin scrape-batch "<url1>" "<url2>" "<url3>" -o .anakin/batch-results.json # With headless browser for JS-heavy sites anakin scrape-batch "<url1>" "<url2>" --browser -o .anakin/batch-spa.json # Geo-targeted batch anakin scrape-batch "<url1>" "<url2>" --country de -o .anakin/batch-de.json # Extended timeout for many pages anakin scrape-batch "<url1>" "<url2>" "<url3>" --timeout 300 -o .anakin/batch.json ``` ### Options - `--browser` — Use headless browser (for JS-rendered pages) - `--country <code>` — Country code for geo-targeting (default: us) - `--timeout <seconds>` — Max wait time (default: 180) - `-o, --output <path>` — Save to file ### Limit Max 10 URLs per request. For more URLs, split into multiple batch calls. ## Guardrails - Maximum 10 URLs per `scrape-batch` command — split larger lists into batches. - Always quote every URL to prevent shell interpretation. - If individual URLs fail within a batch, report which ones failed and suggest retrying with `anakin scrape` using `--browser`. - Always use `-o` to save output to a file rather than flooding the terminal. ## Output - Combined results from all URLs in a single JSON file - Summary listing each URL with its scrape status