anakinscraper

# AnakinScraper Skill Scrape any website and get back clean markdown or structured JSON data. Uses an anti-detect browser (Camoufox) to handle JavaScript-heavy sites and anti-bot protection. ## Setup 1. Start AnakinScraper (if not already running): ```bash git clone https://github.com/Anakin-Inc/anakinscraper-oss.git cd anakinscraper-oss && make up ``` 2. Copy the skill into your OpenClaw workspace: ```bash cp -r openclaw-skill ~/.openclaw/workspace/skills/anakinscraper ``` The skill calls the AnakinScraper REST API at `http://localhost:8080` directly. No additional build step required. ## Available Tools ### anakinscraper_scrape Scrape a single URL synchronously. Returns markdown, cleaned HTML, and optionally structured JSON. Use this when the user asks to: - "Scrape this website" - "Get the content from this URL" - "Turn this page into markdown" - "What's on this webpage?" Parameters: - `url` (required) — The URL to scrape - `useBrowser` (optional, default false) — Force browser rendering for JavaScript-heavy sites - `generateJson` (optional, default false) — Extract structured JSON data using AI ### anakinscraper_extract_json Scrape a URL and extract structured data as JSON. Best for product pages, articles, listings. Use this when the user asks to: - "Extract the product data from this page" - "Get structured data from this URL" - "Parse this listing page into JSON" - "What products are on this page?" Parameters: - `url` (required) — The URL to extract data from ### anakinscraper_batch_scrape Scrape multiple URLs at once (up to 10). Returns results for all URLs. Use this when the user asks to: - "Scrape these 5 URLs" - "Get content from all these pages" - "Batch scrape this list" Parameters: - `urls` (required) — Array of URLs to scrape (max 10) - `useBrowser` (optional) — Force browser rendering - `generateJson` (optional) — Extract structured JSON from each page ### anakinscraper_scrape_async Submit a scrape job asynchronously. Returns a job ID for later polling. Use for pages that take longer than 30 seconds. Parameters: - `url` (required) — The URL to scrape - `useBrowser` (optional) — Force browser rendering - `generateJson` (optional) — Extract structured JSON ### anakinscraper_get_job Check the status of an async scrape job. Poll until status is "completed" or "failed". Parameters: - `jobId` (required) — The job ID returned by anakinscraper_scrape_async ## Usage Guidelines 1. **Default to `anakinscraper_scrape`** for most requests — it's synchronous and returns results immediately. 2. **Use `anakinscraper_extract_json`** when the user wants structured data (products, articles, listings). 3. **Use `anakinscraper_batch_scrape`** when scraping multiple URLs. 4. **Use `anakinscraper_scrape_async` + `anakinscraper_get_job`** only for pages you know will be slow (complex SPAs, heavy anti-bot sites). 5. **Set `useBrowser: true`** for JavaScript-heavy sites, SPAs, or sites with anti-bot protection. 6. Present the **markdown** field for readable content, **generatedJson.data** for structured data.