web-scraping

# Scrape — General-Purpose Web Scraping & Data Extraction ## Setup Read your credentials from ~/.gooseworks/credentials.json: ```bash export GOOSEWORKS_API_KEY=$(python3 -c "import json;print(json.load(open('$HOME/.gooseworks/credentials.json'))['api_key'])") export GOOSEWORKS_API_BASE=$(python3 -c "import json;print(json.load(open('$HOME/.gooseworks/credentials.json')).get('api_base','https://api.gooseworks.ai'))") ``` If ~/.gooseworks/credentials.json does not exist, tell the user to run: `npx gooseworks login` All endpoints use Bearer auth: `-H "Authorization: Bearer $GOOSEWORKS_API_KEY"` Scrape websites, extract structured data, and automate browser interactions. Pick the best API for the task — or combine several for comprehensive extraction. ## 1. Scrapegraph — AI-Powered Scraping with Natural Language Best for: Extracting data using plain English prompts, converting pages to markdown, crawling with AI extraction, and search-based scraping. **AI-powered extraction** (describe what you want in natural language): ```bash curl -s -X POST $GOOSEWORKS_API_BASE/v1/proxy/orthogonal/run \ -H "Authorization: Bearer $GOOSEWORKS_API_KEY" \ -H "Content-Type: application/json" \ -d '{"api":"scrapegraph","path":"/v1/smartscraper"}' "website_url": "https://example.com/products", "user_prompt": "Extract all product names, prices, descriptions, and image URLs" }' ``` **With output schema** (enforce structure): ```bash curl -s -X POST $GOOSEWORKS_API_BASE/v1/proxy/orthogonal/run \ -H "Authorization: Bearer $GOOSEWORKS_API_KEY" \ -H "Content-Type: application/json" \ -d '{"api":"scrapegraph","path":"/v1/smartscraper"}' "website_url": "https://example.com/products", "user_prompt": "Extract all products", "output_schema": { "properties": { "products": { "type": "array", "items": { "type": "object", "properties": { "name": {"type": "string"}, "price": {"type": "number"}, "description": {"type": "string"} } } } } } }' ``` **Search + scrape** (search the web and extract from results): ```bash curl -s -X POST $GOOSEWORKS_API_BASE/v1/proxy/orthogonal/run \ -H "Authorization: Bearer $GOOSEWORKS_API_KEY" \ -H "Content-Type: application/json" \ -d '{"api":"scrapegraph","path":"/v1/searchscraper","body":{"user_prompt":"Find the latest iPhone prices from major retailers"}}' # Poll for results: curl -s -X POST $GOOSEWORKS_API_BASE/v1/proxy/orthogonal/run \ -H "Authorization: Bearer $GOOSEWORKS_API_KEY" \ -H "Content-Type: application/json" \ -d '{"api":"scrapegraph","path":"/v1/searchscraper/{request_id}"}' ``` **Convert page to markdown:** ```bash curl -s -X POST $GOOSEWORKS_API_BASE/v1/proxy/orthogonal/run \ -H "Authorization: Bearer $GOOSEWORKS_API_KEY" \ -H "Content-Type: application/json" \ -d '{"api":"scrapegraph","path":"/v1/markdownify","body":{"website_url":"https://example.com/article"}}' ``` **Crawl with AI extraction:** ```bash curl -s -X POST $GOOSEWORKS_API_BASE/v1/proxy/orthogonal/run \ -H "Authorization: Bearer $GOOSEWORKS_API_KEY" \ -H "Content-Type: application/json" \ -d '{"api":"scrapegraph","path":"/v1/crawl"}' "url": "https://docs.example.com", "prompt": "Extract all API endpoints and their descriptions", "max_pages": 20 }' # Poll for results: curl -s -X POST $GOOSEWORKS_API_BASE/v1/proxy/orthogonal/run \ -H "Authorization: Bearer $GOOSEWORKS_API_KEY" \ -H "Content-Type: application/json" \ -d '{"api":"scrapegraph","path":"/v1/crawl/{task_id}"}' ``` **Raw HTML scrape:** ```bash curl -s -X POST $GOOSEWORKS_API_BASE/v1/proxy/orthogonal/run \ -H "Authorization: Bearer $GOOSEWORKS_API_KEY" \ -H "Content-Type: application/json" \ -d '{"api":"scrapegraph","path":"/v1/scrape","body":{"website_url":"https://example.com"}}' ``` **Get sitemap:** ```bash curl -s -X POST $GOOSEWORKS_API_BASE/v1/proxy/orthogonal/run \ -H "Authorization: Bearer $GOOSEWORKS_API_KEY" \ -H "Content-Type: application/json" \ -d '{"api":"scrapegraph","path":"/v1/sitemap","body":{"website_url":"https://example.com"}}' ``` Key parameters: `stealth` (bypass bot protection, +4 credits), `total_pages` (paginate up to 100), `number_of_scrolls` (infinite scroll pages), `render_heavy_js` (React/Vue/Angular SPAs), `steps` (interaction steps before extraction). ## 2. Olostep — Scalable Scraping & Batch Jobs Best for: High-volume scraping, batch processing, site crawling, URL discovery, and AI-powered answers from pages. **Scrape a single page:** ```bash curl -s -X POST $GOOSEWORKS_API_BASE/v1/proxy/orthogonal/run \ -H "Authorization: Bearer $GOOSEWORKS_API_KEY" \ -H "Content-Type: application/json" \ -d '{"api":"olostep","path":"/v1/scrapes","body":{"url_to_scrape":"https://example.com/page"}}' ``` **AI-powered answer from the web:** ```bash curl -s -X POST $GOOSEWORKS_API_BASE/v1/proxy/orthogonal/run \ -H "Authorization: Bearer $GOOSEWORKS_API_KEY" \ -H "Content-Type: application/json" \ -d '{"api":"olostep","path":"/v1/answers","body":{"task":"What is the pricing for Stripe?"}}' ``` **Discover all URLs on a site:** ```bash curl -s -X POST $GOOSEWORKS_API_BASE/v1/proxy/orthogonal/run \ -H "Authorization: Bearer $GOOSEWORKS_API_KEY" \ -H "Content-Type: application/json" \ -d '{"api":"olostep","path":"/v1/maps","body":{"url":"https://example.com","search_query":"pricing"}}' ``` **Crawl a site** (async): ```bash # Step 1: Start crawl curl -s -X POST $GOOSEWORKS_API_BASE/v1/proxy/orthogonal/run \ -H "Authorization: Bearer $GOOSEWORKS_API_KEY" \ -H "Content-Type: application/json" \ -d '{"api":"olostep","path":"/v1/crawls"}' "start_url": "https://docs.example.com", "max_pages": 100, "include_urls": ["/docs/**"] }' # Step 2: Check status curl -s -X POST $GOOSEWORKS_API_BASE/v1/proxy/orthogonal/run \ -H "Authorization: Bearer $GOOSEWORKS_API_KEY" \ -H "Content-Type: application/json" \ -d '{"api":"olostep","path":"/v1/crawls/{crawl_id}"}' # Step 3: Get pages curl -s -X POST $GOOSEWORKS_API_BASE/v1/proxy/orthogonal/run \ -H "Authorization: Bearer $GOOSEWORKS_API_KEY" \ -H "Content-Type: application/json" \ -d '{"api":"olostep","path":"/v1/crawls/{crawl_id}/pages"}' # Step 4: Retrieve content curl -s -X POST $GOOSEWORKS_API_BASE/v1/proxy/orthogonal/run \ -H "Authorization: Bearer $GOOSEWORKS_API_KEY" \ -H "Content-Type: application/json" \ -d '{"api":"olostep","path":"/v1/retrieve","body":{"retrieve_id":"RETRIEVE_ID"}}' ``` **Batch scrape** (process many URLs at once): ```bash curl -s -X POST $GOOSEWORKS_API_BASE/v1/proxy/orthogonal/run \ -H "Authorization: Bearer $GOOSEWORKS_API_KEY" \ -H "Content-Type: application/json" \ -d '{"api":"olostep","path":"/v1/batches"}' "items": [ {"url_to_scrape": "https://example.com/page1"}, {"url_to_scrape": "https://example.com/page2"}, {"url_to_scrape": "https://example.com/page3"} ] }' # Check status: curl -s -X POST $GOOSEWORKS_API_BASE/v1/proxy/orthogonal/run \ -H "Authorization: Bearer $GOOSEWORKS_API_KEY" \ -H "Content-Type: application/json" \ -d '{"api":"olostep","path":"/v1/batches/{batch_id}"}' # Get items: curl -s -X POST $GOOSEWORKS_API_BASE/v1/proxy/orthogonal/run \ -H "Authorization: Bearer $GOOSEWORKS_API_KEY" \ -H "Content-Type: application/json" \ -d '{"api":"olostep","path":"/v1/batches/{batch_id}/items"}' ``` Key parameters: `formats` (markdown/html/text), `country` (US, CA, IT, IN, GB, JP, etc.), `actions` (page interactions before scraping), `wait_before_scraping`, `remove_css_selectors`, `llm_extract`. ## 3. Riveter — Structured Extraction with Defined Schemas Best for: Extracting data into a consistent, predefined structure. Define input URLs and output fields with prompts. **Simple page scrape:** ```bash curl -s -X POST $GOOSEWORKS_API_BASE/v1/proxy/orthogonal/run \ -H "Authorization: Bearer $GOOSEWORKS_API_KEY" \ -H "Content-Type: application/json" \ -d '{"api":"riveter","path":"/v1/scrape","body":{"url":"https://example.com/article"}}' ``` **Structured extraction** (define your output schema): ```bash curl -s -X POST $GOOSEWORKS_API_BASE/v1/proxy/orthogonal/run \ -H "Authorization: Bearer $GOOSEWORKS_API_KEY" \ -H "Content-Type: application/json" \ -d '{"api":"riveter","path":"/v1/run"}' "input": { "urls": ["https://example.com/products"] }, "output": { "name": {"prompt": "Product name", "contexts": ["urls"]}, "price": {"prompt": "Product price", "contexts": ["urls"], "format": "number"}, "description": {"prompt": "Product description", "contexts": ["urls"]} } }' # Check status: