data-scraper

name	data-scraper
description	Extract data from websites, APIs, and files. Supports structured extraction, pagination, and anti-bot bypass.
triggers	["scrape website","爬虫","extract data","web scraping"]

Data Scraper

Extract data from websites, APIs, and files.

Features

• 🌐 Web Scraping — Extract from any website
• 📡 API Scraping — GET/POST requests
• 📄 File Parsing — CSV, Excel, PDF, JSON
• 🔄 Pagination — Auto-handle paginated content
• 🛡️ Anti-bot — Bypass protection
• 📊 Data Cleaning — Clean extracted data

Usage

Scrape Website

# Simple scrape
hermes scrape "https://example.com/products" \
  --selector ".product-item" \
  --fields name,price,image

# With pagination
hermes scrape "https://example.com/products" \
  --selector ".product" \
  --fields name,price \
  --pages 10

Extract from API

hermes scrape-api "https://api.example.com/data" \
  --headers "Authorization: Bearer TOKEN" \
  --output data.json

File Parsing

# Parse CSV
hermes scrape file data.csv --format json

# Parse Excel
hermes scrape file report.xlsx --sheet "Sales"

# Parse PDF
hermes scrape file document.pdf --extract text

Configuration

# scraper-config.yml
scraping:
  delay: 1000  # ms between requests
  retries: 3
  timeout: 30
  
anti_bot:
  rotate_user_agent: true
  use_proxy: false
  bypass_cloudflare: true
  
output:
  format: json
  encoding: utf-8

Extractors

CSS Selector

hermes scrape --selector ".product .title" --attr text
hermes scrape --selector "img.product" --attr src

JSON Path

hermes scrape-json --path "$.data[*].name" --file data.json

XPath

hermes scrape --xpath "//div[@class='product']/span" --file page.html

Pitfalls

1. Legal Issues — Check robots.txt and terms of service
2. Rate Limits — Don't overload servers
3. Data Quality — Verify extracted data accuracy
4. Anti-bot — Some sites block scrapers