| name | data-scraper-agent |
| description | Automated data collection from structured sources: CVE databases, threat intelligence feeds, exploit databases, and security advisories. Transform unstructured web data into structured knowledge units. |
| origin | openclaw |
| version | 0.2.0.2 |
| compatibility | ["openclaw","claude-code","cursor","windsurf"] |
| allowed-tools | ["Bash","Read","Write","Edit","WebSearch","WebFetch","Agent"] |
| defense_triple_required | false |
| metadata | {"domain":"research","tool_count":0,"guide_count":5,"last_reviewed":"2026-07-26"} |
Data Scraper Agent
Summary
Transform unstructured web data into structured knowledge units.
Domain: research
Skill Identity
| Attribute | Value |
|---|
| Domain | Intelligence Gathering |
| Skill ID | data-scraper-agent |
| Version | 1.0.0 |
| Hacker Laws | Law 3 (Intelligence Over Force), Law 9 (Systematic Over Random) |
| Related Skills | deep-research, osint, knowledge-ops |
Purpose
Automated data collection from structured sources: CVE databases, threat intelligence feeds, exploit databases, and security advisories. Transform unstructured web data into structured knowledge units.
Core Capabilities
- CVE Database Scraping: NVD, MITRE, vendor-specific databases
- Exploit Database Collection: Exploit-DB, GitHub security advisories
- Threat Intel Feeds: Parse and filter IOC feeds
- Structured Data Extraction: JSON/XML APIs, HTML scraping
Use Cases
- Vulnerability Research: Collect CVEs for specific products
- Exploit Availability: Check if public exploits exist for CVEs
- Threat Intelligence: Aggregate IOCs for a campaign
- Vendor Advisories: Monitor vendor security bulletins
Tools
- BeautifulSoup: HTML parsing
- Scrapy: Web scraping framework
- requests: HTTP client
- jq: JSON filtering
Scraping Strategy
- API-first: Prefer JSON APIs (NVD 2.0, GitHub Advisory) over HTML scraping — schemas are stable, rate limits documented, content validated.
- Pagination discipline: Always honor
Link: rel="next" headers or cursor-based tokens; never scrape past published page limits.
- Selective extraction: Pull only the fields you need (
cve.id, descriptions[*].value, metrics.cvssMetricV31) instead of dumping full records.
