data-scraper-agent

name	data-scraper-agent
description	Automated data collection from structured sources: CVE databases, threat intelligence feeds, exploit databases, and security advisories. Transform unstructured web data into structured knowledge units.
origin	openclaw
version	0.2.0.2
compatibility	["openclaw","claude-code","cursor","windsurf"]
allowed-tools	["Bash","Read","Write","Edit","WebSearch","WebFetch","Agent"]
defense_triple_required	false
metadata	{"domain":"research","tool_count":0,"guide_count":5,"last_reviewed":"2026-07-26"}

Data Scraper Agent

Summary

Transform unstructured web data into structured knowledge units.

Domain: research

Skill Identity

Attribute	Value
Domain	Intelligence Gathering
Skill ID	data-scraper-agent
Version	1.0.0
Hacker Laws	Law 3 (Intelligence Over Force), Law 9 (Systematic Over Random)
Related Skills	deep-research, osint, knowledge-ops

Purpose

Automated data collection from structured sources: CVE databases, threat intelligence feeds, exploit databases, and security advisories. Transform unstructured web data into structured knowledge units.

Core Capabilities

1. CVE Database Scraping: NVD, MITRE, vendor-specific databases
2. Exploit Database Collection: Exploit-DB, GitHub security advisories
3. Threat Intel Feeds: Parse and filter IOC feeds
4. Structured Data Extraction: JSON/XML APIs, HTML scraping

Use Cases

• Vulnerability Research: Collect CVEs for specific products
• Exploit Availability: Check if public exploits exist for CVEs
• Threat Intelligence: Aggregate IOCs for a campaign
• Vendor Advisories: Monitor vendor security bulletins

Tools

• BeautifulSoup: HTML parsing
• Scrapy: Web scraping framework
• requests: HTTP client
• jq: JSON filtering

Scraping Strategy

• API-first: Prefer JSON APIs (NVD 2.0, GitHub Advisory) over HTML scraping — schemas are stable, rate limits documented, content validated.
• Pagination discipline: Always honor Link: rel="next" headers or cursor-based tokens; never scrape past published page limits.
• Selective extraction: Pull only the fields you need (cve.id, descriptions[*].value, metrics.cvssMetricV31) instead of dumping full records.