菜单
Web scraping toolkit for extracting content from web pages. Fetch HTML, extract links, parse text content, and download page resources. Use when the user needs to scrape websites, extract data from web pages, gather links, or harvest text content.
用 Codex 或 Claude 帮你安装 复制这段 Prompt，粘贴到 Codex、Claude 或其他助手里，让它检查 Skill 页面并帮你完成安装。
基于 SOC 职业分类
File utility toolkit for searching, analyzing, and comparing files. Find files by pattern/size/date, count lines, get file statistics, and compare file contents. Use when working with file discovery, analysis, or comparison tasks.
JSON processing toolkit for validating, formatting, querying, and comparing JSON data. Use when working with JSON files, API responses, configuration files, or any structured JSON data that needs parsing, validation, transformation, or comparison.
Network connectivity testing toolkit for checking host reachability, port availability, and DNS resolution. Use when diagnosing network issues, verifying service availability, testing connectivity to servers, or troubleshooting DNS problems.
Perform network reconnaissance including host discovery, port scanning, and service enumeration. Use when asked to "scan a network", "find hosts", "discover devices", "enumerate services", "recon a subnet", or "what's on my network".
This skill should be used when the user asks to "run pentest commands", "scan with nmap", "use metasploit exploits", "crack passwords with hydra or john", "scan web vulnerabilities with nikto", "enumerate networks", or needs essential penetration testing command references.
Guide for creating effective skills. This skill should be used when users want to create a new skill (or update an existing skill) that extends Codex's capabilities with specialized knowledge, workflows, or tool integrations.
| name | web-scraper |
| description | Web scraping toolkit for extracting content from web pages. Fetch HTML, extract links, parse text content, and download page resources. Use when the user needs to scrape websites, extract data from web pages, gather links, or harvest text content. |
| license | MIT |
| compatibility | Requires requests and beautifulsoup4 packages |
| metadata | {"author":"dspy-skills","version":"1.0"} |
A toolkit for extracting content from web pages using Python.
Activate this skill when the user needs to:
This skill requires external packages:
pip install requests beautifulsoup4
Always run scripts with --help first to see all available options.
| Script | Purpose |
|---|---|
fetch_page.py | Download HTML content from a URL |
extract_links.py | Extract all links from a page |
extract_text.py | Extract readable text from HTML |
Task → What do you need?
│
├─ Raw HTML content?
│ └─ Use: fetch_page.py <url>
│
├─ List of links on a page?
│ └─ Use: extract_links.py <url>
│
└─ Text content (no HTML tags)?
└─ Use: extract_text.py <url>
Fetch page HTML:
python scripts/fetch_page.py https://example.com
python scripts/fetch_page.py https://example.com --output page.html
Extract all links:
python scripts/extract_links.py https://example.com
python scripts/extract_links.py https://example.com --absolute --filter "\.pdf$"
Extract text content:
python scripts/extract_text.py https://example.com
python scripts/extract_text.py https://example.com --paragraphs
--user-agent flag.--timeout value.--encoding flag if text appears garbled.See references/selectors.md for CSS selector syntax reference.