메뉴
Web crawling and scraping tool with LLM-optimized output. 网页爬虫爬取工具 | Web crawler, web scraper, spider. DuckDuckGo search, site crawling, dynamic page scraping. 智能搜索爬取 | Free, no API key required.
Codex 또는 Claude로 설치 이 Prompt를 복사해 Codex, Claude 또는 다른 어시스턴트에 붙여 넣으면 Skill 페이지를 검토하고 설치를 진행할 수 있습니다.
SOC 직업 분류 기준
| name | crawl4ai-skill |
| description | Web crawling and scraping tool with LLM-optimized output. 网页爬虫爬取工具 | Web crawler, web scraper, spider. DuckDuckGo search, site crawling, dynamic page scraping. 智能搜索爬取 | Free, no API key required. |
| version | 1.1.0 |
| author | Lancelin |
| license | MIT-0 |
| repository | https://github.com/lancelin111/crawl4ai-skill |
| homepage | https://github.com/lancelin111/crawl4ai-skill |
| pypi | crawl4ai-skill |
| issues | https://github.com/lancelin111/crawl4ai-skill/issues |
| tags | ["web-scraping","web-crawling","crawler","scraper","spider","search","markdown","llm","token-optimization","duckduckgo","crawl","scrape","爬虫","爬取","网页抓取"] |
| requires | {"bins":["crawl4ai-skill"]} |
Web Crawling 网页爬虫 | Web Scraping 网页爬取 | LLM 优化输出
智能网页爬虫和爬取工具，支持搜索、全站爬取、动态页面抓取。Free web crawler and scraper with LLM-optimized Markdown output.
pip install crawl4ai-skill
# Search the web with DuckDuckGo
crawl4ai-skill search "python web scraping"
# Scrape a single web page
crawl4ai-skill crawl https://example.com
# Crawl entire website / spider
crawl4ai-skill crawl-site https://docs.python.org --max-pages 50
# Crawl documentation site with spider
crawl4ai-skill crawl-site https://docs.fastapi.com --max-pages 100
爬虫效果 | Crawler Output:
# Search and scrape top results
crawl4ai-skill search-and-crawl "Vue 3 best practices" --crawl-top 3
JavaScript 渲染的页面爬取（雪球、知乎等）：
# Scrape JavaScript-heavy pages
crawl4ai-skill crawl https://xueqiu.com/S/BIDU --wait-until networkidle --delay 2
| 命令 Command | 说明 Description |
|---|---|
search <query> | Web search 网页搜索 |
crawl <url> | Web scraping 单页爬取 |
crawl-site <url> | Web crawling 全站爬虫 |
search-and-crawl <query> | Search + scrape 搜索并爬取 |
# Web Search 搜索
--num-results 10 # Number of results
# Web Scraping 爬取
--format fit_markdown # Output format
--output result.md # Output file
--wait-until networkidle # Wait strategy for dynamic pages
--delay 2 # Additional wait time (seconds)
--wait-for ".selector" # Wait for specific element
# Web Crawling 爬虫
--max-pages 100 # Max pages to crawl
--max-depth 3 # Max crawl depth
智能提取，节省 80% Token。Smart extraction, save 80% tokens.
crawl4ai-skill crawl https://example.com --format fit_markdown
保留完整结构。Preserve full structure.
crawl4ai-skill crawl https://example.com --format raw_markdown
✅ 免费爬虫 Free Crawler - 无需 API key，开箱即用
✅ 智能爬取 Smart Scraper - 自动去噪，提取核心内容
✅ 全站爬虫 Site Crawler - 支持 sitemap，递归爬取
✅ 动态爬取 Dynamic Scraping - JavaScript 渲染页面支持
✅ 搜索集成 Search Integration - DuckDuckGo 搜索内置