Menu
从网页提取和处理数据，使用CSS选择器、XPath智能解析，支持限速和错误处理。
Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.
Based on SOC occupation classification
自动分析B站视频内容，下载视频并拆解成帧图片，使用AI分析并生成详细的专题文档或实操教程。
设计RESTful API并生成OpenAPI/Swagger规范文档，遵循行业最佳实践。包括端点命名、请求/响应模式和错误处理模式。
为GitHub Actions、GitLab CI、Azure DevOps和Jenkins生成CI/CD流水线，包含构建、测试、部署阶段、缓存和密钥管理。
全面的代码审查技能，分析代码质量、识别问题、安全漏洞，并提供带严重性评级的改进建议。
设计和优化数据库模式，支持PostgreSQL、MySQL、SQLite和MongoDB。包括ER建模、规范化、索引优化和迁移脚本生成。
创建和管理多容器应用的Docker Compose配置，包含生产级设置、健康检查和网络配置。
| name | web-scraper |
| description | 从网页提取和处理数据，使用CSS选择器、XPath智能解析，支持限速和错误处理。 |
| metadata | {"short-description":"从网页提取数据"} |
Extract and process data from web pages with intelligent parsing capabilities.
/scrape commandYou are a web scraping expert that extracts data efficiently and ethically.
import puppeteer from 'puppeteer';
interface Product {
name: string;
price: number;
rating: number;
url: string;
}
async function scrapeProducts(url: string): Promise<Product[]> {
const browser = await puppeteer.launch({ headless: 'new' });
const page = await browser.newPage();
// Set user agent to avoid detection
await page.setUserAgent('Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36');
await page.goto(url, { waitUntil: 'networkidle2' });
// Wait for products to load
await page.waitForSelector('.product-card');
const products = await page.evaluate(() => {
const items = document.querySelectorAll('.product-card');
return Array.from(items).map(item => ({
name: item.querySelector('.product-name')?.textContent?.trim() ?? '',
price: parseFloat(item.querySelector('.price')?.textContent?.replace('$', '') ?? '0'),
rating: parseFloat(item.querySelector('.rating')?.getAttribute('data-rating') ?? '0'),
url: item.querySelector('a')?.href ?? '',
}));
});
await browser.close();
return products;
}
import axios from 'axios';
import * as cheerio from 'cheerio';
async function parseArticle(url: string) {
const { data } = await axios.get(url, {
headers: { 'User-Agent': 'Mozilla/5.0' }
});
const $ = cheerio.load(data);
return {
title: $('h1.article-title').text().trim(),
author: $('span.author-name').text().trim(),
date: $('time').attr('datetime'),
content: $('article.content p').map((_, el) => $(el).text()).get().join('\n\n'),
tags: $('a.tag').map((_, el) => $(el).text()).get(),
};
}
class RateLimiter {
private queue: (() => Promise<void>)[] = [];
private processing = false;
constructor(private delayMs: number = 1000) {}
async add<T>(fn: () => Promise<T>): Promise<T> {
return new Promise((resolve, reject) => {
this.queue.push(async () => {
try {
resolve(await fn());
} catch (e) {
reject(e);
}
});
this.process();
});
}
private async process() {
if (this.processing) return;
this.processing = true;
while (this.queue.length > 0) {
const fn = this.queue.shift()!;
await fn();
await new Promise(r => setTimeout(r, this.delayMs));
}
this.processing = false;
}
}
// Usage
const limiter = new RateLimiter(2000); // 2 seconds between requests
const results = await Promise.all(
urls.map(url => limiter.add(() => scrapeProducts(url)))
);
web-scraping, data-extraction, parsing, automation, html