القائمة
Web data extraction using 55+ Apify Actors for AI-driven scraping. Supports Instagram, Facebook, TikTok, YouTube, Google, and more. Auto-selects best Actor for the task. Structured output in JSON/CSV with rate limiting and ethical scraping guidelines.
التثبيت باستخدام Codex أو Claude انسخ هذا Prompt والصقه في Codex أو Claude أو مساعد آخر ليراجع صفحة Skill ويثبّتها لك.
استنادا إلى تصنيف SOC المهني
| name | allbeingsfuture/skills@apify-scraper |
| description | Web data extraction using 55+ Apify Actors for AI-driven scraping. Supports Instagram, Facebook, TikTok, YouTube, Google, and more. Auto-selects best Actor for the task. Structured output in JSON/CSV with rate limiting and ethical scraping guidelines. |
| license | MIT |
| metadata | {"author":"allbeingsfuture","version":"1.0.0"} |
| 配置项 | 说明 |
|---|---|
APIFY_TOKEN | Apify API Token，在 https://console.apify.com/account/integrations 获取 |
将 Token 添加到 .env 文件：
APIFY_TOKEN=apify_api_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
| 依赖 | 用途 | 安装方式 |
|---|---|---|
httpx | HTTP API 调用 | pip install httpx |
| 依赖 | 用途 | 安装方式 |
|---|---|---|
apify-client | Apify Python SDK | pip install apify-client |
pandas | 数据处理与导出 | pip install pandas |
curl -s "https://api.apify.com/v2/user/me?token=$APIFY_TOKEN" | python -m json.tool
Apify 平台上有数千个 Actor（即预构建的爬虫/自动化程序）。本技能聚焦 55+ 个经过验证的、面向 AI 数据提取优化的 Actor。
| 平台 | Actor | Actor ID | 主要功能 |
|---|---|---|---|
| Profile Scraper | apify/instagram-profile-scraper | 用户资料、帖子、粉丝数 | |
| Hashtag Scraper | apify/instagram-hashtag-scraper | 标签下的帖子 | |
| Comment Scraper | apify/instagram-comment-scraper | 帖子评论 | |
| TikTok | Scraper | clockworks/free-tiktok-scraper | 视频、用户、标签 |
| YouTube | Scraper | bernardo/youtube-scraper | 视频信息、评论 |
| YouTube | Channel Scraper | streamers/youtube-channel-scraper | 频道数据 |
| Posts Scraper | apify/facebook-posts-scraper | 页面帖子 | |
| Comments Scraper | apify/facebook-comments-scraper | 帖子评论 | |
| Twitter/X | Scraper | apidojo/tweet-scraper | 推文搜索 |
| Profile Scraper | anchor/linkedin-profile-scraper | 用户资料 |
| 平台 | Actor | Actor ID | 主要功能 |
|---|---|---|---|
| Search Results | apify/google-search-scraper | SERP 结果 | |
| Maps | compass/crawler-google-places | 商家信息、评价 | |
| Trends | emastra/google-trends-scraper | 搜索趋势 | |
| News | lhotanova/google-news-scraper | 新闻搜索 | |
| Shopping | epctex/google-shopping-scraper | 商品价格 | |
| Bing | Search | nicefellow/bing-search-scraper | Bing 搜索结果 |
| 平台 | Actor | Actor ID | 主要功能 |
|---|---|---|---|
| Amazon | Product Scraper | junglee/amazon-scraper | 商品详情、评价 |
| Amazon | Review Scraper | junglee/amazon-reviews-scraper | 商品评论 |
| eBay | Scraper | drobnikj/ebay-scraper | 商品搜索 |
| AliExpress | Scraper | epctex/aliexpress-scraper | 商品数据 |
| 功能 | Actor | Actor ID | 主要功能 |
|---|---|---|---|
| 网页抓取 | Web Scraper | apify/web-scraper | 通用网页数据提取 |
| 网页截图 | Screenshot | apify/screenshot-url | 网页截图 |
| 链接提取 | Link Extractor | apify/link-extractor | 页面链接收集 |
| RSS 解析 | RSS Feed | drobnikj/rss-feed-reader | RSS 源数据 |
| AI 提取 | GPT Scraper | drobnikj/gpt-scraper | AI 驱动智能提取 |
Agent 根据用户需求自动选择最合适的 Actor：
步骤 1 — 确认需求
| 参数 | 说明 | 示例 |
|---|---|---|
| 平台 | 目标社交平台 | |
| 数据类型 | 帖子/评论/用户/标签 | 帖子 |
| 范围 | URL/关键词/用户名 | @openai |
| 数量限制 | 最大抓取条数 | 100 |
| 时间范围 | 时间过滤 | 最近 30 天 |
步骤 2 — 选择并配置 Actor
from apify_client import ApifyClient
client = ApifyClient(os.environ['APIFY_TOKEN'])
run_input = {
"usernames": ["openai"],
"resultsLimit": 100,
"resultsType": "posts",
}
run = client.actor("apify/instagram-profile-scraper").call(run_input=run_input)
步骤 3 — 获取并处理结果
items = list(client.dataset(run["defaultDatasetId"]).iterate_items())
步骤 4 — 格式化输出
将数据转换为用户需要的格式（JSON/CSV/表格摘要）。
步骤 1 — 确认搜索参数
| 参数 | 说明 | 默认值 |
|---|---|---|
| 关键词 | 搜索查询 | — |
| 搜索引擎 | Google/Bing | |
| 国家/语言 | 地域设置 | CN/zh |
| 结果数量 | 抓取条数 | 50 |
| 类型 | 网页/新闻/图片/视频 | 网页 |
步骤 2 — 调用 Actor
run_input = {
"queries": "AI agent 框架 2025",
"maxPagesPerQuery": 3,
"languageCode": "zh",
"countryCode": "cn",
"resultsPerPage": 10,
}
run = client.actor("apify/google-search-scraper").call(run_input=run_input)
items = list(client.dataset(run["defaultDatasetId"]).iterate_items())
步骤 3 — 提取关键字段
| 字段 | 说明 |
|---|---|
title | 结果标题 |
url | 链接地址 |
description | 摘要描述 |
position | 排名位置 |
当没有专用 Actor 时，使用 AI 驱动的通用提取：
方法 A — Web Scraper（基于选择器）
run_input = {
"startUrls": [{"url": "https://example.com/products"}],
"pageFunction": """
async function pageFunction(context) {
const $ = context.jQuery;
const results = [];
$('div.product-card').each((i, el) => {
results.push({
name: $(el).find('.title').text().trim(),
price: $(el).find('.price').text().trim(),
url: $(el).find('a').attr('href'),
});
});
return results;
}
""",
"maxRequestsPerCrawl": 100,
}
run = client.actor("apify/web-scraper").call(run_input=run_input)
方法 B — GPT Scraper（AI 智能提取）
run_input = {
"startUrls": [{"url": "https://example.com/products"}],
"instructions": "Extract all product names, prices, and descriptions from this page",
"openaiApiKey": os.environ.get('OPENAI_API_KEY'),
"maxRequestsPerCrawl": 10,
}
run = client.actor("drobnikj/gpt-scraper").call(run_input=run_input)
同时从多个来源抓取数据：
步骤 1 — 列出所有抓取任务 步骤 2 — 并行启动多个 Actor 步骤 3 — 等待所有任务完成 步骤 4 — 合并结果并去重
import asyncio
from apify_client import ApifyClientAsync
async def batch_scrape(tasks):
client = ApifyClientAsync(os.environ['APIFY_TOKEN'])
results = {}
async def run_actor(name, actor_id, input_data):
run = await client.actor(actor_id).call(run_input=input_data)
items = []
async for item in client.dataset(run["defaultDatasetId"]).iterate_items():
items.append(item)
results[name] = items
await asyncio.gather(*[
run_actor(t['name'], t['actor_id'], t['input'])
for t in tasks
])
return results
{
"metadata": {
"actor": "apify/instagram-profile-scraper",
"total_items": 42,
"scraped_at": "2025-03-01T14:30:00Z",
"run_id": "abc123",
"cost_usd": 0.05
},
"data": [
{
"id": "post_12345",
"text": "帖子内容...",
"likes": 1234,
"comments": 56,
"timestamp": "2025-02-28T10:00:00Z",
"url": "https://instagram.com/p/xxx"
}
]
}
import pandas as pd
df = pd.DataFrame(items)
df.to_csv('output.csv', index=False, encoding='utf-8-sig')
当数据量较大时，先输出摘要统计：
📊 抓取完成
- Actor: Instagram Profile Scraper
- 总条数: 142 条帖子
- 时间范围: 2025-01-01 ~ 2025-03-01
- 平均点赞: 2,345
- 最高互动帖子: [URL]
- 费用: $0.12
症状：所有请求返回 401
解决：确认 .env 中的 APIFY_TOKEN 已正确设置
症状：任务长时间未完成 解决：
maxRequestsPerCrawl 或 resultsLimitmemoryMbytes 增加内存分配症状：返回 403 或空结果 解决：
不同 Actor 返回的数据结构不同。在处理数据前先检查字段：
if items:
print("Available fields:", list(items[0].keys()))
Apify 按计算单元（CU）收费。大规模抓取前：
resultsLimit: 10）确认结果质量Instagram、TikTok 等平台会动态调整反爬策略：
schedule_input = {
"actorId": "apify/google-search-scraper",
"cronExpression": "0 9 * * 1", # 每周一早 9 点
"input": {
"queries": "竞品动态",
"maxPagesPerQuery": 1,
}
}
run = client.actor("apify/web-scraper").call(
run_input=run_input,
webhooks=[{
"eventTypes": ["ACTOR.RUN.SUCCEEDED"],
"requestUrl": "https://your-server.com/webhook",
}]
)
用户可在技能同目录下创建 EXTEND.md 添加：
Creating algorithmic art using p5.js with seeded randomness and interactive parameter exploration. Use this when users request creating art using code, generative art, algorithmic art, flow fields, or particle systems. Create original algorithmic art rather than copying existing artists' work to avoid copyright violations.
Analyzes article structure, identifies positions requiring visual aids, generates illustrations with Type × Style two-dimension approach. Use when user asks to "illustrate article", "add images", "generate images for article", or "为文章配图".
Knowledge comic creator supporting multiple art styles and tones. Creates original educational comics with detailed panel layouts and sequential image generation. Use when user asks to create "知识漫画", "教育漫画", "biography comic", "tutorial comic", or "Logicomix-style comic".
Generates article cover images with 5 dimensions (type, palette, rendering, text, mood) combining 9 color palettes and 6 rendering styles. Supports cinematic (2.35:1), widescreen (16:9), and square (1:1) aspects. Use when user asks to "generate cover image", "create article cover", or "make cover".
Formats plain text or markdown files with frontmatter, titles, summaries, headings, bold, lists, and code blocks. Use when user asks to "format markdown", "beautify article", "add formatting", or improve article layout. Outputs to {filename}-formatted.md.
Generate AI images using multiple providers (OpenAI DALL-E, Google Imagen, DashScope/Tongyi Wanxiang, Replicate). Supports various aspect ratios, quality presets, batch generation, and provider-specific prompt engineering techniques.