Automate web scraping and data extraction with Apify -- run Actors, manage datasets, create reusable tasks, and retrieve crawl results through the Composio Apify integration.
Automate web scraping and data extraction with Apify -- run Actors, manage datasets, create reusable tasks, and retrieve crawl results through the Composio Apify integration.
requires
{"mcp":["rube"]}
Apify Automation
Run Apify web scraping Actors and manage datasets directly from Claude Code. Execute crawlers synchronously or asynchronously, retrieve structured data, create reusable tasks, and inspect run logs without leaving your terminal.
APIFY_DATASETS_GET -- list all your datasets with pagination
APIFY_DATASET_GET -- get metadata for a specific dataset
For debugging:
APIFY_GET_LOG -- retrieve execution logs for a run or build
Example prompt: "List the last 10 runs for the web scraper Actor and show logs for the most recent one"
Known Pitfalls
Actor input schemas vary wildly: Every Actor has its own unique input fields. Generic field names like queries or search_terms will be rejected. Always check the Actor's page on apify.com/store for exact field names (e.g., searchStringsArray for Google Maps, startUrls for web scrapers).
URL format requirements: Always include the full protocol (https:// or http://) in URLs. Many Actors require URLs as objects with a url property: {"startUrls": [{"url": "https://example.com"}]}.
Dataset pagination cap:APIFY_GET_DATASET_ITEMS has a max limit of 1000 per call. For large datasets, loop with offset to collect all items.
Enum values are lowercase: Most Actors expect lowercase enum values (e.g., relevance not RELEVANCE, all not ALL).
Sync timeout at 5 minutes:APIFY_RUN_ACTOR_SYNC_GET_DATASET_ITEMS has a maximum waitForFinish of 300 seconds. For longer runs, use APIFY_RUN_ACTOR (async) and poll with APIFY_GET_DATASET_ITEMS.
Data volume costs: Large datasets can be expensive to fetch. Prefer moderate limits and incremental processing to avoid timeouts or memory pressure.
JSON format recommended: While CSV/XLSX formats are available, JSON is the most reliable for automated processing. Avoid CSV/XLSX for downstream automation.
Quick Reference
Tool Slug
Description
APIFY_RUN_ACTOR_SYNC_GET_DATASET_ITEMS
Run Actor synchronously and get results immediately
