Build a complete web scraping Actor with Crawlee and deploy to Apify.
Use when you need end-to-end web scraping on Apify: defining an input schema, building a router-based Crawlee crawler, extracting structured data, storing results in a dataset, testing locally, and deploying the Actor to the platform.
Trigger with "apify scrape website", "build apify actor", "crawlee scraper", "apify main workflow".
Build a complete web scraping Actor with Crawlee and deploy to Apify.
Use when you need end-to-end web scraping on Apify: defining an input schema, building a router-based Crawlee crawler, extracting structured data, storing results in a dataset, testing locally, and deploying the Actor to the platform.
Trigger with "apify scrape website", "build apify actor", "crawlee scraper", "apify main workflow".
End-to-end workflow: define input schema, build a Crawlee-based Actor, extract structured data, store results in datasets, test locally, and deploy to Apify platform. This is the primary money-path workflow for Apify.
Prerequisites
npm install apify crawlee in your project
npm install -g apify-cli and apify login completed
For programmatic retrieval (Step 6), an API token in APIFY_TOKEN — read it from
the environment (process.env.APIFY_TOKEN), never hard-code it
Familiarity with apify-sdk-patterns
Instructions
Step 1: Define Input Schema
Create .actor/INPUT_SCHEMA.json:
{"title":"E-Commerce Scraper","type":"object","schemaVersion":1,"properties":{"startUrls":{"title":"Start URLs","type":"array","description":"Product listing page URLs to scrape","editor":"requestListSources","prefill":
[
{
"url"
:
"https://example-store.com/products"
}
]
}
,
"maxItems"
:
{
"title"
:
"Max items"
,
"type"
:
"integer"
,
"description"
:
"Maximum number of products to scrape"
,
"default"
:
100
,
"minimum"
:
1
,
"maximum"
:
10000
}
,
"proxyConfig"
:
{
"title"
:
"Proxy configuration"
,
"type"
:
"object"
,
"description"
:
"Select proxy to use"
,
"editor"
:
"proxy"
,
"default"
:
{
"useApifyProxy"
:
true
}
}
}
,
"required"
:
[
"startUrls"
]
}
Step 2: Build the Actor with Router Pattern
Use a Crawlee router that splits handling by page type: the default handler
enqueues product links + pagination from listing pages, and a PRODUCT-labeled
handler extracts structured fields from detail pages. The entry point wires proxy
config, concurrency, a failed-request handler, and a run summary into the key-value
store. Skeleton:
The full typed Actor — Product/ProductInput interfaces, proxy configuration,
failedRequestHandler, and the SUMMARY key-value write — is in
implementation.md, Step 2.
Step 3: Configure Dockerfile
Use the apify/actor-node:20 base with a two-stage build (compile TypeScript in a
builder stage, ship only dist/ + production deps). Full Dockerfile:
implementation.md, Step 3.
Step 4: Test Locally
# Create test inputmkdir -p storage/key_value_stores/default
echo'{"startUrls":[{"url":"https://example.com"}],"maxItems":5}' \
> storage/key_value_stores/default/INPUT.json
# Run locally
apify run
# Check resultsls storage/datasets/default/
cat storage/key_value_stores/default/SUMMARY.json
Step 5: Deploy to Apify Platform
# Push to Apify (creates Actor if it doesn't exist)
apify push
# Or push to a specific Actor
apify push username/my-actor
# Run on platform
apify actors call username/my-actor
Step 6: Retrieve Results Programmatically
From any client, use the apify-client SDK to call the deployed Actor, list its
dataset items, and download results (JSON/CSV). The token comes from
process.env.APIFY_TOKEN — never hard-code it. Full retrieval code:
implementation.md, Step 6.
Once your Actor is deployed and producing data, move on to dataset and key-value
store management — pagination over large datasets, deduplication, exporting to
external stores, and scheduling recurring runs — covered in apify-core-workflow-b.
