| name | apify-reference-architecture |
| description | Production-grade architecture patterns for Apify-powered applications.
Use when designing scraping infrastructure, building multi-Actor pipelines,
or integrating Apify into a larger system architecture.
Trigger with "apify architecture", "apify best practices",
"apify project structure", "scraping architecture", "apify system design".
|
| allowed-tools | Read, Grep |
| version | 1.5.0 |
| license | MIT |
| author | Jeremy Longshore <jeremy@intentsolutions.io> |
| tags | ["saas","scraping","automation","apify"] |
| compatibility | Designed for Claude Code |
Apify Reference Architecture
Overview
Production-ready architecture patterns for applications built on Apify. Three patterns
scale from a single scraper to a full-stack integration:
- Standalone Actor — one scraper deployed to the Apify platform.
- Multi-Actor Pipeline — a discover → scrape → transform chain of Actors.
- Full-Stack Integration — an application using Apify as a data source behind a service layer.
This skill helps you choose the right pattern, lay out the directory structure, and wire
the skeleton code. Full directory trees, diagrams, and code for every pattern live in
references/architecture-patterns.md; the service
layer, configuration loader, and health check live in
references/implementation.md.
Prerequisites
- Runtime: Node.js
>=18, TypeScript, and the Apify CLI (npm i -g apify-cli).
- Packages:
apify + crawlee (inside an Actor), apify-client (calling Actors from an app), zod (input validation).
- Auth: an Apify API token. Set
APIFY_TOKEN in the environment; the Apify SDK and
apify-client read it automatically (or pass it explicitly to new ApifyClient({ token })).
Never hardcode the token — inject it via env var and validate at startup.
- Access:
Read and Grep the target repository so you can match the recommended
layout against the code already on disk before proposing changes.
Instructions
- Pick the pattern. One scraper → Pattern 1. A staged workflow that discovers,
scrapes, then cleans → Pattern 2. An app that consumes scraped data → Pattern 3.
Grep the existing repo for apify, apify-client, and Actor.main to see what
is already wired, so you extend rather than duplicate structure.- Lay out the directory from the pattern's tree in
references/architecture-patterns.md. Keep
routing, extraction, and validation in separate modules.
- Add typed input validation with
zod (see in the reference) so bad
input fails fast at the Actor boundary instead of mid-crawl.
