Skip to main content Inicio Creadores comeonoliver skillshub apify-reference-architecture
apify-reference-architecture Production-grade architecture patterns for Apify-powered applications.
Use when designing scraping infrastructure, building multi-Actor pipelines,
or integrating Apify into a larger system architecture.
Trigger: "apify architecture", "apify best practices",
"apify project structure", "scraping architecture", "apify system design".
Ir a la instalación Skills Marketplace Descubre y explora habilidades de IA creadas por la comunidad.
Instalar con Codex o Claude Copia este prompt, pégalo en Codex, Claude u otro asistente, y deja que revise la página de la skill y la instale por ti.
Mostrar detalles del prompt Un comando directo omite el prompt de revisión. Revisa el origen antes de ejecutarlo.
npx skills add https://github.com/ComeOnOliver/skillshub --skill apify-reference-architectureEl comando permanece en una sola línea. Desplázate horizontalmente para revisarlo antes de copiarlo.
¿Prefieres una copia local? Descarga los archivos que SkillsMP tiene disponibles ahora.
Descargar Zip Descargando... Más de este repositorio Review product and feature risk before an AI coding agent starts implementation.
Use Xquik for X data and confirmation-gated X actions: tweet search, user lookup, follower export, media download, monitors, webhooks, MCP, and SDK workflows.
Canton Network open-source ecosystem guide covering DAML SDK, Canton runtime, and Splice applications. Use when working with Canton Network, DAML smart contracts, or building decentralized applications.
Ocupaciones relacionadas SOC
Basado en la clasificación ocupacional SOC
name apify-reference-architecture description Production-grade architecture patterns for Apify-powered applications.
Use when designing scraping infrastructure, building multi-Actor pipelines,
or integrating Apify into a larger system architecture.
Trigger: "apify architecture", "apify best practices",
"apify project structure", "scraping architecture", "apify system design".
allowed-tools Read, Grep version 1.0.0 license MIT author Jeremy Longshore <jeremy@intentsolutions.io> tags ["saas","scraping","automation","apify"] compatible-with claude-code
Apify Reference Architecture
Overview
Production-ready architecture patterns for applications built on Apify. Covers standalone Actor projects, multi-Actor pipelines, and full-stack applications that integrate Apify as a data source.
Architecture Pattern 1: Standalone Actor
For a single scraper deployed to Apify platform.
my-scraper/
├── .actor/
│ ├── actor.json # Actor metadata
│ ├── INPUT_SCHEMA.json # Input definition (generates UI)
│ └── Dockerfile # Build configuration
├── src/
│ ├── main.ts # Entry point (Actor.main)
│ ├── routes/
│ │ ├── listing.ts # Router handler: listing pages
│ │ └── detail.ts # Router handler: detail pages
│ ├── types.ts # Input/output TypeScript types
│ └── utils/
│ ├── extractors.ts # Data extraction functions
│ └── validators.ts # Input/output validation
├── tests/
│ ├── extractors.test.ts # Unit tests for extraction logic
│ └── integration.test.ts # Integration tests (live API)
├── storage/ # Local storage (git-ignored)
├── package.json
├── tsconfig.json
└── .gitignore
Key Files
import { Actor } from 'apify' ;
import { CheerioCrawler } from 'crawlee' ;
import { router } from './routes/listing' ;
import { validateInput, ScraperInput } from './types' ;
await Actor .main (async () => {
const rawInput = await Actor .getInput <ScraperInput >();
const input = validateInput (rawInput);
const proxyConfiguration = input.proxyConfig ?.useApifyProxy
await
Actor
createProxyConfiguration
groups
proxyConfig
groups
undefined
const
new
CheerioCrawler
requestHandler
maxRequestsPerCrawl
maxItems
100
maxConcurrency
concurrency
10
await
run
startUrls
map
s =>
url
import { z } from 'zod' ;
export const InputSchema = z.object ({
startUrls : z.array (z.object ({ url : z.string ().url () })).min (1 ),
maxItems : z.number ().int ().positive ().optional ().default (100 ),
concurrency : z.number ().int ().min (1 ).max (50 ).optional ().default (10 ),
proxyConfig : z.object ({
useApifyProxy : z.boolean (),
groups : z.array (z.string ()).optional (),
}).optional (),
});
export type ScraperInput = z.infer <typeof InputSchema >;
export function validateInput (raw : unknown ScraperInput {
return InputSchema .parse (raw);
}
export interface ProductOutput {
url : string ;
name : string ;
price : number | null ;
currency : string ;
inStock : boolean ;
scrapedAt : string ;
}
Architecture Pattern 2: Multi-Actor Pipeline For complex scraping workflows with multiple stages.
┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│ Discover │────▶│ Scrape │────▶│ Transform │
│ Actor │ │ Actor │ │ Actor │
│ │ │ │ │ │
│ Finds URLs │ │ Extracts │ │ Dedup, │
│ to scrape │ │ raw data │ │ clean, │
│ │ │ │ │ enrich │
└──────┬───────┘ └──────┬───────┘ └──────┬───────┘
│ │ │
▼ ▼ ▼
Request Queue Dataset A Dataset B
(URLs to scrape) (raw data) (clean data)
Pipeline Orchestrator
import { ApifyClient } from 'apify-client' ;
const client = new ApifyClient ({ token : process.env .APIFY_TOKEN });
interface PipelineConfig {
discoverActorId : string ;
scrapeActorId : string ;
transformActorId : string ;
seedUrls : string [];
maxItems : number ;
}
async function runPipeline (config : PipelineConfig const results = {
discover : { runId : '' , items : 0 , cost : 0 },
scrape : { runId : '' , items : 0 , cost : 0 },
transform : { runId : '' , items : 0 , cost : 0 },
};
console .log ('Stage 1: Discovering URLs...' );
const discoverRun = await client.actor (config.discoverActorId ).call ({
seedUrls : config.seedUrls ,
maxPages : 50 ,
});
const { items : urls } = await client
.dataset (discoverRun.defaultDatasetId )
.listItems ();
results.discover = {
runId : discoverRun.id ,
items : urls.length ,
cost : discoverRun.usageTotalUsd ?? 0 ,
};
console .log (`Stage 2: Scraping ${urls.length} URLs...` );
const scrapeRun = await client.actor (config.scrapeActorId ).call ({
startUrls : urls.map ((u : any ({ url : u.url })),
maxItems : config.maxItems ,
});
results.scrape = {
runId : scrapeRun.id ,
items : scrapeRun.stats ?.datasetItemCount ?? 0 ,
cost : scrapeRun.usageTotalUsd ?? 0 ,
};
console .log ('Stage 3: Transforming...' );
const transformRun = await client.actor (config.transformActorId ).call ({
sourceDatasetId : scrapeRun.defaultDatasetId ,
dedupField : 'url' ,
filterEmpty : true ,
});
results.transform = {
runId : transformRun.id ,
items : transformRun.stats ?.datasetItemCount ?? 0 ,
cost : transformRun.usageTotalUsd ?? 0 ,
};
const finalDs = await client.datasets ().getOrCreate ('pipeline-output' );
const { items : cleanData } = await client
.dataset (transformRun.defaultDatasetId )
.listItems ();
await client.dataset (finalDs.id ).pushItems (cleanData);
const totalCost = Object .values (results).reduce ((s, r ) => s + r.cost , 0 );
console .log ('\n=== Pipeline Summary ===' );
console .log (`Discovered: ${results.discover.items} URLs` );
console .log (`Scraped: ${results.scrape.items} items` );
console .log (`Clean: ${results.transform.items} items` );
console .log (`Total cost: $${totalCost.toFixed(4 )} ` );
return results;
}
Architecture Pattern 3: Full-Stack Integration Application that uses Apify as a data source.
┌─────────────────────────────────────────────────────────┐
│ Your Application │
│ │
│ ┌─────────┐ ┌──────────────┐ ┌─────────────────┐ │
│ │ Frontend │──▶│ API Server │──▶│ Apify Service │ │
│ │ (React) │ │ (Express/ │ │ (apify-client) │ │
│ │ │◀──│ Next.js) │◀──│ │ │
│ └─────────┘ └──────┬───────┘ └────────┬─────────┘ │
│ │ │ │
│ ▼ ▼ │
│ ┌──────────┐ ┌────────────┐ │
│ │ Your DB │ │ Apify │ │
│ │ (results)│ │ Platform │ │
│ └──────────┘ └────────────┘ │
│ │
│ ┌──────────────────────────────────────────────────┐ │
│ │ Webhook Handler │ │
│ │ Receives run completion → saves results to DB │ │
│ └──────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────┘
Service Layer
import { ApifyClient } from 'apify-client' ;
export class ApifyService {
private client : ApifyClient ;
constructor (token : string this .client = new ApifyClient ({ token });
}
async startScrape (urls : string []): Promise <{ runId : string }> {
const run = await this .client .actor ('username/scraper' ).start ({
startUrls : urls.map (url =>return { runId : run.id };
}
async getRunStatus (runId : string ): Promise <{
status : string ;
progress ?: { finished : number ; failed : number };
}> {
const run = await this .client .run (runId).get ();
return {
status : run.status ,
progress : {
finished : run.stats ?.requestsFinished ?? 0 ,
failed : run.stats ?.requestsFailed ?? 0 ,
},
};
}
async getResults<T>(runId : string ): Promise <T[]> {
const run = await this .client .run (runId).get ();
if (run.status !== 'SUCCEEDED' ) {
throw new Error (`Run not ready: ${run.status} ` );
}
const { items } = await this .client
.dataset (run.defaultDatasetId )
.listItems ();
return items as T[];
}
async checkHealth (): Promise <boolean > {
try {
const user = await this .client .user ().get ();
return !!user.username ;
} catch {
return false ;
}
}
}
Configuration Management
interface ApifyConfig {
token : string ;
actorId : string ;
defaultMemory : number ;
defaultTimeout : number ;
webhookUrl ?: string ;
}
export function loadConfig (ApifyConfig {
const env = process.env .NODE_ENV || 'development' ;
const base : ApifyConfig = {
token : process.env .APIFY_TOKEN !,
actorId : process.env .APIFY_ACTOR_ID !,
defaultMemory : 1024 ,
defaultTimeout : 3600 ,
};
const overrides : Record <string , Partial <ApifyConfig >> = {
development : { defaultMemory : 256 , defaultTimeout : 300 },
staging : { defaultMemory : 512 },
production : { webhookUrl : process.env .APIFY_WEBHOOK_URL },
};
return { ...base, ...overrides[env] };
}
Health Check
export async function healthCheck (apifyService : ApifyService const start = Date .now ();
const healthy = await apifyService.checkHealth ();
return {
service : 'apify' ,
status : healthy ? 'healthy' : 'unhealthy' ,
latencyMs : Date .now () - start,
timestamp : new Date ().toISOString (),
};
}
Error Handling Issue Cause Solution Circular dependencies Service imports service Use dependency injection Missing config Env var not set Validate at startup with loadConfig() Pipeline stage failure Actor crash mid-pipeline Add retry logic per stage State management Tracking run status Use webhook handler + database
Resources
Flagship Skills For multi-environment setup, see apify-deploy-integration.
