Skip to main content Startseite Ersteller comeonoliver skillshub apify-reference-architecture
apify-reference-architecture Production-grade architecture patterns for Apify-powered applications.
Use when designing scraping infrastructure, building multi-Actor pipelines,
or integrating Apify into a larger system architecture.
Trigger: "apify architecture", "apify best practices",
"apify project structure", "scraping architecture", "apify system design".
Zur Installation springen Skills Marktplatz Entdecken und erkunden Sie KI-Skills, die von der Community erstellt wurden.
Mit Codex oder Claude installieren Kopieren Sie diesen Prompt, fügen Sie ihn in Codex, Claude oder einen anderen Assistant ein und lassen Sie die Skill-Seite prüfen und installieren.
Prompt-Details anzeigen Ein direkter Befehl überspringt den Prüf-Prompt. Prüfen Sie die Quelle, bevor Sie ihn ausführen.
npx skills add https://github.com/ComeOnOliver/skillshub --skill apify-reference-architectureDer Befehl bleibt in einer Zeile. Scrollen Sie horizontal, um ihn vor dem Kopieren vollständig zu prüfen.
Sie bevorzugen eine lokale Kopie? Laden Sie die Dateien herunter, die SkillsMP derzeit vorliegen.
ZIP herunterladen Herunterladen... Mehr aus diesem Repository Review product and feature risk before an AI coding agent starts implementation.
Use Xquik for X data and confirmation-gated X actions: tweet search, user lookup, follower export, media download, monitors, webhooks, MCP, and SDK workflows.
Canton Network open-source ecosystem guide covering DAML SDK, Canton runtime, and Splice applications. Use when working with Canton Network, DAML smart contracts, or building decentralized applications.
Verwandte Berufe SOC
Basierend auf der SOC-Berufsklassifikation
name apify-reference-architecture description Production-grade architecture patterns for Apify-powered applications.
Use when designing scraping infrastructure, building multi-Actor pipelines,
or integrating Apify into a larger system architecture.
Trigger: "apify architecture", "apify best practices",
"apify project structure", "scraping architecture", "apify system design".
allowed-tools Read, Grep version 1.0.0 license MIT author Jeremy Longshore <jeremy@intentsolutions.io> tags ["saas","scraping","automation","apify"] compatible-with claude-code
Apify Reference Architecture
Overview
Production-ready architecture patterns for applications built on Apify. Covers standalone Actor projects, multi-Actor pipelines, and full-stack applications that integrate Apify as a data source.
Architecture Pattern 1: Standalone Actor
For a single scraper deployed to Apify platform.
my-scraper/
├── .actor/
│ ├── actor.json # Actor metadata
│ ├── INPUT_SCHEMA.json # Input definition (generates UI)
│ └── Dockerfile # Build configuration
├── src/
│ ├── main.ts # Entry point (Actor.main)
│ ├── routes/
│ │ ├── listing.ts # Router handler: listing pages
│ │ └── detail.ts # Router handler: detail pages
│ ├── types.ts # Input/output TypeScript types
│ └── utils/
│ ├── extractors.ts # Data extraction functions
│ └── validators.ts # Input/output validation
├── tests/
│ ├── extractors.test.ts # Unit tests for extraction logic
│ └── integration.test.ts # Integration tests (live API)
├── storage/ # Local storage (git-ignored)
├── package.json
├── tsconfig.json
└── .gitignore
Key Files
import { Actor } from 'apify' ;
import { CheerioCrawler } from 'crawlee' ;
import { router } from './routes/listing' ;
import { validateInput, ScraperInput } from './types' ;
await Actor .main (async () => {
const rawInput = await Actor .getInput <ScraperInput >();
const input = validateInput (rawInput);
const proxyConfiguration = input.proxyConfig ?.useApifyProxy
await
Actor
createProxyConfiguration
groups
proxyConfig
groups
undefined
const
new
CheerioCrawler
requestHandler
maxRequestsPerCrawl
maxItems
100
maxConcurrency
concurrency
10
await
run
startUrls
map
s =>
url
import { z } from 'zod' ;
export const InputSchema = z.object ({
startUrls : z.array (z.object ({ url : z.string ().url () })).min (1 ),
maxItems : z.number ().int ().positive ().optional ().default (100 ),
concurrency : z.number ().int ().min (1 ).max (50 ).optional ().default (10 ),
proxyConfig : z.object ({
useApifyProxy : z.boolean (),
groups : z.array (z.string ()).optional (),
}).optional (),
});
export type ScraperInput = z.infer <typeof InputSchema >;
export function validateInput (raw : unknown ScraperInput {
return InputSchema .parse (raw);
}
export interface ProductOutput {
url : string ;
name : string ;
price : number | null ;
currency : string ;
inStock : boolean ;
scrapedAt : string ;
}
Architecture Pattern 2: Multi-Actor Pipeline For complex scraping workflows with multiple stages.
┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│ Discover │────▶│ Scrape │────▶│ Transform │
│ Actor │ │ Actor │ │ Actor │
│ │ │ │ │ │
│ Finds URLs │ │ Extracts │ │ Dedup, │
│ to scrape │ │ raw data │ │ clean, │
│ │ │ │ │ enrich │
└──────┬───────┘ └──────┬───────┘ └──────┬───────┘
│ │ │
▼ ▼ ▼
Request Queue Dataset A Dataset B
(URLs to scrape) (raw data) (clean data)
Pipeline Orchestrator
import { ApifyClient } from 'apify-client' ;
const client = new ApifyClient ({ token : process.env .APIFY_TOKEN });
interface PipelineConfig {
discoverActorId : string ;
scrapeActorId : string ;
transformActorId : string ;
seedUrls : string [];
maxItems : number ;
}
async function runPipeline (config : PipelineConfig const results = {
discover : { runId : '' , items : 0 , cost : 0 },
scrape : { runId : '' , items : 0 , cost : 0 },
transform : { runId : '' , items : 0 , cost : 0 },
};
console .log ('Stage 1: Discovering URLs...' );
const discoverRun = await client.actor (config.discoverActorId ).call ({
seedUrls : config.seedUrls ,
maxPages : 50 ,
});
const { items : urls } = await client
.dataset (discoverRun.defaultDatasetId )
.listItems ();
results.discover = {
runId : discoverRun.id ,
items : urls.length ,
cost : discoverRun.usageTotalUsd ?? 0 ,
};
console .log (`Stage 2: Scraping ${urls.length} URLs...` );
const scrapeRun = await client.actor (config.scrapeActorId ).call ({
startUrls : urls.map ((u : any ({ url : u.url })),
maxItems : config.maxItems ,
});
results.scrape = {
runId : scrapeRun.id ,
items : scrapeRun.stats ?.datasetItemCount ?? 0 ,
cost : scrapeRun.usageTotalUsd ?? 0 ,
};
console .log ('Stage 3: Transforming...' );
const transformRun = await client.actor (config.transformActorId ).call ({
sourceDatasetId : scrapeRun.defaultDatasetId ,
dedupField : 'url' ,
filterEmpty : true ,
});
results.transform = {
runId : transformRun.id ,
items : transformRun.stats ?.datasetItemCount ?? 0 ,
cost : transformRun.usageTotalUsd ?? 0 ,
};
const finalDs = await client.datasets ().getOrCreate ('pipeline-output' );
const { items : cleanData } = await client
.dataset (transformRun.defaultDatasetId )
.listItems ();
await client.dataset (finalDs.id ).pushItems (cleanData);
const totalCost = Object .values (results).reduce ((s, r ) => s + r.cost , 0 );
console .log ('\n=== Pipeline Summary ===' );
console .log (`Discovered: ${results.discover.items} URLs` );
console .log (`Scraped: ${results.scrape.items} items` );
console .log (`Clean: ${results.transform.items} items` );
console .log (`Total cost: $${totalCost.toFixed(4 )} ` );
return results;
}
Architecture Pattern 3: Full-Stack Integration Application that uses Apify as a data source.
┌─────────────────────────────────────────────────────────┐
│ Your Application │
│ │
│ ┌─────────┐ ┌──────────────┐ ┌─────────────────┐ │
│ │ Frontend │──▶│ API Server │──▶│ Apify Service │ │
│ │ (React) │ │ (Express/ │ │ (apify-client) │ │
│ │ │◀──│ Next.js) │◀──│ │ │
│ └─────────┘ └──────┬───────┘ └────────┬─────────┘ │
│ │ │ │
│ ▼ ▼ │
│ ┌──────────┐ ┌────────────┐ │
│ │ Your DB │ │ Apify │ │
│ │ (results)│ │ Platform │ │
│ └──────────┘ └────────────┘ │
│ │
│ ┌──────────────────────────────────────────────────┐ │
│ │ Webhook Handler │ │
│ │ Receives run completion → saves results to DB │ │
│ └──────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────┘
Service Layer
import { ApifyClient } from 'apify-client' ;
export class ApifyService {
private client : ApifyClient ;
constructor (token : string this .client = new ApifyClient ({ token });
}
async startScrape (urls : string []): Promise <{ runId : string }> {
const run = await this .client .actor ('username/scraper' ).start ({
startUrls : urls.map (url =>return { runId : run.id };
}
async getRunStatus (runId : string ): Promise <{
status : string ;
progress ?: { finished : number ; failed : number };
}> {
const run = await this .client .run (runId).get ();
return {
status : run.status ,
progress : {
finished : run.stats ?.requestsFinished ?? 0 ,
failed : run.stats ?.requestsFailed ?? 0 ,
},
};
}
async getResults<T>(runId : string ): Promise <T[]> {
const run = await this .client .run (runId).get ();
if (run.status !== 'SUCCEEDED' ) {
throw new Error (`Run not ready: ${run.status} ` );
}
const { items } = await this .client
.dataset (run.defaultDatasetId )
.listItems ();
return items as T[];
}
async checkHealth (): Promise <boolean > {
try {
const user = await this .client .user ().get ();
return !!user.username ;
} catch {
return false ;
}
}
}
Configuration Management
interface ApifyConfig {
token : string ;
actorId : string ;
defaultMemory : number ;
defaultTimeout : number ;
webhookUrl ?: string ;
}
export function loadConfig (ApifyConfig {
const env = process.env .NODE_ENV || 'development' ;
const base : ApifyConfig = {
token : process.env .APIFY_TOKEN !,
actorId : process.env .APIFY_ACTOR_ID !,
defaultMemory : 1024 ,
defaultTimeout : 3600 ,
};
const overrides : Record <string , Partial <ApifyConfig >> = {
development : { defaultMemory : 256 , defaultTimeout : 300 },
staging : { defaultMemory : 512 },
production : { webhookUrl : process.env .APIFY_WEBHOOK_URL },
};
return { ...base, ...overrides[env] };
}
Health Check
export async function healthCheck (apifyService : ApifyService const start = Date .now ();
const healthy = await apifyService.checkHealth ();
return {
service : 'apify' ,
status : healthy ? 'healthy' : 'unhealthy' ,
latencyMs : Date .now () - start,
timestamp : new Date ().toISOString (),
};
}
Error Handling Issue Cause Solution Circular dependencies Service imports service Use dependency injection Missing config Env var not set Validate at startup with loadConfig() Pipeline stage failure Actor crash mid-pipeline Add retry logic per stage State management Tracking run status Use webhook handler + database
Resources
Flagship Skills For multi-environment setup, see apify-deploy-integration.
