Scale Clay enrichment pipelines for high-volume processing (10K-100K+ leads/month).
Use when planning capacity for large enrichment runs, optimizing batch processing,
or designing high-volume Clay architectures.
Trigger with phrases like "clay scale", "clay high volume", "clay large batch",
"clay capacity planning", "clay 100k leads", "clay bulk enrichment".
Scale Clay enrichment pipelines for high-volume processing (10K-100K+ leads/month).
Use when planning capacity for large enrichment runs, optimizing batch processing,
or designing high-volume Clay architectures.
Trigger with phrases like "clay scale", "clay high volume", "clay large batch",
"clay capacity planning", "clay 100k leads", "clay bulk enrichment".
allowed-tools
Read, Write, Edit, Bash(curl:*), Bash(node:*)
version
1.14.0
license
MIT
author
Jeremy Longshore <jeremy@intentsolutions.io>
tags
["saas","clay","testing","performance","scaling"]
compatibility
Designed for Claude Code, also compatible with Codex and OpenClaw
Clay Load & Scale
Overview
Strategies for processing 10K-100K+ leads through Clay monthly. Clay is a hosted platform -- you can't add servers. Scaling focuses on: table partitioning, webhook management, batch submission pacing, credit budgeting at scale, and multi-table architectures.
Prerequisites
Clay Growth or Enterprise plan
Understanding of Clay's credit model (Data Credits + Actions)
Queue infrastructure for batch processing (Redis, SQS, or BullMQ)
For large volumes, split data across multiple Clay tables:
# Large-volume table strategytables:outbound-leads-tech:focus:"Technology companies"filter:"industry IN ('Software', 'SaaS', 'Technology')"enrichment:Fullwaterfall+Claygentvolume:~5Krows/monthoutbound-leads-finance:focus:"Financial services companies"filter:"industry IN ('Financial Services', 'Banking', 'Insurance')"enrichment:Fullwaterfall(noClaygent—regulateddata)volume:~3Krows/monthinbound-leads:focus:"Website form submissions"source:Webhookfromwebformsenrichment:Companylookup+emailverificationonlyvolume:~2Krows/monthauto_delete:true# Stream-through: enrich, push to CRM, deleteevent-attendees:focus:"Conference/webinar registrants"source:CSVimportenrichment:Fullwaterfall+AIpersonalizationvolume:~1Krows/month(batchafterevents)
Step 4: Webhook Rotation for High Volume
// src/clay/webhook-rotation.tsclassWebhookRotator {
privatewebhooks: { url: string; count: number; maxCount: number }[];
private currentIndex = 0;
constructor(webhookUrls: string[], maxPerWebhook = 45_000) {
this.webhooks = webhookUrls.map(url => ({
url,
count: 0,
maxCount: maxPerWebhook, // Leave 5K buffer under 50K limit
}));
}
getNextWebhook(): string {
// Find a webhook with remaining capacityfor (let i = 0; i < this.webhooks.length; i++) {
const idx = (this.currentIndex + i) % this.webhooks.length;
if (this.webhooks[idx].count < this.webhooks[idx].maxCount) {
this.currentIndex = idx;
returnthis.webhooks[idx].url;
}
}
thrownewError('All webhooks exhausted! Create new webhooks in Clay.');
}
recordSubmission() {
this.webhooks[this.currentIndex].count++;
}
getStatus() {
returnthis.webhooks.map((w, i) => ({
index: i,
remaining: w.maxCount - w.count,
percentUsed: ((w.count / w.maxCount) * 100).toFixed(1),
}));
}
}
// Usage: rotate across multiple webhooks for the same tableconst rotator = newWebhookRotator([
process.env.CLAY_WEBHOOK_URL_1!,
process.env.CLAY_WEBHOOK_URL_2!,
process.env.CLAY_WEBHOOK_URL_3!,
]);
Step 5: Auto-Delete for Stream-Through Processing
For high-volume use cases where Clay enriches and pushes data onward, enable auto-delete to keep tables lean:
In Clay UI: Table Settings > Auto-delete
When enabled, Clay enriches incoming webhook data, sends results via HTTP API column to your destination, then deletes the rows. This keeps Clay functioning as a streaming enrichment service rather than a database.