Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.
A direct command skips the review prompt. Inspect the source before running it.
Designed for Claude Code, also compatible with Codex
LangChain Rate Limits (Python)
Overview
A team deploys 10 Cloud Run workers. Each worker initializes its ChatAnthropic
with InMemoryRateLimiter(requests_per_second=10) โ they read the docs, they
picked a safe-looking number, they shipped. Thirty seconds later the dashboard
lights up with 429s: the cluster is pushing 100 RPS to Anthropic's 50 RPM
tier-1 ceiling, not the 10 RPS they configured. The name is the fix โ
InMemoryRateLimiter is in-process. Each worker has its own counter. Ten
workers ร 10 RPS = 100 RPS to the provider. This is pain-catalog entry P29
and it lands on every team that scales past one pod.
Three more traps wait on the same code path:
P07 โ .with_fallbacks([backup]) defaults exceptions_to_handle=(Exception,),
which on Python <3.12 swallows KeyboardInterrupt. Ctrl+C during a 429
retry storm silently falls through to the backup chain and keeps billing.
P30 โ ChatOpenAI and ChatAnthropic default max_retries=6. That is
retries, not attempts: 7 total requests per logical call on flaky
networks. One .invoke() can bill 7x.
P31 โ Anthropic's RPM counts cache reads, cache writes, and uncached
calls uniformly. Cache-heavy workloads at 50 RPM can 429 on cache writes
while the ITPM dashboard shows headroom.
This skill covers measuring demand before picking a limit; the
InMemoryRateLimiter vs Redis-backed limiter vs asyncio.Semaphore decision
tree; the narrow exceptions_to_handle whitelist; max_retries=2 math; and
the provider-specific limit taxonomy (RPM, ITPM, OTPM, concurrent,
cached-vs-uncached). Pin: langchain-core 1.0.x, langchain-anthropic 1.0.x,
langchain-openai 1.0.x. Pain-catalog anchors: P07, P08, P29, P30, P31.
For .batch(max_concurrency=...) tuning, see the sibling skill
langchain-performance-tuning โ this skill is about provider-facing rate caps.
Prerequisites
Python 3.10+ (3.12+ fixes the KeyboardInterrupt half of P07)
langchain-core >= 1.0, < 2.0
At least one provider: pip install langchain-anthropic langchain-openai
For multi-worker prod: redis >= 4.5 client and a Redis server reachable from every worker
Completed langchain-model-inference โ the chat-model factory from that skill is where rate_limiter= gets attached
Instructions
Step 1 โ Measure actual demand before picking a number
Do not guess at requests_per_second. Instrument first, size second.
Attach a BaseCallbackHandler that logs per-call input_tokens,
output_tokens, and cache_read_input_tokens from response.generations[].message.usage_metadata:
Collect 24-48 hours of representative traffic. Roll up: p50 and p95 RPM, p95
ITPM, p95 OTPM, cache hit rate. Size the limiter at 70% of the binding
constraint's tier ceiling on your p95.
See Measuring Demand for the full
DemandLogger implementation, pandas roll-up, OTEL integration, load-test
harness, and multi-tenant sizing strategies.
Step 2 โ InMemoryRateLimiter for single-process dev only; never multi-worker prod
LangChain 1.0 ships InMemoryRateLimiter as a first-class BaseChatModel parameter:
Step 3 โ Redis-backed limiter for cluster-wide enforcement
For multi-worker deployments, cluster-wide rate limiting requires shared state.
Redis is the default answer โ atomic Lua script for sliding-window, or Redis
6.2+ CL.THROTTLE for GCRA.
import redis
from langchain_anthropic import ChatAnthropic
# RedisRateLimiter class defined in references/redis-limiter-pattern.mdfrom your_app.limiters import RedisRateLimiter
client = redis.Redis.from_url("redis://redis.internal:6379/0")
limiter = RedisRateLimiter(
client,
key="anthropic:prod",
requests_per_second=35 / 60, # 35 RPM cluster-wide, not per-worker
)
llm = ChatAnthropic(
model="claude-sonnet-4-6",
rate_limiter=limiter,
max_retries=2,
timeout=30,
)
Key scoping decisions:
key="anthropic:prod" โ all tenants share one global budget (simplest)
key=f"anthropic:tenant:{tenant_id}" โ per-tenant quota (requires cleanup for dead tenants)
Two-level: per-tenant + global, acquire both (best for multi-tenant SaaS)
See Redis Limiter Pattern for the full
RedisRateLimiter implementation (atomic Lua sliding window), the GCRA
alternative via CL.THROTTLE, failure modes (Redis down, clock skew), and
per-tenant cleanup strategy.
Step 4 โ asyncio.Semaphore for per-worker in-flight concurrency cap
The rate limiter throttles request rate. A semaphore throttles in-flight
count. Use both:
import asyncio
# Cluster: 35 RPM (Redis enforces)# Worker: 20 in-flight at once (semaphore enforces)
worker_sem = asyncio.Semaphore(20)
asyncdefbounded_invoke(inp):
asyncwith worker_sem:
returnawait llm.ainvoke(inp)
# Fanout
results = await asyncio.gather(*[bounded_invoke(x) for x in inputs])
Why both: a semaphore prevents a single worker from queueing hundreds of
pending limiter acquires against Redis (head-of-line blocking on the event
loop). The limiter prevents the cluster from exceeding the provider tier. They
solve different problems.
Semaphore sizing: target latency-bandwidth-product. If p95 request latency
is 2s and the worker's RPS cap is 10, in-flight count โ 2 ร 10 = 20. Overshoot
is wasted memory; undershoot leaves throughput on the table.
Step 5 โ Narrow with_fallbacks(exceptions_to_handle=...) โ never (Exception,)
.with_fallbacks([backup]) defaults to catching Exception. This is P07 โ on
Python <3.12, Exception edge-cases include KeyboardInterrupt propagation.
Ctrl+C during a retry storm silently hands off to the backup and keeps running.
Always narrow the tuple:
The whitelist is only transient provider errors. AuthenticationError,
BadRequestError, and ValidationError are bugs in your code/credentials โ
fallback produces the same crash. See the sibling skill's reference
langchain-sdk-patterns/references/fallback-exception-list.md for the full
per-provider whitelist (Anthropic, OpenAI, Gemini).
Step 6 โ max_retries=2, never the default max_retries=6
max_retries is retries, not attempts. Default max_retries=6 on
ChatOpenAI / ChatAnthropic means initial + 6 retries = 7 billed requests
per logical call (P30). On a flaky network, one .invoke() costs 7x what you
budgeted.
# BAD โ default
llm = ChatOpenAI(model="gpt-4o") # max_retries=6# GOOD โ production default
llm = ChatOpenAI(
model="gpt-4o",
max_retries=2, # initial + 2 retries = 3 total billed requests max
timeout=30,
rate_limiter=redis_limiter,
)
Trade resilience off to the fallback layer โ with_fallbacks is strictly
cheaper than retry amplification when the primary is genuinely unhealthy.
Instrument retry count via callback and alert if retry rate exceeds ~5%.
See Backoff and Retry for the full math,
Retry-After header handling, and circuit-breaker pattern for sustained
overload.
Step 7 โ Understand the provider limit taxonomy
Different providers expose different limit types. Know which one binds your
workload before you size:
Limit
Meaning
Who enforces
Binds for
RPM
Requests/minute (counts every call)
All three providers
Short chat replies
ITPM
Input tokens/minute
Anthropic, OpenAI (as TPM combined)
Long document Q&A
OTPM
Output tokens/minute
Anthropic separately; OpenAI as combined TPM
Long completions
Concurrent
In-flight request cap
Mainly OpenAI higher tiers
Burst traffic
Cached reads
Cache-read input tokens (Anthropic)
Anthropic separate budget line
Cache-heavy workloads (but still counts toward RPM โ P31)
Critical for Anthropic cache workloads (P31): RPM counts uniformly across
cached reads, cache writes, and uncached calls. A workload at 90% cache hit
rate still trips the 50 RPM ceiling at 51 requests/min. Separate monitors for
cache_read_input_tokens vs input_tokens (minus cache read/write) give
early warning.
Step 8 โ Decision tree: which limiter to use
โโ Single process (dev, notebooks, sync CLI, --workers 1)?
โ โโ InMemoryRateLimiter
โ
โโ Multi-process but single host (same-machine pool, local gunicorn)?
โ โโ Redis-backed limiter (even localhost Redis beats InMemoryRateLimiter โ
โ which still has per-process counters)
โ
โโ Multi-host cluster (Cloud Run --min-instances>1, K8s, ECS)?
โ โโ Redis-backed limiter (mandatory)
โ
โโ Multi-region or cross-cloud?
โ โโ Regional Redis per zone + provider-side account quota
โ (cross-region Redis latency adds 30-200ms per acquire)
โ
โโ Any of the above + multi-tenant SaaS?
โโ Two-level Redis limiter: per-tenant + global, acquire both
Always pair with asyncio.Semaphore(N) per-worker for in-flight concurrency.
Step 9 โ Provider tier snapshot (verify before shipping)
2026-04-21 snapshot โ re-verify against the official console before shipping.
Tiers change quarterly. A limiter sized six months ago on a different tier is
a liability. See Provider Tier Matrix for
the full matrix including ITPM / OTPM / cached-read separation, binding-limit
math, and the pre-ship verification checklist.
Output
Instrumented DemandLogger callback attached to your chains for 24-48h before sizing
InMemoryRateLimiter in dev / notebooks / single-worker only
RedisRateLimiter (sliding-window Lua or CL.THROTTLE GCRA) for any multi-worker deployment, keyed per-tenant or global
asyncio.Semaphore(N) per-worker in-flight cap paired with the cluster-wide limiter
max_retries=2 on every ChatAnthropic / ChatOpenAI / ChatGoogleGenerativeAI
.with_fallbacks(exceptions_to_handle=(RateLimitError, APITimeoutError, APIConnectionError, InternalServerError)) โ never (Exception,)
Per-provider tier re-verified from the official console, sized at 70% of the binding constraint
Error Handling
Error
Cause
Fix
anthropic.RateLimitError: 429 THROTTLED at cluster RPM = N ร InMemoryRateLimiter ceiling
InMemoryRateLimiter is per-process; N workers each send at their limit (P29)
Switch to Redis-backed limiter (Step 3)
429 on cache writes while ITPM dashboard shows headroom
Anthropic RPM counts cache writes uniformly (P31)
Budget at RPM level with limiter; separate cached vs uncached metrics
One .invoke() bills as 7 requests on flaky networks
Default max_retries=6 (P30)
max_retries=2 + fallback layer for resilience
Ctrl+C during retry storm silently falls through to backup chain
exceptions_to_handle=(Exception,) catches KeyboardInterrupt on Python <3.12 (P07)
Narrow tuple to (RateLimitError, APITimeoutError, APIConnectionError, InternalServerError)
Limiter queue p95 wait > 500ms
Limiter is oversubscribed for real traffic
Re-measure demand (Step 1); upgrade provider tier OR shed load
redis.exceptions.ConnectionError blocks all LLM calls
Redis unavailable and limiter is fail-closed
Instrument Redis health; decide fail-open (log loudly) vs fail-closed (shed load) โ for provider safety, prefer fail-closed
retry-after header climbing 2โ4โ8โ16
Pushing past tier; backoff amplifying, not absorbing
Lower limiter target RPS by 20%; upgrade tier if sustained
google.api_core.exceptions.ResourceExhausted on Gemini
Gemini free tier 15 RPM is brutal
Upgrade to paid Gemini tier 1 (2000 RPM) or use Redis limiter at 10 RPM
Examples
Multi-worker Cloud Run deployment with Anthropic tier-1 50 RPM
Ten workers, single region, Redis in same VPC. Target: 35 RPM cluster-wide
(70% of 50 RPM ceiling), 20 in-flight per worker.
Cluster behavior: every worker's limiter call hits the same Redis key. At 35
RPM cluster-wide, individual workers see fair-share throughput. max_retries=2
narrow fallback tuple means transient 429s surface quickly and hand off to
GPT-4o instead of amplifying cost.
Multi-tenant SaaS with per-tenant isolation
Two-level Redis limiter. Per-tenant limit prevents noisy neighbors; global limit
protects the provider tier.
See Redis Limiter Pattern for the
two-level acquire implementation (acquire tenant key first, then global key;
release tenant if global fails) and the per-tenant cleanup cron.
Single-process dev โ InMemoryRateLimiter is fine
For local debugging, notebook work, or a sync CLI tool:
