3:07am. PagerDuty: "LangChain p95 latency > 10s for 5 minutes." You open LangSmith,
filter by service="triage-agent" over the last 15 minutes, and the first trace
is 43 seconds long — an agent is on step 24 of 25 iterations, bouncing between
the same two tools on a vague user prompt ("help me with my account"). The cost
dashboard shows $400 spent in the last 10 minutes, up from a $6/hour baseline.
This is P10: create_react_agent defaults to recursion_limit=25 with no cost
cap; vague prompts never converge; the spend hits before GraphRecursionError
surfaces. First move is not to push a code fix — it is to flip
recursion_limit=5 via config reload and add a middleware token-budget cap per
session, then deal with the stuck sessions.
Or: same alert, different signature. p95 is healthy at 1.8s, but p99 is 12s and
spiky. The spikes correlate with instance starts in Cloud Run. P36: Python +
LangChain + embedding preloads = 5–15s cold start; Cloud Run scales to zero by
default, so first-request p99 is 10x p95. First move is --min-instances=1
(or a keepalive pinger), not more CPU.
The shape of the page decides the first move. This runbook gives you:
The LLM-specific SLO set most teams do not have: p95 TTFT <1s, p99 total
latency <10s, error-rate <0.5%, cost-per-req <$0.05 — with Prometheus
burn-rate recording rules that page on user-visible regression.
A triage decision tree with three root paths (latency / cost / error-rate),
each with a 3-step diagnostic and first-response action.
Provider outage runbook wired to .with_fallbacks(backup) so failover is a
config flip, not a code change.
Agent-loop containment via recursion_limit tuning and middleware
token-budget caps so runaway agents stop burning cost before the
GraphRecursionError.
Post-incident debug bundle (cross-ref langchain-debug-bundle) and write-up
template.
See LLM SLOs for the canonical set (free / paid /
enterprise tiers), burn-rate recipes (fast + slow), and a TTFT-specific rule
that requires streaming to be instrumented.
Step 2 — Triage decision tree: which root path?
The alert name tells you the root path. Do not mix diagnostics across paths —
the first-response action differs.
Alert fired
├── Latency (p95/p99 breach, TTFT breach)
│ ├── 1. Provider status page (Anthropic, OpenAI) green? → if red, Step 3
│ ├── 2. Cold start pattern? (p99 >> p95, correlates with instance starts) → P36
│ └── 3. Streaming configured? (TTFT only makes sense with .stream/.astream)
│
├── Cost (spend/req or absolute spend/hour breach)
│ ├── 1. Agent recursion depth? (LangSmith: max steps per trace) → P10
│ ├── 2. Retry rate elevated? (callback log: attempt count / logical call) → P30
│ └── 3. Token-use per req regression? (input + output tokens from callbacks)
│
└── Error rate (5xx + uncaught exceptions)
├── 1. Provider 429/500 spike? (distinguish client 4xx from provider 5xx)
├── 2. Auth? (API key rotation, expired token, org quota exhausted)
└── 3. Schema drift on structured output? (Pydantic ValidationError in traces)
Step 3 — Provider outage: detect, circuit-break, fail over
Detection precedes failover. Do not flip fallbacks on an application bug.
Detect via three signals — all three should agree before declaring a
provider outage:
Vendor status page watcher (status.anthropic.com, status.openai.com) —
poll every 30s, surface into Slack
In-app canary probe — a 1-req/min call to each configured provider with a
trivial prompt, tracked as a separate SLO
Error-rate spike on the primary provider in your own metrics (distinguishes
a real outage from your app's bug)
Circuit-break the primary: a CircuitBreaker middleware (see
langchain-middleware-patterns if available, or a simple
aiobreaker-backed runnable) opens after N consecutive APIError /
APITimeoutError within a window. Once open, calls skip the primary and go
straight to the backup. This bounds the latency cost of a down provider.
Fail over via .with_fallbacks(backup) — the fallback chain is already
wired (see langchain-rate-limits). During an outage, either flip a feature
flag that swaps the default factory, or temporarily set the primary's
max_retries=0 so the chain reaches the fallback immediately.
Comms: post a user-facing status page entry ("Degraded performance on
feature X — monitoring upstream provider") and an internal Slack update with
the canary graph attached. Provider Outage Playbook
has the full comms template and the circuit-breaker middleware snippet.
Step 4 — Agent loop containment: stop the bleed before GraphRecursionError
P10 is the most common cost-spike cause. create_react_agent defaults to
recursion_limit=25, meaning 25 model calls per user turn — with Claude Sonnet
at ~$3/MTok input, a 10k-token tool-call loop burns real money per minute.
Three containment layers, applied in order:
Set recursion_limit per agent depth — interactive chat agents rarely
need more than 5–8 steps; background research agents can justify 15; never
leave the default 25 in production.
from langgraph.prebuilt import create_react_agent
agent = create_react_agent(
llm, tools,
recursion_limit=8, # P10 — was default 25
)
Middleware token-budget cap per session — a callback that tracks
cumulative input + output tokens for a session id and raises a custom
BudgetExceeded exception once the cap is hit. The agent terminates
cleanly; the user sees a polite "I could not finish this task in budget,
try rephrasing" instead of a spinning UI until GraphRecursionError.
Circuit on repeated tool calls — a LangGraph edge that routes to END
when the same tool has been called with the same args twice in a row. This
is a cheap heuristic for "the agent is stuck in a loop."
Cost Overrun Response has the middleware
implementation, the repeat-tool edge pattern, and a per-tenant budget
enforcement example.
Debug bundle — capture the failing LangSmith trace URL(s), the
Prometheus dashboard screenshot at the breach window, the agent's config
(recursion_limit, model id, max_retries), and the provider's status page
state at incident time. Cross-reference langchain-debug-bundle if present.
Write-up template — timeline (detection → triage → mitigation →
all-clear), root cause in one sentence with pain-catalog anchor (e.g.
"P10: recursion_limit=25 default, vague prompt, no token cap"), permanent
fix ticket link, follow-up SLO tuning.
Comms — close the user-facing status page entry, post a short Slack
summary (one-paragraph timeline + link to write-up), schedule the
post-mortem review in the next weekly SRE sync.
Output
LLM SLO set (TTFT, p99 latency, error-rate, cost-per-req) published and
alerted via Prometheus burn-rate rules
Triage decision tree posted in the runbook with three root paths (latency /
cost / error-rate) and first-response actions per leaf
Provider outage playbook with circuit breaker + .with_fallbacks(backup)
wired to a feature flag for one-flip failover
recursion_limit set per agent depth (never default 25 in prod); middleware
token-budget cap per session
Post-incident debug-bundle template + write-up template wired to the
on-call workflow
Error Handling
Symptom
Likely cause
First-response action
p95 latency breach, TTFT degraded
Streaming disabled, or provider-side latency
Verify .stream()/.astream() used; check provider status page
recursion_limit=5–8, add middleware token-budget cap
Cost spike, callback log shows 7 attempts per logical call
max_retries=6 inflates cost 7x (P30)
max_retries=2 + circuit breaker; log retries via callbacks
429 storm despite requests_per_second=10 on each of N workers
InMemoryRateLimiter is per-process (P29)
Switch to RedisRateLimiter or provider-side quota
Anthropic 429 while token budget has headroom
Cache RPM throttled separately (P31)
Client-side semaphore on RPM, not token count; monitor cached-read vs uncached separately
Error-rate spike, all on primary provider
Provider outage
Canary probe confirms; flip failover to .with_fallbacks(backup) via flag
ValidationError surge on structured output
Schema drift — model added fields
ConfigDict(extra="ignore") on the Pydantic schema (see langchain-sdk-patterns)
Agent never terminates, no GraphRecursionError yet
Stuck in tool-call loop
Add "repeated tool call" edge routing to END; raise BudgetExceeded from middleware
Examples
On-call page: cost spike from agent runaway
PagerDuty: "cost-per-req > $0.20 for 15 minutes." LangSmith filtered to the
last 15 minutes shows average trace depth = 22 steps (baseline 4). Single
tenant, single conversation pattern — a user who asked an open-ended question
the agent cannot resolve. First-response action: flip recursion_limit=5 via
config reload (no deploy), add session to the blocklist in middleware, post
internal Slack with the trace URL.
See Cost Overrun Response for the
middleware token-budget implementation and the per-tenant budget pattern.
On-call page: p99 latency spike during traffic ramp
p95 healthy at 1.8s, p99 at 12s, spikes correlate with Cloud Run instance
starts — classic P36. First-response action: gcloud run services update <svc> --min-instances=1, verify heavy imports are at module top level,
schedule follow-up ticket to move embedding preload to a warm-up hook.
See Latency Triage for the cold-start
detection recipe and the p95-vs-p99 attribution decision tree.
On-call page: provider outage mid-day
Anthropic status page goes red. Canary probe error-rate jumps from 0% to 100%
on Anthropic, stays at 0% on OpenAI. Flip the failover flag — the
.with_fallbacks(backup=ChatOpenAI(...)) chain (already wired via
langchain-rate-limits) takes over. Post user-facing status entry, monitor
cost (OpenAI pricing differs — watch cost-per-req SLO), revert when upstream
recovers.
See Provider Outage Playbook for the
circuit-breaker middleware, the canary probe snippet, and the user-comms
template.
