Troubleshoot and respond to Langfuse-related incidents and outages.
Use when experiencing Langfuse outages, debugging production issues,
or responding to LLM observability incidents.
Trigger with phrases like "langfuse incident", "langfuse outage",
"langfuse down", "langfuse production issue", "langfuse troubleshoot".
Troubleshoot and respond to Langfuse-related incidents and outages.
Use when experiencing Langfuse outages, debugging production issues,
or responding to LLM observability incidents.
Trigger with phrases like "langfuse incident", "langfuse outage",
"langfuse down", "langfuse production issue", "langfuse troubleshoot".
Designed for Claude Code, also compatible with Codex and OpenClaw
Langfuse Incident Runbook
Overview
Step-by-step procedures for Langfuse-related incidents, from initial triage (2 min) through resolution and post-incident review. Your application should work without Langfuse -- these procedures focus on restoring observability.
Check shutdown handlers; set flushAt: 1 temporarily
401 Unauthorized
Key rotation or mismatch
Verify keys match the correct project
429 Too Many Requests
Rate limited
Increase batch size, reduce flush frequency
SDK throwing errors
Unhandled exception
Wrap in try/catch; check SDK version
High request latency
Sync flush in hot path
Switch to async; increase requestTimeout
Complete Langfuse outage
Service-side issue
Enable fallback mode
Step 3: Fallback Mode (P1 -- App Impacted)
If Langfuse is causing application issues, disable tracing immediately:
// Emergency disable via environment variable// Set LANGFUSE_ENABLED=false in your deployment// In your tracing initialization:if (process.env.LANGFUSE_ENABLED === "false") {
console.warn("Langfuse tracing DISABLED (emergency fallback)");
// Don't initialize SDK -- all observe/startActiveObservation calls// will still work but produce no-op spans
}
// 1. Verify SDK is initializedconsole.log("Langfuse configured:", !!process.env.LANGFUSE_PUBLIC_KEY);
// 2. Check flush is happening// v4+: Verify NodeSDK is started and shutdown is registered// v3: Verify flushAsync() or shutdownAsync() is called// 3. Temporarily set aggressive flush for debuggingconst processor = newLangfuseSpanProcessor({
exportIntervalMillis: 1000,
maxExportBatchSize: 1,
});
Procedure B: Rate Limit (429) Recovery
// Increase batching to reduce API callsconst processor = newLangfuseSpanProcessor({
exportIntervalMillis: 30000, // 30s flushmaxExportBatchSize: 200, // Large batches
});
// Or temporarily enable samplingconstEMERGENCY_SAMPLE_RATE = 0.1; // Only trace 10%
Procedure C: Self-Hosted Instance Down
set -euo pipefail
# Check container status
docker ps -a | grep langfuse
# Check logs
docker logs langfuse-langfuse-1 --tail 50
# Check database
docker exec langfuse-postgres-1 pg_isready -U langfuse
# Restart if needed
docker compose restart langfuse
