Troubleshoot and respond to Langfuse-related incidents and outages.
Use when experiencing Langfuse outages, debugging production issues,
or responding to LLM observability incidents.
Trigger with phrases like "langfuse incident", "langfuse outage",
"langfuse down", "langfuse production issue", "langfuse troubleshoot".
Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.
A direct command skips the review prompt. Inspect the source before running it.
The command stays on one line. Scroll horizontally to inspect it before copying.
Prefer a local copy? Download the files currently available to SkillsMP.
File Explorer
2 files
Showing SKILL.md
SKILL.md
Source instructions · Read-only preview
name
langfuse-incident-runbook
description
Troubleshoot and respond to Langfuse-related incidents and outages.
Use when experiencing Langfuse outages, debugging production issues,
or responding to LLM observability incidents.
Trigger with phrases like "langfuse incident", "langfuse outage",
"langfuse down", "langfuse production issue", "langfuse troubleshoot".
Step-by-step procedures for Langfuse-related incidents, from initial triage (2 min) through resolution and post-incident review. Your application should work without Langfuse -- these procedures focus on restoring observability.
Check shutdown handlers; set flushAt: 1 temporarily
401 Unauthorized
Key rotation or mismatch
Verify keys match the correct project
429 Too Many Requests
Rate limited
Increase batch size, reduce flush frequency
SDK throwing errors
Unhandled exception
Wrap in try/catch; check SDK version
High request latency
Sync flush in hot path
Switch to async; increase requestTimeout
Complete Langfuse outage
Service-side issue
Enable fallback mode
Step 3: Fallback Mode (P1 -- App Impacted)
If Langfuse is causing application issues, disable tracing immediately:
// Emergency disable via environment variable// Set LANGFUSE_ENABLED=false in your deployment// In your tracing initialization:if (process.env.LANGFUSE_ENABLED === "false") {
console.warn("Langfuse tracing DISABLED (emergency fallback)");
// Don't initialize SDK -- all observe/startActiveObservation calls// will still work but produce no-op spans
}
// 1. Verify SDK is initializedconsole.log("Langfuse configured:", !!process.env.LANGFUSE_PUBLIC_KEY);
// 2. Check flush is happening// v4+: Verify NodeSDK is started and shutdown is registered// v3: Verify flushAsync() or shutdownAsync() is called// 3. Temporarily set aggressive flush for debuggingconst processor = newLangfuseSpanProcessor({
exportIntervalMillis: 1000,
maxExportBatchSize: 1,
});
Procedure B: Rate Limit (429) Recovery
// Increase batching to reduce API callsconst processor = newLangfuseSpanProcessor({
exportIntervalMillis: 30000, // 30s flushmaxExportBatchSize: 200, // Large batches
});
// Or temporarily enable samplingconstEMERGENCY_SAMPLE_RATE = 0.1; // Only trace 10%
Procedure C: Self-Hosted Instance Down
set -euo pipefail
# Check container status
docker ps -a | grep langfuse
# Check logs
docker logs langfuse-langfuse-1 --tail 50
# Check database
docker exec langfuse-postgres-1 pg_isready -U langfuse
# Restart if needed
docker compose restart langfuse
