audit-langfuse-llm

name	audit-langfuse-llm
description	Run a PDCA quality audit on LLM/AI features: traces, prompts, costs, evals, grounding, hallucination. Use for "audit LLM", "check Langfuse", "audit prompts", "check AI quality", "audit AI costs", "check traces", "audit eval scores", "verify AI pipeline".

Langfuse LLM Quality Audit

Automated PDCA audit for LLM/AI features: trace completeness, prompt quality, cost efficiency, eval health, grounding accuracy, and end-to-end pipeline verification. Works with any project — auto-detects Langfuse setup from the codebase.

Critical Rules

NEVER skip the auto-detect phase. Every project configures Langfuse differently.

Research before judging. Use Firecrawl to find current LLM best practices so recommendations are evidence-based, not opinion.

Verify live, not just statically. Trigger AI features via Playwright and confirm traces land in Langfuse — static code analysis alone misses runtime issues.

Use concrete numbers. "Costs seem high" is not an audit finding. "gpt-4.1 used for intent classification at $0.02/call when gpt-4.1-mini at $0.002/call achieves equivalent accuracy" is.

Always use the protocol-browser-anti-stall protocol when using playwright-cli.

---

Phase 0: Auto-Detect Langfuse Integration

0a. Find Langfuse Configuration

Search for environment variables and config files (in order):

1. .env, .env.local, .env.production — look for LANGFUSE_PUBLIC_KEY, LANGFUSE_SECRET_KEY, LANGFUSE_BASE_URL, LANGFUSE_HOST
2. langfuse.config.ts, langfuse.config.js — dedicated config files
3. instrumentation.ts / instrumentation.js — Next.js instrumentation with Langfuse
4. Supabase Edge Functions — Glob("**/supabase/functions/**/index.ts") and search for Langfuse imports

Grep(pattern: "LANGFUSE_PUBLIC_KEY|LANGFUSE_SECRET_KEY|LANGFUSE_BASE_URL|LANGFUSE_HOST", glob: ".env*")
Grep(pattern: "langfuse|Langfuse|@langfuse", glob: "*.{ts,js,tsx,jsx,py,rb,go}")

Record:

• LANGFUSE_HOST (cloud or self-hosted URL)
• LANGFUSE_PUBLIC_KEY (identifies the project)
• Which source files import/use Langfuse
• Whether the CLI env vars are available (the Shell commands below require LANGFUSE_PUBLIC_KEY and LANGFUSE_SECRET_KEY in the environment)

0b. Detect LLM Framework and Provider

Grep(pattern: "openai|OpenAI|anthropic|Anthropic|@google/generative-ai|gemini|cohere|mistral|groq|together|replicate", glob: "*.{ts,js,py}")
Grep(pattern: "langchain|LangChain|@langchain|vercel/ai|ai/core|createOpenAI|createAnthropic", glob: "*.{ts,js,py}")

Record:

• LLM providers (OpenAI, Anthropic, Google, etc.)
• LLM frameworks (LangChain, Vercel AI SDK, direct API calls, etc.)
• Model names used (grep for model name strings like gpt-4.1, claude-opus-4-8, gemini-2.5-pro)

0c. Map AI Features

SemanticSearch(query: "Where are LLM/AI features called in the codebase?", target_directories: [])

Build a feature map:

Feature	File(s)	Provider	Model	Traced?
e.g. Chat	app/api/chat/route.ts	OpenAI	gpt-4.1	Yes

0d. Detect Eval and Prompt Management Setup

Grep(pattern: "createScore|langfuse.score|annotation|eval|judge|dataset", glob: "*.{ts,js,py}")
Grep(pattern: "getPrompt|langfuse.prompt|fetchPrompt|compilePrompt", glob: "*.{ts,js,py}")

Record:

• Prompt management approach: Langfuse managed prompts vs hardcoded vs config file
• Eval setup: annotation queues, programmatic scoring, judge LLM, dataset runs
• Whether prompts are versioned and labeled

---

Phase 1: Research LLM Best Practices

Before auditing, establish the current state of the art so findings are grounded in evidence.

1a. Firecrawl Research

firecrawl:firecrawl_search
{
 "query": "LLM observability best practices production monitoring [current year]",
 "limit": 5
}

firecrawl:firecrawl_search
{
 "query": "prompt engineering evaluation scoring hallucination detection [current year]",
 "limit": 5
}

firecrawl:firecrawl_search
{
 "query": "LLM cost optimization token usage model selection production [current year]",
 "limit": 5
}

Scrape the top 2-3 most relevant results for detailed guidance:

firecrawl:firecrawl_scrape
{
 "url": "<BEST_RESULT_URL>",
 "formats": ["markdown"]
}

1b. Langfuse Documentation

Research Langfuse-specific features relevant to the detected setup:

firecrawl:firecrawl_search
{
 "query": "site:langfuse.com docs tracing prompts evaluation scores",
 "limit": 5
}

If the project uses a specific LLM framework (LangChain, Vercel AI SDK, etc.), also fetch its Langfuse integration docs.

1c. Context7 for LLM Framework Docs

If detected in Phase 0b, fetch the framework-specific documentation:

context7:resolve-library-id
{
 "libraryName": "<DETECTED_FRAMEWORK e.g. langchain or vercel-ai>"
}

Then query for integration patterns:

context7:query-docs
{
 "libraryId": "<RESOLVED_ID>",
 "query": "Langfuse integration tracing observability"
}

---

Phase 2: Audit via Langfuse CLI

All commands below use the Langfuse CLI via the Shell tool. Ensure the environment has LANGFUSE_PUBLIC_KEY and LANGFUSE_SECRET_KEY set (from .env or exported).

2a. Trace Completeness

npx langfuse-cli api traces list --limit 50

For each AI feature identified in Phase 0c, verify:

• Trace exists with a matching name/metadata
• Trace has spans/generations (not just a top-level trace with no children)
• Trace includes input/output (not empty)
• Trace has proper metadata (userId, sessionId, tags)
• Latency is recorded

Red flags:

• AI feature exists in code but produces no traces → missing instrumentation
• Traces exist but have no generations → incomplete tracing (wrapper created but LLM call not captured)
• Traces with empty output → output not being captured (fire-and-forget pattern)

2b. Prompt Quality Audit

npx langfuse-cli api prompts list

For each prompt:

npx langfuse-cli api prompts get --name "<PROMPT_NAME>"

Evaluate:

• Versioning: Are prompts versioned (v1, v2, v3+) or stuck at v1?
• Labels: Is there a production label? Are there staging/experiment labels for A/B testing?
• System message quality: Clear role definition, constraints, output format instructions?
• Few-shot examples: Does the prompt include examples for complex tasks?
• Guardrails: Does the prompt include instructions to refuse off-topic/harmful requests?
• Variables: Are dynamic parts properly templated with {{variables}} not string concatenation?
• Freshness: When was the prompt last updated? Stale prompts may not use newer model capabilities.

If prompts are hardcoded in source code instead of managed via Langfuse:

Grep(pattern: "You are|system.*message|systemPrompt|SYSTEM_PROMPT", glob: "*.{ts,js,py}")

Flag hardcoded prompts as a finding — they should be migrated to Langfuse for versioning and A/B testing.

2c. Model and Cost Efficiency

From trace data, analyze:

npx langfuse-cli api traces list --limit 50

For each trace, check the generation details (model, usage tokens, latency, cost).

Build a cost table:

Feature	Model	Avg Input Tokens	Avg Output Tokens	Avg Latency	Est. Cost/Call

Red flags:

• Expensive model (gpt-4.1, claude-opus-4-8) used for simple classification/extraction → recommend cheaper model (e.g. gpt-4.1-mini, claude-haiku-4-5)
• High input token counts → check for unnecessary context stuffing
• Output tokens much larger than needed → add max_tokens or response format constraints
• High latency on user-facing features → consider streaming, caching, or smaller model
• Same content sent repeatedly → implement semantic caching

2d. Eval Score Health

npx langfuse-cli api scores list --limit 50

Evaluate:

• Score existence: Are evals running at all?
• Score types: What's being measured (relevance, faithfulness, toxicity, custom)?
• Score distribution: Are scores clustered (all 1.0 = useless eval) or distributed?
• Annotation queues: Are humans reviewing AI outputs?
• Judge LLM: If using LLM-as-judge, which model? Is the judge prompt well-designed?

Red flags:

• No scores at all → no quality feedback loop
• Only manual scores, no automated → quality is not continuously monitored
• All scores are identical → eval criteria too loose or rubric too vague
• Scores declining over time → model degradation or prompt drift

2e. Session and User Attribution

npx langfuse-cli api sessions list --limit 20

Verify:

• Sessions group related interactions (multi-turn conversations have one session ID)
• User IDs are attributed (not all anonymous)
• Session metadata is useful (page, feature, user segment)

2f. Dataset Health

npx langfuse-cli api datasets list

Evaluate:

• Datasets exist: Are there regression test datasets?
• Dataset freshness: When were items last added?
• Coverage: Do datasets cover all AI features or just one?
• Expected outputs: Do dataset items have expected outputs for automated comparison?

---

Phase 3: Live Verification

3a. Trigger AI Features via Playwright

For each AI feature identified in Phase 0c, use playwright-cli to trigger it live.

Important: Apply the protocol-browser-anti-stall protocol — set 15-second timeouts, use the incremental sleep 2 → snapshot cycle rather than one long block, and use snapshot to detect ready state.

PW="npx --yes @playwright/cli@latest"
$PW -s=langfuse-audit open --headed "<APP_URL>"

Navigate to the feature, interact with it (fill form, click button, send message), and capture:

• The AI-generated response (via snapshot)
• Console messages (via console) — look for errors
• Network requests (via requests) — look for failed API calls

3b. Verify Trace Pipeline

After triggering each feature, wait 5-10 seconds, then verify the trace landed:

npx langfuse-cli api traces list --limit 5

Check:

• New trace appeared with correct name
• Trace has generations with model and token data
• Trace latency matches observed UX latency
• Input/output captured correctly

If a trace is missing after triggering a feature → pipeline break (critical finding).

3c. Cross-Check with Sentry

sentry:search_issues
{
 "organizationSlug": "<ORG_SLUG>",
 "projectSlug": "<PROJECT_SLUG>",
 "query": "is:unresolved ai OR llm OR openai OR anthropic OR langfuse OR completion OR embedding"
}

Check for:

• LLM timeout errors
• Rate limiting (429) errors
• Token limit exceeded errors
• Langfuse SDK errors (failed to send trace)
• JSON parse errors on LLM responses

3d. Cross-Check with Supabase (if AI results stored in DB)

If the project stores AI outputs in the database:

supabase:list_tables
{
 "project_id": "<PROJECT_ID>"
}

Find tables that store AI outputs and verify data landed:

supabase:execute_sql
{
 "project_id": "<PROJECT_ID>",
 "query": "SELECT id, created_at, <ai_output_column> FROM <table> ORDER BY created_at DESC LIMIT 5"
}

3e. Grounding / Hallucination Check

For features where the AI should reference source data (RAG, summarization, data extraction):

1. Get the source data from the database (Supabase execute_sql)
2. Trigger the AI feature via Playwright
3. Compare the AI output against the source data

Red flags:

• AI mentions facts not in the source data → hallucination
• AI omits critical facts from the source data → incomplete extraction
• AI contradicts the source data → grounding failure
• AI generates plausible but wrong numbers → numerical hallucination

---

Phase 4: Report

Generate a structured report with the following sections.

═══════════════════════════════════════════════════════
 LANGFUSE LLM QUALITY AUDIT REPORT
 Project: <PROJECT_NAME>
 Date: <DATE>
 Langfuse Host: <HOST_URL>
═══════════════════════════════════════════════════════

## 1. TRACE COVERAGE

| Feature | Traced? | Generations? | Input/Output? | Metadata? | Status |
|---------|---------|-------------|---------------|-----------|--------|
| ... | ... | ... | ... | ... | ✅/❌ |

Coverage: X/Y features traced (Z%)
Missing instrumentation: [list features with no traces]

## 2. PROMPT QUALITY

| Prompt | Version | Label | System Msg | Few-Shot | Guardrails | Variables | Score |
|--------|---------|-------|------------|----------|------------|-----------|-------|
| ... | ... | ... | ... | ... | ... | ... | A-F |

Hardcoded prompts found: [list files with inline prompts]
Recommendations: [specific improvements per prompt]

## 3. COST EFFICIENCY

| Feature | Model | Avg Tokens (in/out) | Avg Latency | Est. Cost/Call | Recommendation |
|---------|-------|---------------------|-------------|----------------|----------------|
| ... | ... | ... | ... | ... | ... |

Monthly estimate: $X (at current usage rate)
Savings opportunity: $Y (by implementing recommendations)

## 4. EVAL HEALTH

| Metric | Status | Details |
|------------------|-----------|----------------------------------|
| Automated evals | ✅/❌ | [count and types] |
| Manual reviews | ✅/❌ | [annotation queue status] |
| Score distribution| ✅/❌ | [healthy spread vs clustered] |
| Datasets | ✅/❌ | [count, freshness, coverage] |
| Regression tests | ✅/❌ | [dataset run frequency] |

## 5. PIPELINE INTEGRITY

| Step | Status | Evidence |
|-------------------------|--------|-------------------------------------|
| FE triggers AI feature | ✅/❌ | [Playwright observation] |
| API receives request | ✅/❌ | [network request captured] |
| LLM call executes | ✅/❌ | [trace generation exists] |
| Trace lands in Langfuse | ✅/❌ | [CLI verification] |
| Result stored in DB | ✅/❌ | [Supabase query result] |
| Result displayed in FE | ✅/❌ | [Playwright snapshot] |
| Eval score recorded | ✅/❌ | [score attached to trace] |

## 6. GROUNDING & HALLUCINATION

| Feature | Source Data | AI Output Match | Hallucinations | Score |
|---------|-------------|-----------------|----------------|-------|
| ... | ... | ... | ... | A-F |

## 7. SENTRY LLM ERRORS

| Issue | Error Type | Events | Impact | Fix Needed |
|-------|------------|--------|--------|------------|
| ... | ... | ... | ... | ... |

## 8. CRITICAL FINDINGS (Action Required)

P0 — Must fix immediately:
1. [finding with evidence]

P1 — Should fix this sprint:
1. [finding with evidence]

P2 — Improvement opportunity:
1. [finding with evidence]

## 9. RECOMMENDATIONS

| # | Category | Current State | Recommended State | Effort | Impact |
|---|----------|---------------|-------------------|--------|--------|
| 1 | ... | ... | ... | S/M/L | S/M/L |

## 10. PDCA IMPROVEMENT RESULTS

| Prompt | Baseline Score | Iter 1 Score | Iter 2 Score | Iter 3 Score | Final Score | Action Taken |
|--------|---------------|-------------|-------------|-------------|-------------|--------------|
| ... | ... | ... | ... | ... | ... | Promoted / Rolled back / Needs manual |

## Further reading

- [Improvement Details and more](references/details.md)