Skip to main content ホーム クリエイター jeremylongshore tons-of-skills-marketplace langfuse-core-workflow-b
langfuse-core-workflow-b Execute Langfuse secondary workflow: Evaluation, scoring, and datasets.
Use when implementing LLM evaluation, adding user feedback,
or setting up automated quality scoring and experiment datasets.
Trigger with phrases like "langfuse evaluation", "langfuse scoring",
"rate llm outputs", "langfuse feedback", "langfuse datasets", "langfuse experiments".
インストールへ移動 Skills Marketplace コミュニティが作成したAIスキルを発見・探索
Codex または Claude でインストール この Prompt をコピーして Codex、Claude、または他のアシスタントに貼り付けると、Skill ページを確認してインストールできます。
直接コマンドでは確認用 Prompt が省略されます。実行前にソースを確認してください。
npx skills add https://github.com/jeremylongshore/tons-of-skills-marketplace --skill langfuse-core-workflow-bコマンドは1行のまま表示されます。コピー前に横へスクロールして全体を確認してください。
ローカルで確認しますか？SkillsMP が現在取得できるファイルをダウンロードできます。
Zipをダウンロード ダウンロード中... このリポジトリの他の Skills langchain-deploy-integration Deploy a LangChain 1.0 / LangGraph 1.0 app to Cloud Run, Vercel, or LangServe correctly — with timeouts sized for chain length, cold-start mitigation, SSE anti-buffering headers, and Secret Manager over .env. Use when prepping a first production deploy, debugging a stream that hangs behind a proxy, or diagnosing p99 latency spikes. Trigger with "langchain deploy", "langchain cloud run", "langchain vercel python", "langchain langserve", or "langchain docker".
langchain-langgraph-agents Build a correct LangGraph 1.0 ReAct agent with create_react_agent — typed tools, error propagation, recursion caps, and stop conditions that actually stop. Use when writing a first tool-calling agent, migrating from AgentExecutor or initialize_agent, or diagnosing an agent that loops on vague prompts. Trigger with "langgraph agent", "create_react_agent", "langgraph tool calling", "AgentExecutor migration", or "agent loop cost".
langchain-langgraph-human-in-loop Build LangGraph 1.0 human-in-the-loop approval flows with interrupt_before /
interrupt_after and Command(resume=...) — JSON-serializable state, clean
resume semantics, and UI wiring for approval decisions. Use when adding an
approval gate before an expensive tool call, wiring a Slack/web UI for agent
approvals, or debugging a graph that crashes on interrupt.
Trigger with "langgraph human in loop", "langgraph interrupt_before",
"langgraph approval flow", "Command resume", "langgraph HITL".
name langfuse-core-workflow-b description Execute Langfuse secondary workflow: Evaluation, scoring, and datasets.
Use when implementing LLM evaluation, adding user feedback,
or setting up automated quality scoring and experiment datasets.
Trigger with phrases like "langfuse evaluation", "langfuse scoring",
"rate llm outputs", "langfuse feedback", "langfuse datasets", "langfuse experiments".
allowed-tools Read, Write, Edit, Bash(npm:*), Grep version 1.12.0 license MIT author Jeremy Longshore <jeremy@intentsolutions.io> tags ["saas","langfuse","llm","workflow","evaluation"] compatibility Designed for Claude Code
Langfuse Core Workflow B: Evaluation, Scoring & Datasets
Overview
Implement LLM output evaluation using Langfuse scores (numeric, categorical, boolean), the experiment runner SDK for dataset-driven benchmarks, prompt management with versioned prompts, and LLM-as-a-Judge evaluation patterns.
Prerequisites
Langfuse SDK configured with API keys
Traces already being collected (see langfuse-core-workflow-a)
For v4+: @langfuse/client installed
Instructions
Step 1: Score Traces via SDK
Langfuse supports three score data types: Numeric , Categorical , and Boolean .
import { LangfuseClient } from "@langfuse/client" ;
const langfuse = new LangfuseClient ();
await langfuse.score .create ({
traceId : "trace-abc-123" ,
name : "relevance" ,
value : 0.92 ,
dataType : "NUMERIC" ,
comment : "Highly relevant answer with good context usage" ,
});
await langfuse.score .create ({
traceId : "trace-abc-123" ,
observationId : "gen-xyz-456" ,
name : "quality-tier" ,
value : "excellent" ,
dataType : "CATEGORICAL"
await
score
create
traceId
"trace-abc-123"
name
"user-approved"
value
1
dataType
"BOOLEAN"
comment
"User clicked thumbs up"
Step 2: User Feedback Collection
app.post ("/api/feedback" , async (req, res) => {
const { traceId, rating, comment } = req.body ;
await langfuse.score .create ({
traceId,
name : "user-feedback" ,
value : rating === "positive" ? 1 : 0 ,
dataType : "BOOLEAN" ,
comment,
});
if (req.body .stars ) {
await langfuse.score .create ({
traceId,
name : "star-rating" ,
value : req.body .stars ,
dataType : "NUMERIC" ,
comment : `${req.body.stars} /5 stars` ,
});
}
res.json ({ success : true });
});
Step 3: Prompt Management
const textPrompt = await langfuse.prompt .get ("summarize-article" , {
type : "text" ,
label : "production" ,
});
const compiled = textPrompt.compile ({
maxLength : "100 words" ,
tone : "professional" ,
});
const chatPrompt = await langfuse.prompt .get ("customer-support" , {
type : "chat" ,
});
const messages = chatPrompt.compile ({
customerName : "Alice" ,
issue : "billing question" ,
});
Step 4: Create and Populate Datasets
await langfuse.api .datasets .create ({
name : "customer-support-v1" ,
description : "Test cases for customer support chatbot" ,
metadata : { version : "1.0" , domain : "support" },
});
const testCases = [
{
input : { query : "How do I cancel my subscription?" },
expectedOutput : { intent : "cancellation" , sentiment : "neutral" },
metadata : { category : "billing" },
},
{
input : { query : "Your product is amazing!" },
expectedOutput : { intent : "feedback" , sentiment : "positive" },
metadata : { category : "feedback" },
},
];
for (const testCase of testCases) {
await langfuse.api .datasetItems .create ({
datasetName : "customer-support-v1" ,
input : testCase.input ,
expectedOutput : testCase.expectedOutput ,
metadata : testCase.metadata ,
});
}
Step 5: Run Experiments with the Experiment Runner import { LangfuseClient } from "@langfuse/client" ;
const langfuse = new LangfuseClient ();
async function classifyIntent (input : { query: string }Promise <string > {
const response = await openai.chat .completions .create ({
model : "gpt-4o-mini" ,
messages : [
{ role : "system" , content : "Classify the user intent. Return one word." },
{ role : "user" , content : input.query },
],
temperature : 0 ,
});
return response.choices [0 ].message .content ?.trim () || "" ;
}
function exactMatch ({ output, expectedOutput }: {
output: string ;
expectedOutput: { intent: string };
} ) {
return {
name : "exact-match" ,
value : output.toLowerCase () === expectedOutput.intent .toLowerCase () ? 1 : 0 ,
dataType : "BOOLEAN" as const ,
};
}
const result = await langfuse.runExperiment ({
datasetName : "customer-support-v1" ,
runName : "gpt-4o-mini-classifier-v1" ,
runDescription : "Testing intent classification with gpt-4o-mini" ,
task : classifyIntent,
evaluators : [exactMatch],
});
console .log (`Experiment complete. ${result.runs.length} items evaluated.` );
Step 6: LLM-as-a-Judge Evaluation async function llmJudge ({ output, input, expectedOutput }: {
output: string ;
input: { query: string };
expectedOutput: { intent: string ; sentiment: string };
} ) {
const judgment = await openai.chat .completions .create ({
model : "gpt-4o" ,
temperature : 0 ,
messages : [
{
role : "system" ,
content : `You are an AI evaluator. Score the response 0-10 on accuracy and helpfulness.
Return JSON: {"score": <number>, "reasoning": "<explanation>"}` ,
},
{
role : "user" ,
content : `Query: ${input.query} \nExpected: ${JSON .stringify(expectedOutput)} \nActual: ${output} ` ,
},
],
response_format : { type : "json_object" },
});
const result = JSON .parse (judgment.choices [0 ].message .content || "{}" );
return {
name : "llm-judge-quality" ,
value : result.score / 10 ,
dataType : "NUMERIC" as const ,
comment : result.reasoning ,
};
}
await langfuse.runExperiment ({
datasetName : "customer-support-v1" ,
runName : "judge-evaluation-v1" ,
task : classifyIntent,
evaluators : [exactMatch, llmJudge],
});
Error Handling Issue Cause Solution Scores not appearing API call failed silently Await score.create() and check for errors Score validation error Wrong data type Match value type to dataType (number/string/0-1) LLM judge inconsistent High temperature Set temperature: 0 for evaluation calls Dataset item missing Wrong dataset name Verify exact name match (case-sensitive) Experiment not in UI Run not flushed Check runExperiment completed without errors
Resources
Next Steps For common error debugging, see langfuse-common-errors. For CI/CD integration of evaluations, see langfuse-ci-integration.
