skill_spec_version	0.1.0
name	trulens-evaluation-setup
version	1.0.0
description	Configure feedback functions and selectors for TruLens evaluations
tags	["trulens","llm","evaluation","feedback","selectors"]

TruLens Evaluation Setup

Configure feedback functions to evaluate your LLM app's quality.

Interactive Evaluation Selection

Before proceeding, let's determine the right evaluations for your app.

Question 1: What type of app are you building?

Option A: RAG (Retrieval-Augmented Generation)

Your app retrieves documents/chunks from a knowledge base
It generates responses grounded in the retrieved context
Examples: Q&A systems, document search, knowledge assistants

→ Recommended: RAG Triad metrics

Context Relevance
Groundedness
Answer Relevance

Option B: Agent

Your app uses tools to accomplish tasks
It may involve multi-step reasoning or planning
Examples: research agents, coding assistants, task automation

→ Recommended: Agent GPA metrics (continue to Question 2)

Question 2 (Agents only): Does your agent do explicit planning?

Yes, my agent creates plans before executing:

Agent outputs a plan/strategy before taking actions
Agent references its plan during execution

→ Use all Agent GPA metrics:

Logical Consistency
Plan Quality
Plan Adherence
Execution Efficiency
Tool Selection
Tool Calling
Tool Quality

No, my agent acts without explicit planning:

Agent takes actions directly without stating a plan
Agent uses reactive decision-making

→ Use Agent GPA metrics (excluding plan metrics):

Logical Consistency
Execution Efficiency
Tool Selection
Tool Calling
Tool Quality

Question 3: Do you want to add any additional evaluations?

Consider adding these based on your needs:

Evaluation	Use Case
Coherence	Check if output is well-structured and readable
Conciseness	Ensure responses aren't unnecessarily verbose
Harmlessness	Detect potentially harmful content
Sentiment	Analyze emotional tone of responses
Custom metrics	Domain-specific evaluations (see below)

Creating Custom Metrics

If you need domain-specific evaluations, describe what you want to measure:

What aspect of your app do you want to evaluate?

Examples:

"Check if the response follows our brand voice guidelines"
"Verify the output contains required legal disclaimers"
"Measure technical accuracy for code generation"
"Evaluate if customer support responses show empathy"

Template for custom metrics:

def my_custom_metric(input_text: str, output_text: str) -> float:
    """
    Describe what this metric evaluates.

    Returns:
        float: Score between 0.0 (worst) and 1.0 (best)
    """
    # Option 1: Rule-based logic
    # score = 1.0 if "required phrase" in output_text else 0.0

    # Option 2: Use LLM-as-judge
    # provider = OpenAI()
    # response = provider.client.chat.completions.create(
    #     model="gpt-4o",
    #     messages=[{
    #         "role": "user",
    #         "content": f"Rate this response on [YOUR CRITERIA]. Input: {input_text} Output: {output_text}. Return only a number 0-10."
    #     }]
    # )
    # score = float(response.choices[0].message.content) / 10.0

    return score

f_custom = Metric(
    implementation=my_custom_metric,
    name="My Custom Metric",
    selectors={
        "input_text": Selector.select_record_input(),
        "output_text": Selector.select_record_output(),
    },
)

Custom metric with context:

def custom_with_context(query: str, context: str, response: str) -> float:
    """Evaluate using query, retrieved context, and response."""
    # Your evaluation logic
    return score

f_custom_context = Metric(
    implementation=custom_with_context,
    name="Custom Context Metric",
    selectors={
        "query": Selector.select_record_input(),
        "context": Selector.select_context(collect_list=True),
        "response": Selector.select_record_output(),
    },
)

Tell me what you want to evaluate and I'll help you create the metric!

Overview

Feedback functions evaluate specific aspects of your app by:

Selecting data from instrumented spans (inputs, outputs, retrieved contexts)
Applying evaluation logic (LLM-as-judge, similarity metrics, etc.)
Returning scores between 0.0 and 1.0

Prerequisites

pip install trulens trulens-providers-openai

Instructions

Step 1: Initialize a Feedback Provider

from trulens.providers.openai import OpenAI

provider = OpenAI(model_engine="gpt-4o")

Step 2: Create Feedback Functions with Selector Shortcuts

TruLens provides shortcuts for common selection patterns:

from trulens.core import Metric, Selector

# Answer relevance: input → output
f_answer_relevance = Metric(
    implementation=provider.relevance_with_cot_reasons,
    name="Answer Relevance",
    selectors={
        "prompt": Selector.select_record_input(),
        "response": Selector.select_record_output(),
    },
)

# Context relevance: input → each context chunk
f_context_relevance = Metric(
    implementation=provider.context_relevance_with_cot_reasons,
    name="Context Relevance",
    selectors={
        "question": Selector.select_record_input(),
        "context": Selector.select_context(collect_list=False),
    },
)

# Groundedness: all contexts → output
f_groundedness = Metric(
    implementation=provider.groundedness_measure_with_cot_reasons,
    name="Groundedness",
    selectors={
        "source": Selector.select_context(collect_list=True),
        "statement": Selector.select_record_output(),
    },
)

Shortcut Reference:

Shortcut	Selects	Required Span Type
`on_input()`	App input	`RECORD_ROOT`
`on_output()`	App output	`RECORD_ROOT`
`on_context()`	Retrieved contexts	`RETRIEVAL`

⚠️ IMPORTANT: .on_input() and .on_output() require RECORD_ROOT spans!

These shortcuts look for spans with span_type=SpanAttributes.SpanType.RECORD_ROOT. If you use manual instrumentation with a different span type (like AGENT), the shortcuts will not find any data.

Solutions:

Use framework wrappers (TruGraph, TruChain, TruLlama) which create RECORD_ROOT automatically
Use explicit @instrument(span_type=SpanAttributes.SpanType.RECORD_ROOT, ...) on your entry point
Use explicit Selector objects instead of shortcuts (see Step 3)

Step 3: Using Explicit Selectors

For more control, use Selector to target specific span attributes:

from trulens.core import Metric
from trulens.core.feedback.selector import Selector
from trulens.otel.semconv.trace import SpanAttributes

f_answer_relevance = Metric(
    implementation=provider.relevance_with_cot_reasons,
    name="Answer Relevance",
    selectors={
        "prompt": Selector(
            span_type=SpanAttributes.SpanType.RECORD_ROOT,
            span_attribute=SpanAttributes.RECORD_ROOT.INPUT,
        ),
        "response": Selector(
            span_type=SpanAttributes.SpanType.RECORD_ROOT,
            span_attribute=SpanAttributes.RECORD_ROOT.OUTPUT,
        ),
    },
)

Step 4: Understanding collect_list

The collect_list parameter controls how multiple values are handled:

Setting	Behavior	Use Case
`collect_list=False`	Evaluate each value individually	Context relevance (score each chunk)
`collect_list=True`	Concatenate all values	Groundedness (check against all context)

# Evaluate each retrieved context individually (returns multiple scores)
f_context_relevance = Metric(
    implementation=provider.context_relevance_with_cot_reasons,
    name="Context Relevance",
    selectors={
        "question": Selector.select_record_input(),
        "context": Selector(
            span_type=SpanAttributes.SpanType.RETRIEVAL,
            span_attribute=SpanAttributes.RETRIEVAL.RETRIEVED_CONTEXTS,
            collect_list=False,
        ),
    },
)

# Evaluate against all contexts combined (returns single score)
f_groundedness = Metric(
    implementation=provider.groundedness_measure_with_cot_reasons,
    name="Groundedness",
    selectors={
        "source": Selector(
            span_type=SpanAttributes.SpanType.RETRIEVAL,
            span_attribute=SpanAttributes.RETRIEVAL.RETRIEVED_CONTEXTS,
            collect_list=True,
        ),
        "statement": Selector.select_record_output(),
    },
)

Step 5: Aggregating Multiple Scores

When collect_list=False produces multiple scores, aggregate them:

import numpy as np

f_context_relevance = Metric(
    implementation=provider.context_relevance_with_cot_reasons,
    name="Context Relevance",
    selectors={
        "question": Selector.select_record_input(),
        "context": Selector.select_context(collect_list=False),
    },
    agg=np.mean,
)

Common aggregation functions:

np.mean - Average score
np.min - Worst score (conservative)
np.max - Best score (optimistic)

Common Patterns

RAG Triad Setup

import numpy as np
from trulens.core import Metric, Selector
from trulens.providers.openai import OpenAI

provider = OpenAI()

# Context Relevance: Is each retrieved chunk relevant to the query?
f_context_relevance = Metric(
    implementation=provider.context_relevance_with_cot_reasons,
    name="Context Relevance",
    selectors={
        "question": Selector.select_record_input(),
        "context": Selector.select_context(collect_list=False),
    },
    agg=np.mean,
)

# Groundedness: Is the response grounded in the retrieved context?
f_groundedness = Metric(
    implementation=provider.groundedness_measure_with_cot_reasons,
    name="Groundedness",
    selectors={
        "source": Selector.select_context(collect_list=True),
        "statement": Selector.select_record_output(),
    },
)

# Answer Relevance: Does the response answer the original question?
f_answer_relevance = Metric(
    implementation=provider.relevance_with_cot_reasons,
    name="Answer Relevance",
    selectors={
        "prompt": Selector.select_record_input(),
        "response": Selector.select_record_output(),
    },
)

rag_feedbacks = [f_context_relevance, f_groundedness, f_answer_relevance]

Agent GPA Setup

from trulens.core import Metric, Selector
from trulens.providers.openai import OpenAI

provider = OpenAI()

# Logical Consistency
f_logical_consistency = Metric(
    implementation=provider.logical_consistency_with_cot_reasons,
    name="Logical Consistency",
    selectors={"trace": Selector(trace_level=True)},
)

# Plan Quality (exclude if agent doesn't do explicit planning)
f_plan_quality = Metric(
    implementation=provider.plan_quality_with_cot_reasons,
    name="Plan Quality",
    selectors={"trace": Selector(trace_level=True)},
)

# Plan Adherence (exclude if agent doesn't do explicit planning)
f_plan_adherence = Metric(
    implementation=provider.plan_adherence_with_cot_reasons,
    name="Plan Adherence",
    selectors={"trace": Selector(trace_level=True)},
)

# Execution Efficiency
f_execution_efficiency = Metric(
    implementation=provider.execution_efficiency_with_cot_reasons,
    name="Execution Efficiency",
    selectors={"trace": Selector(trace_level=True)},
)

# Tool Selection
f_tool_selection = Metric(
    implementation=provider.tool_selection_with_cot_reasons,
    name="Tool Selection",
    selectors={"trace": Selector(trace_level=True)},
)

# Tool Calling
f_tool_calling = Metric(
    implementation=provider.tool_calling_with_cot_reasons,
    name="Tool Calling",
    selectors={"trace": Selector(trace_level=True)},
)

# Tool Quality
f_tool_quality = Metric(
    implementation=provider.tool_quality_with_cot_reasons,
    name="Tool Quality",
    selectors={"trace": Selector(trace_level=True)},
)

# Use all for agents with planning
agent_feedbacks_with_planning = [
    f_logical_consistency,
    f_plan_quality,
    f_plan_adherence,
    f_execution_efficiency,
    f_tool_selection,
    f_tool_calling,
    f_tool_quality,
]

# For agents without explicit planning, exclude plan metrics
agent_feedbacks_no_planning = [
    f_logical_consistency,
    f_execution_efficiency,
    f_tool_selection,
    f_tool_calling,
    f_tool_quality,
]

Custom Feedback Function

def my_custom_metric(input_text: str, output_text: str) -> float:
    """Custom evaluation returning score between 0.0 and 1.0."""
    # Your evaluation logic here
    score = len(output_text) / (len(input_text) + len(output_text))
    return min(max(score, 0.0), 1.0)

f_custom = Metric(
    implementation=my_custom_metric,
    name="Custom Metric",
    selectors={
        "input_text": Selector.select_record_input(),
        "output_text": Selector.select_record_output(),
    },
)

Troubleshooting

Selector not finding data: Ensure the span attribute was set during instrumentation
Empty context: Verify RETRIEVAL.RETRIEVED_CONTEXTS is mapped in your @instrument() decorator
Aggregation errors: Check that collect_list=False is set when using .aggregate()
.on_input()/.on_output() returning no data: These shortcuts require RECORD_ROOT span type. Use framework wrappers or explicit @instrument(span_type=SpanAttributes.SpanType.RECORD_ROOT, ...). See the instrumentation skill for details.
Feedback columns show empty/null: Verify your instrumentation creates RECORD_ROOT spans with INPUT and OUTPUT attributes

trulens-evaluation-setup

このリポジトリの他の Skills

このリポジトリの他の Skills