langchain-agent-evals

name	langchain-agent-evals
description	Evaluate LangChain agent trajectories with AgentEvals trajectory matching or LLM-as-judge, optionally log to LangSmith. Use when scoring tool-call sequences vs a reference, catching prompt or tool regressions, pytest LangSmith runs, or the user mentions agentevals, trajectory match, or evaluators.
disable-model-invocation	true

Documentation Index

Fetch the complete documentation index at: https://docs.langchain.com/llms.txt Use this file to discover all available pages before exploring further.

---

Agent Evals (trajectory evaluation)

Evals score trajectories (message + tool-call sequences), not just final text. They catch regressions when prompts, tools, or models change. Integration tests check basic wiring; evals compare behavior to a reference or rubric.

Custom evaluator shape:

def evaluator(*, outputs: dict, reference_outputs: dict):
    output_messages = outputs["messages"]
    reference_messages = reference_outputs["messages"]
    score = compare_messages(output_messages, reference_messages)
    return {"key": "evaluator_score", "score": score}

Install

pip install agentevals

Optional: AgentEvals repository for source and advanced options (tool_args_match_mode, tool_args_match_overrides).

Trajectory match (create_trajectory_match_evaluator)

Deterministic, no LLM cost. Compares actual messages/tool calls to a reference trajectory.

Mode	Behavior
strict	Same structure and tool calls in order (content may differ)
unordered	Same tool calls as reference, any order
subset	Agent only calls tools from reference (no extras)
superset	Agent calls at least reference tools (extras allowed)

from langchain.agents import create_agent
from langchain_core.messages import AIMessage, HumanMessage, ToolMessage
from langchain_core.tools import tool
from agentevals.trajectory.match import create_trajectory_match_evaluator


@tool
def get_weather(city: str) -> str:
    """Get weather information for a city."""
    return f"It's 75 degrees and sunny in {city}."


agent = create_agent("claude-sonnet-4-6", tools=[get_weather])

evaluator = create_trajectory_match_evaluator(trajectory_match_mode="strict")


def test_weather_tool_called_strict():
    result = agent.invoke(
        {"messages": [HumanMessage(content="What's the weather in San Francisco?")]}
    )

    reference_trajectory = [
        HumanMessage(content="What's the weather in San Francisco?"),
        AIMessage(
            content="",
            tool_calls=[
                {"id": "call_1", "name": "get_weather", "args": {"city": "San Francisco"}}
            ],
        ),
        ToolMessage(
            content="It's 75 degrees and sunny in San Francisco.",
            tool_call_id="call_1",
        ),
        AIMessage(content="The weather in San Francisco is 75 degrees and sunny."),
    ]

    evaluation = evaluator(
        outputs=result["messages"],
        reference_outputs=reference_trajectory,
    )
    # {"key": "trajectory_strict_match", "score": True, "comment": None}
    assert evaluation["score"] is True

outputs / reference_outputs may be LangChain message lists (as above) or OpenAI-style message dicts; see the AgentEvals README for the dict schema.

Use trajectory_match_mode="unordered" | "subset" | "superset" as needed. For argument matching nuances, see the AgentEvals README section on tool args match modes.

LLM-as-judge (create_trajectory_llm_as_judge)

Uses an LLM to score trajectory quality. Reference trajectory optional.

from langchain_core.messages import HumanMessage
from agentevals.trajectory.llm import (
    create_trajectory_llm_as_judge,
    TRAJECTORY_ACCURACY_PROMPT,
)

evaluator = create_trajectory_llm_as_judge(
    model="openai:o3-mini",
    prompt=TRAJECTORY_ACCURACY_PROMPT,
)

def test_trajectory_quality():
    result = agent.invoke(
        {"messages": [HumanMessage(content="What's the weather in Seattle?")]}
    )
    evaluation = evaluator(outputs=result["messages"])
    assert evaluation["score"] is True

With reference: use TRAJECTORY_ACCURACY_PROMPT_WITH_REFERENCE and pass reference_outputs=reference_trajectory.

Async

Async factories: insert async after create_ (e.g. create_async_trajectory_llm_as_judge, create_async_trajectory_match_evaluator). Await the returned evaluator on outputs / reference_outputs.

LangSmith

Set env vars:

export LANGSMITH_API_KEY="your_langsmith_api_key"
export LANGSMITH_TRACING="true"

pytest — use langsmith.testing to log inputs/outputs/reference, then run with LangSmith output flag (see LangSmith pytest docs).

Datasets — Client().evaluate(target_fn, data="dataset_name", evaluators=[...]) expects dataset rows with input {"messages": [...]} and output {"messages": [...]} (shape per LangSmith dataset docs).

Key rules

1. Trajectory match for cheap, deterministic tool-sequence checks; LLM judge for holistic quality when references are fuzzy.
2. Reference trajectories must use valid HumanMessage / AIMessage / ToolMessage pairing (tool_call_id matches tool calls).
3. Pick match mode to match how strict ordering must be (strict vs unordered vs subset/superset).
4. Judge evals cost money — run in CI selectively, like other integration tests.

Related

• No-network agent tests: langchain-agent-unit-testing in this repo.
• Live API tests and markers: langchain-agent-integration-testing in this repo.
• LangSmith pytest details: LangSmith documentation (pytest / evaluate).