langsmith-run-evals

name	langsmith_run_evals
description	Guidance for writing run_evals.py scripts that use LangSmith evaluate() with a dataset ID.

Overview

This skill provides guidance on writing a run_evals.py script that runs evaluations using LangSmith's evaluate() function with a pre-existing dataset.

Before Writing run_evals.py

Always ask the user for:

1. Evaluator/scoring function: What metric to use (e.g., exact match on label, custom scorer, LLM-as-judge)
2. Experiment naming conventions: How to name experiments for tracking

Key Components of run_evals.py

1. Load the prompt from prompt.txt - The prompt file should be read and used in the target function.

2. Use a hardcoded dataset ID - Do NOT create a new dataset on each run. Use the dataset ID from the one-time upload script.

3. Define a target function - This is the function that will be evaluated. It takes an input dict and returns an output dict.

4. Define an evaluator - This scores the target function's output against the expected output.

5. Call langsmith.evaluate() - Pass the target function, dataset ID, and evaluators.

Anthropic Model Names (Claude 4.5 family)

When using Anthropic models via langchain_anthropic, use these model IDs:

• claude-sonnet-4-5 - Best balance of intelligence, speed, and cost
• claude-haiku-4-5 - Fastest model with near-frontier intelligence
• claude-opus-4-5 - Premium model with maximum intelligence

Reference: https://docs.anthropic.com/en/docs/about-claude/models/all-models

Example Structure

from langsmith import Client, evaluate
from langchain_openai import ChatOpenAI

# Config
DATASET_ID = "your-dataset-id-here"
PROMPT_FILE = "prompt.txt"

# Load prompt
with open(PROMPT_FILE) as f:
    prompt = f.read()

# Initialize model (OpenAI example)
model = ChatOpenAI(model="gpt-4o", temperature=0)

# Or for Anthropic:
# from langchain_anthropic import ChatAnthropic
# model = ChatAnthropic(model="claude-haiku-4-5", temperature=0)

# Target function - takes inputs dict, returns outputs dict
def target(inputs: dict) -> dict:
    # Use the prompt and inputs to generate output
    response = model.invoke([
        {"role": "system", "content": prompt},
        {"role": "user", "content": inputs["text"]},
    ])
    return {"label": response.content}

# Evaluator - compares output to expected
def exact_match(outputs: dict, reference_outputs: dict) -> dict:
    match = outputs.get("label") == reference_outputs.get("label")
    return {"score": int(match), "key": "exact_match"}

# Run evaluation
results = evaluate(
    target,
    data=DATASET_ID,
    evaluators=[exact_match],
    experiment_prefix="my-experiment",
)

print(f"Results: {results.aggregate_metrics}")

Best Practices

• Keep dataset creation separate from evaluation (use create_dataset.py for upload, run_evals.py for evaluation)
• Use structured output when the model needs to return specific formats
• Save results locally (CSV) in addition to LangSmith for easy analysis
• Use descriptive experiment prefixes to track iterations
• Always use max_concurrency=5 when calling evaluate() by default

CRITICAL: Parsing evaluate() Results

The evaluate() function returns an iterator of result dicts. Each result has this structure:

for result in results:
    # Result is a dict with these keys:
    run = result["run"]           # RunTree object with outputs
    example = result["example"]   # Example object with inputs/outputs
    eval_results = result["evaluation_results"]  # Dict with "results" list
    
    # Get model outputs from the RUN, not from result directly
    outputs = run.outputs  # {"predicted_outcome": "...", "full_response": "..."}
    
    # Get example data
    example_id = example.id
    inputs = example.inputs       # {"transcript": "..."}
    ref_outputs = example.outputs # {"label": "...", "expected_outcome": "..."}
    
    # Get evaluation scores
    for er in eval_results.get("results", []):
        if er.key == "accuracy":
            score = er.score

WRONG (common mistake):

outputs = result.get("outputs")  # Returns None!
outputs = result.outputs         # AttributeError!

CORRECT:

outputs = result["run"].outputs  # Access via the run object

Limiting Examples for Testing

When iterating on run_evals.py to make sure it works correctly, limit to a small number of examples (e.g., 5) to save time and cost. Only run the full dataset once the script is confirmed working.

To limit examples, use client.list_examples() with a limit:

from langsmith import Client, evaluate

client = Client()
MAX_EXAMPLES = 5  # Set to None for full dataset

if MAX_EXAMPLES:
    data = list(client.list_examples(dataset_id=DATASET_ID, limit=MAX_EXAMPLES))
else:
    data = DATASET_ID

results = evaluate(
    target,
    data=data,
    evaluators=[exact_match],
    experiment_prefix="my-experiment",
    max_concurrency=5,
)

Common Evaluators

• Exact match: Compare output label to expected label exactly
• Contains: Check if expected value is contained in output
• LLM-as-judge: Use another LLM to score the output
• Custom metrics: F1 score, precision, recall for classification tasks