langsmith-code-eval

name	langsmith-code-eval
description	Creates code-based evaluators for LangSmith-traced agents. Use when building custom evaluation logic, testing tool usage patterns, or scoring agent outputs programmatically. Triggers on requests to evaluate agents, create evaluators, or run experiments against LangSmith datasets.

LangSmith Code Evaluator Creation

Creates evaluators for LangSmith experiments through structured inspection and implementation.

Prerequisites

• langsmith Python package installed
• LANGSMITH_API_KEY environment variable set (check project's .env file)

Workflow

Copy this checklist and track progress:

Evaluator Creation Progress:
- [ ] Step 1: Gather info from user
- [ ] Step 2: Inspect trace and dataset structure
- [ ] Step 3: Read agent code
- [ ] Step 4: Write evaluator
- [ ] Step 5: Write experiment runner
- [ ] Step 6: Run and iterate

Step 1: Gather Info from User

IMPORTANT: Do NOT search or explore the codebase. Ask the user all of these questions upfront using AskUserQuestion before doing anything else.

Ask the user the following in a single AskUserQuestion call:

1. Python command: How do you run Python in this project? (e.g., python, python3, uv run python, poetry run python)
2. Agent file path: What is the path to your agent file?
3. LangSmith project name: What is your LangSmith project name (where traces are logged)?
4. LangSmith dataset name: What is the name of the dataset to evaluate against?
5. Evaluation goal: What behavior should pass vs fail? Common types:
   • Tool usage: Did the agent call the correct tool?