CLI tool for capturing agent trajectories. Execute prompts against headless CLI agents via schema-driven adapters, capture full trajectories (tools, thoughts, plans), and output structured JSONL for downstream scoring.
CLI tool for capturing agent trajectories. Execute prompts against headless CLI agents via schema-driven adapters, capture full trajectories (tools, thoughts, plans), and output structured JSONL for downstream scoring.
compatibility
Bun >= 1.2.9
Agent Eval Harness
Purpose
CLI tool for capturing trajectories from headless CLI agents, optimized for TypeScript/JavaScript projects using Bun.
The harness captures. You score.
Harness Provides
You Provide
Prompt execution via headless adapters
Scoring logic (Braintrust, custom scripts)
Full trajectory capture (thoughts, tools, plans)
Pass/fail determination via graders
Structured JSONL output
LLM-as-judge prompts
Reproducible execution environment
CI integration, golden file comparison
Use this when:
Capturing trajectories for downstream evaluation
Generating training data (SFT/DPO) with full context
Building regression test fixtures for agent behavior
Comparing agent responses across configurations
Installation
# Run without installing (recommended)
bunx @plaited/agent-eval-harness capture prompts.jsonl --schema ./claude.json -o results.jsonl
# Or install as project dependency
bun add @plaited/agent-eval-harness
Single output format: Full trajectory JSONL (always)
No --format flag: Derive views with separate commands
Schema exports: Zod schemas + JSON Schema for any tooling
Commands
Core Commands
Command
Input
Output
Purpose
capture
prompts.jsonl + schema
results.jsonl
Trajectory capture (full)
trials
prompts.jsonl + schema
trials.jsonl
Multi-run + optional metrics
summarize
results.jsonl
summary.jsonl or .md
Derive compact views
calibrate
results.jsonl
calibration.md
Sample failures for review
validate-refs
prompts.jsonl
validation.jsonl
Check reference solutions
balance
prompts.jsonl
balance.json
Analyze test set coverage
schemas
(none)
JSON Schema
Export schemas for non-TS users
Pipeline Commands (Unix-style)
Command
Input
Output
Purpose
run
prompts.jsonl + schema
raw.jsonl
Execute prompts, raw output
extract
raw.jsonl + schema
extracted.jsonl
Parse trajectories
grade
extracted.jsonl + grader
graded.jsonl
Apply grader scoring
format
results.jsonl
jsonl/markdown/csv
Convert output format
compare
multiple results.jsonl
comparison.json
Compare runs (aggregate report)
All commands support optional --grader ./grader.ts for scoring.
Workspace cleanup:
Directories persist after completion for debugging. Clean up manually:
# After capturerm -rf ./workspaces
# In CI (add as post-step)
- run: rm -rf ./workspaces
if: always()
Output
Without grader:
{"id":"search-001","input":"Find the CEO","k":5,"trials":[{"trialNum":1,"output":"...","trajectory":[...],"duration":1234},...]}
With grader:
{"id":"search-001","input":"Find the CEO","k":5,"passRate":0.8,"passAtK":0.99,"passExpK":0.33,"trials":[{"trialNum":1,"output":"...","pass":true,"score":1.0},...]}
Summarize Command
Derive compact views from full trajectory results.
Sample failures for grader review. Calibration helps you distinguish between agent failures (agent did wrong thing) and grader bugs (agent was correct, grader too strict).
# Sample failures for human review
bunx @plaited/agent-eval-harness calibrate results.jsonl --sample 10 -o calibration.md
# Re-score with different grader to compare
bunx @plaited/agent-eval-harness calibrate results.jsonl --grader ./loose-grader.ts --sample 10 -o comparison.md
An eval with only "make X work" misses "don't break Y". Balance analysis shows:
Category distribution (from metadata.category)
Positive/negative case ratio
Coverage gaps
Output Format
{"totalCases":50,"categories":[{"name":"ui","count":20,"percentage":40},{"name":"logic","count":15,"percentage":30},{"name":"api","count":10,"percentage":20},{"name":"edge-case","count":5,"percentage":10}],"underrepresented":["edge-case"],"suggestions":["Consider adding more test cases for: edge-case"]}
⚠️ Security Warning: The --simple and --shell modes execute prompts via shell commands. Prompts are escaped but do not use untrusted prompt content with these modes. Malicious prompt text could potentially escape the quoting and execute arbitrary commands. Use --schema mode (headless adapter) for untrusted inputs.
Extract Command
Parse raw output into structured trajectories:
# From file
bunx @plaited/agent-eval-harness extract raw.jsonl --schema claude.json -o extracted.jsonl
# Piped from run
bunx @plaited/agent-eval-harness run prompts.jsonl -s claude.json | \
bunx @plaited/agent-eval-harness extract -s claude.json
# Markdown report
bunx @plaited/agent-eval-harness format results.jsonl --style markdown -o report.md
# CSV for spreadsheets
bunx @plaited/agent-eval-harness format results.jsonl --style csv -o results.csv
# JSONL (pass-through, default)
bunx @plaited/agent-eval-harness format results.jsonl --style jsonl
Compare Command
Compare multiple runs of the same prompts. Supports both CaptureResult (single-run) and TrialResult (multi-run reliability) formats with auto-detection.
Export schemas for validation in Python, Go, etc.:
# Export all schemas
bunx @plaited/agent-eval-harness schemas --json -o schemas.json
# Use in Python with jsonschema
python -c "
import json
from jsonschema import validate
with open('schemas.json') as f:
schemas = json.load(f)
with open('results.jsonl') as f:
for line in f:
result = json.loads(line)
validate(result, schemas['CaptureResult'])
print(f'{result[\"id\"]}: valid')
"
Grader Interface
Graders provide semantic pass/fail scoring for captured trajectories. The harness supports graders written in any language.
Git-Based Grading (Recommended for Coding Tasks)
Grade outcomes, not paths. Use the optional cwd parameter to detect environmental changes with git:
See inline-graders.md for complete grader documentation including LLM-as-Judge patterns.
Input Format
Each line in prompts.jsonl:
{"id":"test-001","input":"Create a button","hint":"should contain <button>"}
{"id":"test-002","input":["Create a button","Make it blue"],"metadata":{"category":"ui"}}
Field
Required
Description
id
Yes
Unique identifier
input
Yes
Single prompt (string) or conversation turns (string[])
hint
No
Grader context - what to look for (not strict match)
reference
No
Reference solution (for validate-refs)
metadata
No
Tags, category, difficulty for filtering
timeout
No
Override default timeout for this prompt
Session behavior: Each JSONL entry = 1 fresh session
input: string → 1 session, 1 prompt
input: string[] → 1 session, N prompts (sequential turns)
Output Format
Full trajectory JSONL (always):
{
"id": "test-001",
"input": "Find the CEO of Anthropic",
"output": "The CEO of Anthropic is Dario Amodei.",
"hint": "should mention Dario Amodei",
"trajectory": [
{"type": "thought", "content": "I'll search for this...", "timestamp": 100},
{"type": "tool_call", "name": "WebSearch", "status": "completed", "input": {...}, "output": {...}, "duration": 500},
{"type": "message", "content": "The CEO of Anthropic is Dario Amodei.", "timestamp": 700}
],
"metadata": {
"category": "search",
"agent": "--schema ./claude.json",
"trajectoryRichness": "full",
"turnCount": 1
},
"timing": {
"start": 1704067200000,
"end": 1704067201234,
"firstResponse": 100,
"sessionCreation": 234,
"total": 1234,
"inputTokens": 150,
"outputTokens": 85
},
"toolErrors": false
}
Output Fields
Field
Description
input
Original prompt (string or string[] for multi-turn)
hint
Grader context hint (if provided)
metadata.trajectoryRichness
"full" | "messages-only" | "minimal"
metadata.turnCount
Number of conversation turns (1 for string, N for array)
timing.sessionCreation
Time to create session (ms)
timing.total
Total duration (end - start)
timing.inputTokens
Input tokens consumed (if available from adapter)
timing.outputTokens
Output tokens generated (if available from adapter)
toolErrors
Whether any tool calls failed
Note:toolErrors replaces misleading status: 'passed'|'failed'. Real pass/fail comes from YOUR grader.
Recommendation: Run the harness in Docker containers for consistent, isolated execution.
# Run integration tests via Docker
docker compose -f docker-compose.test.yml run --rmtest# Or with explicit API keys
ANTHROPIC_API_KEY=sk-... GEMINI_API_KEY=... docker compose -f docker-compose.test.yml run --rmtest
Docker Requirements
Requirement
Reason
Node.js 24+
Gemini CLI uses modern JS features (optional chaining)
Non-root user
Claude CLI blocks --dangerously-skip-permissions as root
Gemini API key
Pass GEMINI_API_KEY for Gemini CLI
See docker-evals.md for complete Docker setup guide, debugging tips, and CI integration patterns.
Multi-turn Conversations
Use input: string[] to execute multi-turn conversations within a single session:
{"id":"context-001","input":["Remember this number: 42","What number did I ask you to remember?"],"hint":"42"}
{"id":"context-002","input":["My name is Alice","What is my name?"],"hint":"Alice"}
Run with the headless adapter:
# Using Claude Code via headless adapter
bunx @plaited/agent-eval-harness capture multi-turn.jsonl \
bunx @plaited/agent-eval-harness headless --schema ./claude-headless.json \
-o results.jsonl
# Using Gemini CLI via headless adapter
GEMINI_API_KEY=... bunx @plaited/agent-eval-harness capture multi-turn.jsonl \
bunx @plaited/agent-eval-harness headless --schema ./gemini-headless.json \
-o results.jsonl
Key points:
Each JSONL entry = 1 fresh session
input: string[] sends sequential turns to the same session
Works with both stream mode (Claude) and iterative mode (Gemini)
The adapter handles context preservation automatically
Downstream Integration
The harness outputs standard JSONL that pipes to any tool:
