| name | benchmarking |
| description | This skill should be used when the user asks to "run the benchmark", "run locomo", "benchmark ragzoom", "evaluate recall quality", "compare with letta", "check accuracy", "run the evaluation harness", "agentic evaluation", "max iterations", or mentions benchmarking, evaluation, or competitive comparison of the memory system. |
RagZoom Benchmarking
Run and interpret memory system benchmarks to measure RagZoom's recall quality against the competitive landscape.
Quick Start: Running LoCoMo
LoCoMo is the de facto standard benchmark for conversational memory systems. Every competitor reports numbers here.
Dataset location: test_data/locomo10.json (10 conversations, ~1500 QA pairs)
Prerequisites
- Source the
.env file: set -a && source .env && set +a
- Dataset at
test_data/locomo10.json (already in the repo)
Running a Benchmark
Benchmarks use an isolated server by default (port 50053, state in /tmp/ragzoom-bench-state), completely separated from dev and production servers.
set -a && source .env && set +a
PYTHONPATH=. python scripts/run-locomo --data test_data/locomo10.json
Ingestion takes ~5 minutes (summarization tree building). Evaluation takes ~5 minutes at concurrency=10.
To opt out of the isolated server (e.g. to target a custom --server address), pass --no-isolated-server.
Re-running After Non-Index Changes
When you change only the search agent (prompts, model, config, scoring) — anything that doesn't affect the ingested index — use --skip-ingest to reuse the existing server and data:
PYTHONPATH=. python scripts/run-locomo --data test_data/locomo10.json --skip-ingest --sample 50 --profiling
This skips ingestion entirely and just re-runs evaluation against the existing index. The server stays alive between runs — do not manually kill port 50053 between --skip-ingest runs.
When to re-ingest (omit --skip-ingest):
- Changes to ingestion logic, summarization, or tree building
- Changes to the dataset file
- First run after clearing state
When to use --skip-ingest:
- Search agent prompt changes
- Model swaps (
--search-model)
- Search config changes (iterations, budget)
- Scoring/judge changes
Server Lifecycle
The benchmark server is persistent — it stays alive after a run so you can do --skip-ingest follow-ups. The server manager handles everything automatically:
- Without
--skip-ingest: start_fresh() kills any existing server, wipes /tmp/ragzoom-bench-state, starts a new server, then ingests.
- With
--skip-ingest: verify_running() checks the server is healthy, then runs evaluation only.
CRITICAL: NEVER use pkill -f ragzoom or other broad patterns. This will kill the production daemon on port 50051, breaking the MCP server, stop hooks, and CLI.
If a crashed run left a stale server, start_fresh() (i.e. running without --skip-ingest) will clean it up automatically. Manual cleanup is only needed if start_fresh itself fails:
lsof -ti :50053 | xargs kill -9 2>/dev/null
rm -rf /tmp/ragzoom-bench-state
Key CLI Parameters
| Flag | Default | Purpose |
|---|
--search-model MODEL | gpt-5-mini | LLM for agentic search (OpenAI or Anthropic) |
--judge-model MODEL | gpt-4.1 | LLM-as-Judge (matches Letta leaderboard) |
--sample N | all | Random subset of N questions (seed=42) |
--max-iterations N | 5 | Max recall iterations per question |
--max-budget N | 20000 | Max token budget per recall call |
--f1-only | off | Skip LLM judge, token F1 only |
--rejudge PATH | — | Re-judge from previous results.json |
--no-isolated-server | off | Disable isolated server, use --server address |
--skip-ingest | off | Skip ingestion (docs already indexed) |
--reasoning-level LEVEL | auto | Reasoning effort for search model (none/minimal/low/medium/high) |
--docker-cli | auto | Force Docker CLI (auto-enabled for Claude models) |
--no-docker-cli | off | Disable Docker CLI even for Claude models |
--profiling | off | Search profiling (retrospective per question) |
-v | off | Verbose logging |
Cheap Iteration Modes
Three modes to reduce benchmark cost from ~$22 to <$1 or $0:
Sample mode (--sample N): Evaluate a random subset of N questions (seed=42 for reproducibility).
PYTHONPATH=. python scripts/run-locomo --data test_data/locomo10.json --sample 20
F1-only mode (--f1-only): Skip the LLM judge entirely, compute token F1 only. No judge API costs.
PYTHONPATH=. python scripts/run-locomo --data test_data/locomo10.json --f1-only
Rejudge mode (--rejudge PATH): Re-run the LLM judge on previously cached answers. No RagZoom server needed.
PYTHONPATH=. python scripts/run-locomo --data test_data/locomo10.json --rejudge locomo_results/results.json
Modes combine: --sample 5 --f1-only is the cheapest way to smoke-test evaluation.
Apples-to-Apples Comparison with Letta Leaderboard
Critical: Use Letta's exact methodology for comparable numbers.
| Setting | Letta Leaderboard | Our Defaults |
|---|
| Judge model | gpt-4.1 | gpt-4.1 |
| Judge prompt | 3-way GRADER_TEMPLATE (A/B/C) | Same (adopted from Letta) |
| Answer model | GPT-4o-mini (via Letta agent) | gpt-4o-mini |
| Retrieval | Full conversation via file search | Token-budgeted hierarchical |
| Scoring | A=1.0, B/C=0.0 | Same |
The judge rubric (in scoring.py) is Letta's exact GRADER_TEMPLATE with 3-way grading:
- A (CORRECT): Contains key facts, no contradictions. Hedging OK. Typos OK. Inferable info can be omitted.
- B (INCORRECT): Contains factual contradictions, even with hedging.
- C (NOT_ATTEMPTED): Missing key info but no contradictions ("I don't know").
For detailed comparison methodology, see references/letta-comparison.md.
Interpreting Results
Accuracy and F1
The report outputs aggregate scores (overall + per-category accuracy and F1):
- Accuracy: Judge verdict A=1.0, B/C=0.0 (only present with a judge, not in
--f1-only mode)
- Token F1: Token-level overlap between generated and gold answer
Agent Cost Summary
The markdown report includes an Agent Cost Summary with average retrieval calls, reasoning turns, and input/output/retrieved tokens per question. Compare across models/configs to assess the cost-accuracy tradeoff.
Category Breakdown
| Category | Tests | RagZoom's Challenge |
|---|
| Single-hop | One specific fact | Summaries may lose the exact detail |
| Multi-hop | Connect facts across turns | Hardest — needs multiple specific nodes |
| Temporal | Time-based reasoning | Timestamps lost in summarization |
| Open-domain | Broad understanding | Best fit for hierarchical summaries |
Diagnostic Signals
- High NOT_ATTEMPTED rate: Retrieval not surfacing relevant content. Need higher budget or better seed selection.
- High INCORRECT rate: Model hallucinating from summaries. Summaries may be misleading.
- Low F1 but high judge accuracy: Judge is lenient (semantic matching), F1 is strict (token overlap). Expected divergence.
Architecture
ragzoom/agent/ # Model-agnostic agent layer (shared by search + evaluation)
├── protocol.py # BenchmarkingAgent protocol, CostMetrics, ToolDefinition
├── factory.py # create_backend() — routes to OpenAI or Anthropic
└── backends/
├── openai.py # OpenAI function-calling agent loop
└── claude_agent_sdk.py # Claude Agent SDK backend
ragzoom/search/ # Production search agent
├── agent.py # SearchAgent — uses BenchmarkingAgent backend
├── retrospective.py # Self-critique via backend (profiling only)
├── config.py # SearchConfig (model, iterations, budget)
└── prompt.py # System prompt + retrospective prompt
ragzoom/evaluation/locomo/ # Benchmark harness
├── types.py # Data types: AnswerResult, AggregateScores, BenchmarkReport
├── ingest.py # Conversation → AppendUnit ingestion
├── scoring.py # Token F1 + Letta GRADER_TEMPLATE judge
├── runner.py # Orchestration: ingest → evaluate via agentic search → aggregate
├── report.py # JSON + Markdown output with cost metrics
└── agent/
└── prompt.py # Benchmark-specific system prompt
CLI entry point: scripts/run-locomo
Common Issues
"No module named ragzoom.evaluation.locomo"
Production ragzoom is installed non-editable. Use PYTHONPATH=. to pick up local code.
Stale server process blocking port 50053
Running without --skip-ingest automatically kills and restarts the server. If that fails:
lsof -ti :50053 | xargs kill -9 2>/dev/null
rm -rf /tmp/ragzoom-bench-state
API key not propagating
source .env doesn't export. Use: set -a && source .env && set +a
Agent zoom errors
If the agent zooms into a time range with no content, rz.query may error. The backend handles this gracefully by returning the error as a tool result, letting the agent try a different approach.
Additional Resources
Reference Files
references/letta-comparison.md — Detailed Letta Leaderboard methodology, their exact grader prompt, and how to ensure apples-to-apples comparisonreferences/dataset-format.md — LoCoMo JSON format gotchas discovered during implementationreferences/competitive-landscape.md — Summary of competing systems and benchmarks. Full research at docs/benchmarking/competitive-landscape.md
Strategy Documents
docs/benchmarking/strategy.md — Multi-objective optimization framework, Pareto frontiers, parameter space designdocs/benchmarking/TODO.md — Roadmap for model coverage, parameter sweep framework, duration tracking
