agentbench-eval

name	agentbench-eval
description	AgentBench evaluation harness for claudemem. Covers pre-indexed repos, experiment conditions, running benchmarks, analyzing results, and managing index archives. Use when working on eval infrastructure, running experiments, or interpreting benchmark results.

AgentBench Evaluation Skill

Run and manage claudemem evaluation experiments against the eth-sri/agentbench benchmark (138 instances, 12 Python repos).

Repo Location

The agentbench repo is a sibling of the claudemem repo:

../agentbench/

All paths below are relative to the agentbench repo root unless noted otherwise.

Data Layout

data/
├── eval-repos/              # 12 cloned repos with pre-built .claudemem/ indexes
│   └── {slug}/.claudemem/   # AST, symbols, vectors, enrichment per repo
├── archives/                # Immutable index snapshots (tar.gz)
│   ├── indexes-20260304-deepseek.tar.gz          # v2 full: 12 repos (1.2GB)
│   └── indexes-20260304-deepseek-11of12.tar.gz   # v2: 11 repos enriched (830MB)
├── eval-cache/              # Runtime sentinel cache
└── eval-generated/          # Generated CLAUDE.md files

Index Specs (v2, 2026-03-04)

• Enrichment model: deepseek/deepseek-v3.2 via OpenRouter
• Embedding model: qwen/qwen3-embedding-8b via OpenRouter
• Total: 12 repos, ~39K symbols, ~36K enrichment docs, ~1.9GB indexes
• Cost: ~$0.25 total via OpenRouter

Repo Inventory

Slug	Files	Symbols	Docs	Size
ansible_ansible	1758	4,597	7,214	176M
getzep_graphiti