| name | build-memory-eval |
| description | Prove a memory system earns its write path — golden set with provenance, judged failure modes, the runnable bake-off, and cost per correct answer. Reach for this before adopting, replacing or retiring any memory system. |
Skill: Build a memory eval
Decision 6 of the six-decision spine — how would you know it works? There is no trustworthy public leaderboard for agent memory. That is not a reason to skip the measurement; it is the reason this skill ships a procedure you run yourself.
Every published memory-system ranking is self-reported by its vendor or reported by a competitor, none is independently corroborated, and no neutral head-to-head was located (§3 #8). So the deliverable here is not a citation. It is a bake-off on your data, with your queries, at your budget.
cost-per-correct is shared with budget-memory-costs and owned by neither skill. This one supplies the accuracy half; that one supplies the cost half. Neither is the mode's home, and a reader landing here must not conclude the cost half lives somewhere else.
Part A — The golden set
Step 1 — Build it before you build the memory system
A memory project with no golden set cannot tell you whether it improved anything, which is why Tree 1 in the decision trees opens with a stop. Fifty well-chosen items beat five hundred scraped ones.
Every item carries, at minimum — the sheet for this is memory-eval-sheet.md:
| Field | Why |
|---|
| The query, verbatim | Paraphrasing at judging time is how a system gets credit for the wrong answer |
| The expected answer, or the acceptance criteria | "Looks right" is not a criterion |
| The session or turn where the fact was established | Without this you cannot tell recall from a lucky guess |
| How many sessions ago that was | Memory decays with distance; a single-session set measures nothing durable |
| Provenance and date of the ground truth | And who decided it |
| The failure mode it is there to catch | Step 2's taxonomy |
Cover the long horizon deliberately. A set whose facts were all established in the previous turn is a context-window test wearing a memory system's clothes. Include items whose fact was established many sessions back, items where the fact changed since it was written, and items that should return nothing.
No user data, no real memory-store contents, no PII in the golden set (§4). Synthesize or de-identify.
Step 2 — Label every item with the failure mode it is there to catch
Six modes. An eval that only measures "right answer / wrong answer" cannot distinguish them, and they have completely different fixes.
| Failure mode | What it looks like | Where the fix lives |
|---|
| Stale fact | Was true when written, is not now | design-forgetting-policy — TTL, decay, re-verification |
| Unresolved contradiction | Two entries disagree; the answer picks one silently | Same skill — detect, never auto-merge, surface for a human |
| Confabulated recall | Confidently recalls something never stored | Extraction or grounding defect; check the write path's logic |
| Poisoned recall | Recalls something that was never true and could have come from untrusted input | memory-poisoning-review — this is a security incident, not an eval miss |
| Over-retention / leak | Surfaces something that should have been deleted, or crosses a tenant or user boundary | Retention and access scoping; a boundary crossing is a Blocker |
| Under-retention / amnesia | Cannot recall something it was supposed to keep | The bound is too tight, or the write never happened |
Poisoned recall and stale fact look identical in a transcript. The discriminator is "was this ever true?" — and if the answer is no and untrusted input could reach that write path, stop the eval and open a security review.
Step 3 — Decide who judges, before you see any results
- Two judges, blind to which arm produced the answer. Adjudicate disagreements by a written rule agreed in advance, not by whoever cares most.
- If a model is a judge, it is an instrument that needs its own calibration: run it against a human-labelled slice first and report its agreement rate. A new checker's first output is a claim about the checker.
- Freeze the rubric before the first run. A rubric edited after seeing results measures the editor.
Part B — The bake-off (run this)
The answer to "no trustworthy leaderboard exists" is a procedure, not a warning. This one is deliberately small enough to actually run.
Step 4 — Fix everything that is not the memory system
One variable moves. Write down and hold constant: the model and version, the corpus, the golden set, the retrieval budget (top-k, context allowance), the prompt template, and the judging rubric. Record every one of them in the results sheet — a bake-off whose configuration was not written down is not reproducible, which makes it an anecdote.
Step 5 — Define the arms. Three are mandatory
| Arm | What it is | Why it is mandatory |
|---|
| A0 — stateless | No injected history at all | Sizes the gap. Note it is not the same job — it cannot answer a memory-dependent query at any accuracy |
| A1 — flat lexical retrieval | BM25 or an embedding index over the same history, no LLM in construction | The humbling baseline: in the published suite this shape was both the most accurate and the cheapest per correct answer |
| A2 — the candidate | The memory system you are actually considering | — |
| A3…An | Any further candidates | Optional; the same rules apply to each |
A1 is not optional. Skipping it is how a team spends a quarter building something a keyword index would have beaten.
Step 6 — Run each arm and record two numbers plus a receipt
For each arm, over the whole golden set:
- Accuracy — correct items ÷ total items, by the frozen rubric.
- Total cost over the run — and this must include the construction / write-path cost, not only query cost. Excluding the build is the single most common way a bake-off flatters a memory system, because that is where the money lives (§3 #1).
- The receipt — per-item outcome and failure-mode label, so a later argument is settled by data rather than memory.
Also record, because they are decisions in disguise: build wall-clock, per-query latency (p50 and p95), and footprint at the end of the run.
Step 7 — Rank on cost per correct answer, never on either half alone
memory_engineering_calc.py cost-per-correct \
--total-cost-a 240.00 --queries-a 2000 --accuracy-a 0.62 --system-name-a "memory build" \
--total-cost-b 90.00 --queries-b 2000 --accuracy-b 0.20 --system-name-b "lexical baseline"
With those inputs the tool reports memory build : $0.193548 per correct answer against lexical baseline : $0.225, and then prints:
>> NOTE: the raw cost-per-query ranking DISAGREES with this one.
Which is exactly the case worth catching: on raw cost the lexical arm is less than half the price and still loses. Run the mode once per pair of arms. System-B flags are all-or-nothing, and an accuracy of 0 or above 1 exits 2 rather than producing a number.
Accuracy without cost and cost without accuracy are both unfalsifiable. That is why the mode belongs to neither skill.
Step 8 — Report the axes a benchmark will never cover
Rank on cost per correct answer, then report these beside it. Each maps to a failure mode from step 2 and none appears on any published leaderboard:
| Metric | How to compute it | What it catches |
|---|
| Staleness rate | Items whose stored fact was outdated at answer time ÷ items whose fact changed | Retention that never fires |
| Contradiction rate | Items where two stored entries disagreed ÷ total | Consolidation that merges silently |
| Confabulation rate | Confident answers with no stored support ÷ total | Grounding failure |
| Erasure verification rate | Deleted items still recoverable — from the row, the vector, a derived summary, or the version history — ÷ deletions attempted | An erasure story that stops at the row (§3 #7) |
| Recall-by-distance | Accuracy bucketed by how many sessions ago the fact was established | The long-horizon claim, tested |
| Cost per correct answer | Step 7 | Whether it should exist at all |
Erasure verification is measured by reading back, never by checking a return code. A redaction call can return success and leave the value in place.
Step 9 — Freeze, re-run, report the delta
- Freeze the golden set and the rubric. Version them beside the code.
- Every change to the memory system re-runs the whole set and reports a before/after delta. No eval, no ship.
- Every production failure becomes a permanent item, labelled with its failure mode. This is the only mechanism that makes the set get better over time.
- Re-run the baselines too, not just the candidate. A model upgrade can quietly make the stateless arm win, and nobody notices if it stopped being measured.
Guardrails
- Do not cite a vendor or paper leaderboard as evidence that a memory system works here. Every one of them is self- or competitor-reported, and the two most-cited benchmarks are contested on methodology (§3 #8).
- Report the configuration with every number. A result without its model, corpus, budget and query count is not transferable, and a number stripped of its conditions makes a claim it cannot support.
- No user data or memory-store contents in the set, the receipts, or the write-up (§4).
- Mark anything you did not measure this run as unmeasured, with the route that would settle it.
Output
A frozen golden set with provenance and failure-mode labels, a bake-off with at least the stateless and lexical arms, cost per correct answer per arm from the calculator, the six uncovered metrics reported beside it, and a named next action with owner, date and expected movement.
