eval-memory

name

eval-memory

description

Autonomously check the health of the personal-knowledge memory system AND validate whether the autolearn improvements are actually working (not just active). Runs a blind LLM-as-judge efficacy eval against a frozen probe set, scores re-rank vs raw retrieval, tracks the trend in a ledger, and judges how well the dashboard surfaces that progress. Invoke when the user wants a memory-system health + efficacy report, asks "is autolearn working / improving", or on a schedule (cron-ready).

eval-memory — memory system health & efficacy runner

You are the runner. This skill is a procedure you execute live, mostly quiet, pausing to converse only where the user's judgment matters (the verdict and the dashboard gap). Default runs are read-only and safe to run unattended.

The one idea that makes this non-trivial

Every autolearn signal is self-referential: co-access feeds re-rank, which shapes what co-surfaces, which feeds co-access. So "the numbers are growing" proves activity, not improvement — it can be a self-confirming loop. The only way to claim "it's working" is a ground-truth judge external to the signal loop. That judge is you, ruling blind on whether re-ranked retrieval beats raw retrieval. Protect that blindness or the whole eval is theater.

Config (read, don't hardcode elsewhere)

• Server base URL: http://127.0.0.1:7345 (from MEMORY_HTTP_HOST/MEMORY_HTTP_PORT in .env).
• Repo data dir: data/ (gitignored). Key files: search-history.json, auto-signals.json, .autolearn-state.json, autolearn.config.json.
• Frozen probe set: eval/probe-set.json (local, gitignored — copy from eval/probe-set.example.json and author your own; the controlled ruler — see Q4 below).
• Outputs: eval/ledger.jsonl (trend, append-only) and eval/reports/<YYYY-MM-DD>.md (narrative).

Procedure

Phase 0 — Setup (quiet)

1. Read eval/probe-set.json. If missing, stop and tell the user to seed it (see "Seeding" below).
2. GET /health/detail. Record autolearn_level, totals, last_flip_at, last_backup_*, and whether learning_stats is present.
3. Read the last few rows of eval/ledger.jsonl (if any) for trend comparison.

Phase 1 — Health (quiet batch, Q9/B)

Collect, don't narrate each item. Three layers:

• Plumbing: /health reachable; /health/detail.autolearn_level == expected (activation per current .env); totals sane vs ledger; learning_stats present iff level==activation.
• Freshness (high-value — catches a silently-dead 60s flush timer): is auto-signals.json and search-history.json mtime / newest-entry ts recent relative to known activity? A stale file at activation level is a red flag — learning has quietly stopped while everything looks fine.
• Corpus hygiene: run analyze_subject on the top live subjects (your most active projects and the memory system) and/or scan totals for: orphans, untagged, state past expected_until, decisions-without-outcomes, contradictions, and obvious noise (test/tombstone/probe records). Produce a compact health summary with PASS/WARN/FAIL per check.

Phase 2 — Efficacy (quiet batch, Q3/B + Q5/A + Q8/A)

Gate first: if autolearn_level is off or surfacing, re-rank is inactive — report efficacy as "N/A — re-rank inactive at level X" and skip to Phase 4. Do NOT fabricate a verdict.

Otherwise, for the frozen probes (and a small live-history drift sample — see below):

1. For each probe, call mcp__memory__search_memory with the probe query and source: "eval". This keeps eval replays out of the genuine-usage signals (co_access/co_surface edges, auto-cite tracker) — searches without it polluted co_access 38→2675 on the 2026-06-10 run. Note: searches never bump access_count anymore (only surface_count); real use = get/cite.
2. Read the just-appended search-history.json entry (match newest by q==query) to recover raw_topk_ids and reranked_topk_ids. If they're identical, record "no re-rank" for that probe.
3. mcp__memory__get_memory for the union of ids; assemble the two ordered candidate lists.
4. Blind pairwise judge (Q5/A): present the two orderings as "List 1 / List 2" with position randomized per probe (vary by probe index so it's reproducible, not fixed). Decide which list better answers the probe's intent. Record {probe_id, winner: raw|reranked|tie, reasoning: <one line>}. Unblind only after deciding. Never let the judge see which list is the re-ranked one.
5. Parallelize: fan the per-probe judging out across Agent subagents (each gets one probe's two blind lists + intent, returns the structured verdict). ~20 probes is real fan-out — batch them.

Drift sample (Q4/B): also pull ~5 recent queries from search-history.json (real usage), run the same blind judge. These are NOT added to the trend — they're a staleness alarm: if the frozen set says "great" but the live sample says "bad", the fixture has drifted from reality; flag it.

Aggregate the frozen probes → win/tie/loss counts and win_rate = wins / (wins+losses) (ties excluded from rate, reported separately).

Phase 3 — Verdict (CONVERSE HERE, Q14/C + Q11/C)

This is the only loud phase. Hand the user a digest:

• Health summary (PASS/WARN/FAIL).
• Efficacy: win/tie/loss, win_rate, the band it falls in, trend vs last ledger row, drift result.
• The per-probe verdicts with one-line reasoning (so a suspicious win is auditable, not a black box).

Bands (mechanical, logged):

• win_rate >= 0.60 and not trending down → WORKING
• 0.45 <= win_rate < 0.60 → MARGINAL
• win_rate < 0.45 → HURTING — re-rank is degrading results, investigate

Then narrate around the band: explicitly flag small-N unreliability (e.g. "win_rate dropped but only 8 probes re-ranked this run — likely noise, not regression"). Invite the user to override a judge call, question the band, or sharpen findings before you write anything. Apply their edits.

Phase 4 — Persist (after user is satisfied, Q6/C)

• Append one row to eval/ledger.jsonl (schema below).
• Write eval/reports/<YYYY-MM-DD>.md: narrative + per-probe verdicts + reasoning + health detail + dashboard findings. This is the human artifact.

Phase 5 — Dashboard coverage (prose, Q10/A + Q7/B)

Source-level gap analysis: read the dashboard .astro pages (dashboard/src/pages/index.astro, learn.astro) and dashboard/src/lib/*. Enumerate what's collected vs what's actually rendered. Headline finding to expect: the ledger trend (the efficacy progress line) exists and nothing plots it. Emit a concrete spec for a /eval dashboard page that reads eval/ledger.jsonl and plots win_rate over time — same disk-read pattern as existing pages, no server change. This is prose, not a tracked score — report it; building the page is a fast-follow (v1.1), justified by this finding.

Ledger row schema (eval/ledger.jsonl, one JSON object per line)

{
  "date": "2026-06-02",
  "probe_set_version": 1,
  "level": "activation",
  "probes_total": 20,
  "probes_reranked": 14,
  "win": 9, "tie": 3, "loss": 2,
  "win_rate": 0.818,
  "band": "WORKING",
  "drift_sample_n": 5,
  "drift_win_rate": 0.6,
  "drift_flag": false,
  "signal_totals": { "co_access": 0, "co_surface": 0, "auto_cite": 0 },
  "health": { "plumbing": "PASS", "freshness": "PASS", "hygiene": "WARN" },
  "notes": "short free text; e.g. small-N caveat or what changed since last run"
}

Stamp date from the system clock at write time (the user's session date), not from inside any randomized logic.

Seeding the probe set (first run, Q13/B)

The user authors intent; you audit coverage. On first run (or when asked): read the corpus (get_stats, list_recent per type, analyze_subject on major subjects), check the frozen probes cover the live subject regions, and propose additions for blind spots ("40 knowledge records on X, zero probes touch it") — but only commit additions the user approves. Never silently expand the set you grade yourself against.

Deep mode (DOCUMENTED ONLY — not implemented in v1, Q12/B)

A future --deep would run the same probes with MEMORY_AUTOLEARN toggled off vs capture/activation for a true causal A/B. This mutates server env/state, so it is out of the safe read-only default. If ever built: snapshot via the existing backup-on-flip, flip in a guarded block, and restore the original level in a finally so a crash can't strand the wrong level. Do not do this without the user present.

Cron note (Q1)

This skill is the unit of work. To schedule unattended runs later, point a cron at it — everything in the default path is read-only and writes only to eval/. Keep deep mode out of any unattended run.