memory-system

# Memory System Use the current `recall` and `session-search` contracts through the saved memory integration. Private queries belong in structured MCP arguments. All raw session searches, extracts, and local transcript summaries require a scope-bound `evidenceDrilldownReceipt`; stale or empty projections do not grant access. The command history below does not waive this boundary. The joelclaw memory system in one breath: accepted Runs remain immutable evidence → flowing memory derives scoped observations and reflections → composed recall returns flowing reflections, flowing observations, and curated Brain pages as separate lanes. The raw-first Observer tick is retired. Dreams still condense aging Brain pages, and the neat-memory curator still grades existing eligible Brain candidates. The wiki renders every registered Brain at `brain.joelclaw.com/<root>/<slug>/`. **The living map with per-subsystem health verdicts:** https://brain.joelclaw.com/joelclaw/resources/memory-system/ — read it before deep debugging; regenerate it (fresh survey, not sentence patches) after material changes. Effort trails: `.brain/projects/memory-system-review/` (survey assets = the receipts) and `.brain/projects/memory-system-repair/` (fixes + open decisions). > **Memory retirement (2026-07-17).** Versions ≤0.1.0 of this skill described > the retired stack — `memory_observations`, write gates, and reflect/promote > into `MEMORY.md`. That collection was archived to NAS and dropped; its > functions and MEM/FRIC suites were deleted. `joelclaw recall` kept its > command name but now composes flowing reflections, flowing observations, and > curated Brain pages. Do NOT follow old copies of this skill or write > `memory/observation.submitted` events. ## Where everything lives | Surface | Location | | --------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | | Observer dev repo | `~/Code/joelhooks/joelclaw-observer` (branchless, detached HEAD by design) | | Observer release (what actually runs) | `~/.joelclaw/observer-release` | | Promotion (the ONLY way to deploy observer) | `bin/promote-release.sh` — clean-tree gate, smoke checks, receipts to `/tmp/observer-release-promotion.log`; `--dry-run` first if unsure | | Observation pages | `~/Code/joelhooks/dark-wizard/.brain/observations/` (condensed originals under `archive/YYYY-MM/`) | | Retros (curator input; hook-fired Inngest event + condenser is being built) | `~/Code/joelhooks/dark-wizard/.brain/resources/retros/` | | Curator state (0600, atomic writes only) | `~/.joelclaw/observer-neat-memories.json` | | Observer cursor / session / family / pending | `~/.joelclaw/observer-*.json` | | Dispatch rules (what wakes the observer) | `<observer repo>/dispatch-rules.json` — dispatcher re-reads per classify; no restart needed | | Logs | `/tmp/observer-{tick,dream,neat-memory,self-maintenance,release-promotion}.log` | | LaunchAgents | Observer tick retired 2026-08-25; `com.joelclaw.observer-dream` (06:10) and `com.joelclaw.observer-neat-memory` (16:15 dead-man) remain active | | Message journal fail-open spool | `~/.joelclaw/spool/message-journal/` | | Current flowing wake spool | `~/.joelclaw/flowing-memory/native-wakes.jsonl` (single collector ownership) | | Legacy Central capture outboxes | flat and namespaced JSON queues; preserve as migration evidence, not current writes | | Brain root registry | `~/Code/joelhooks/dark-wizard/brain-roots.json` | | Wiki repo + build | `~/Code/joelhooks/joelclaw-wiki`; `bun run build` (includes `graph:build`; retry once on transient exit 1); 200-check exact URLs before sharing | | Bus (Inngest) | `localhost:8288`, signing key in `~/.config/system-bus.env` | ## Reviewed behavior cards `ReflectionV2` cards can rank before V1 reflections inside the existing `flowing-reflections` lane. They do not add a record kind or recall lane. Production admits only source-grounded, digest-pinned review artifacts through the sole semantic worker while daemon mode is stopped. Withdrawal and restoration append receipts and update the scope head; the card row remains immutable. Automatic formation, embeddings, and session-start briefing remain off. Production receipt: `joelclaw-memory/.brain/projects/joelclaw-memory/reviewed-memory-card-production-receipt-2026-08-28.svx`. ## Debug recipes (symptom → first moves) **"The curator went silent."** `joelclaw wake list --json` — is a `neat-memory-beat` pending? Chain alive → read `/tmp/observer-neat-memory.log` for the last verdict (holds are normal; "beat pending; dead-man exits" means the daily LaunchAgent correctly deferred). Chain dead → the 16:15 dead-man revives it, or run `NEAT_MEMORY_TRIGGER=beat bash ~/.joelclaw/observer-release/bin/neat-memory.sh`. Last output: `/tmp/observer-neat-memory-last.json` (can be stale — check log lines first). Cancel pending beats (`joelclaw wake cancel <id>`) before manual runs or you'll double-schedule. **"A DM never arrived."** Messaging rides the canonical Chat SDK path (since 2026-07-17). Machine-check visible delivery from the joelclaw repo: `bun scripts/messaging-visible-delivery-canary.ts` — exit 0 requires a confirmed journal row whose Telegram platform id matches. A `notify.compat_v2.confirmed` OTEL event alone is NOT proof. Journal rows: `~/.joelclaw/spool/message-journal/` (`origin_system_id` is a `source:eventId` composite). Deeper: `skills/messaging/SKILL.md`. **"A reaction didn't grade."** The pipe: `message/inbound.reaction` → `message/reaction-bridge` → `message/reaction.received` → `message/neat-memory-reaction-grade` → state-file `outcome`. Verify: `jq --arg slug "<slug>" '.sent[] | select(.slug == $slug) | {slug, outcome}' ~/.joelclaw/observer-neat-memories.json`. Mapping: 👍 ❤️ 🔥 💯 → `worked`; 👎 💩 → `did-not-work`; others ignored; only Joel's reactions count. Entries without `flowId` fall back to a two-minute timestamp match — safe only while sends are sparse. **"The dream did nothing."** Read `/tmp/observer-dream.log`. Since the 2026-07-17 fix, `pagesRead` is the whole corpus and `capped:true` means eligible work exceeded the nightly cap (fine, remainder waits) — not blindness. `eligiblePages: 0` usually means the corpus is younger than `minAgeDays` (7 for plain pages; rollups re-blur at 30/90 days). That is patience, not breakage. **"A Brain page isn't on the wiki."** In order: root registered in `brain-roots.json`? File is `.svx` (the law — `.md` is dead to the renderer)? Excluded (sections `captures`/`compaction-dumps`/`people`/ `private-family-health`, or `privacy: sensitive`)? Raw `{...}` in prose outside code spans breaks the page at prerender — backtick-escape braces. Then build on flagg and 200-check the exact URL. Relative `.svx` links and `[[root/slug]]` wikilinks both compile to page URLs. **"Is the observer even running?"** `launchctl list | grep joelclaw` — tick/dream/neat-memory should show status 0. The self-maintenance check (inside each tick) surfaces warnings in `/tmp/observer-tick.log`, including release-drift (dev HEAD ≠ release HEAD). **"Session capture / SQLite index is stale"** (ADR-0243): raw truth is `~/.joelclaw/runs-dev/<user>/<yyyy-mm>/*.jsonl`; the live FTS projection is `~/.joelclaw/search/sessions.db` via `memory/run.captured`. Verify raw and SQLite timestamps separately; check Inngest queue health; dry-run `bun scripts/backfill-session-index.ts`, then apply with `bun scripts/backfill-session-index.ts --apply`. Never flood Inngest with replay events. Typesense `runs_dev` and `run_chunks_dev` are retired and must not be recreated. ## Usage (getting things in and out) - **Write memory**: `.svx` pages into a registered `.brain` tree — frontmatter `title` + `privacy` (`public`/`private` render; `sensitive` never renders and never reaches the curator). MDSvX-escape literal braces. All `.brain` prose is `.svx`, always. - **Reach the curator**: candidates = observation + retro pages modified in the last 7 days with `privacy: public|private`, a title, and a non-empty body. Dedupe is forever — a sent slug never resends. - **Teach the curator**: react to its DMs. The taste bench (2026-07-17) proved one grade changes nothing measurable; ~20–30 graded sends with both polarities is the dataset that could. Every 👍/👎 counts. - **Query**: use composed `joelclaw recall` for broad retrieval. It keeps `flowing-reflections`, `flowing-observations`, and `curated-pages` separate; never compare scores across lanes. Interactive use must run in a trusted GitHub checkout or pass `--project` and `--workstream`; automatic callers must also name principal, purpose, and allowed privacy through private stdin. Use `joelclaw sessions search <query> --source both --extract` to drill into exact captured session evidence, and `joelclaw docs search` for books. Read the canonical Brain `.svx` or accepted flowing record before a durable claim. ## Retired and parked surfaces (don't build on these) - `memory_observations`, reflect/promote/proposal/batch-review functions, and the MEM/FRIC suites are retired and deleted from the live stack. - The system-log JSONL / `slog` path is retired. Telemetry is OTel in Typesense/ClickHouse; durable receipts are Brain `.svx`. - Claude auto-memory is a pointer index only: `MEMORY.md` may point into `~/.brain`, but must not grow topic pages or hold content. - Chorus/Rhizomatic is parked: no session briefing injection and no real claims. Service shutdown still needs the steering sudo step. If claim graphs return, they return as a Brain-owned projection. ## Doctor `joelclaw memory doctor` is chartered (`.brain/projects/memory-system-repair/build-memory-doctor.svx`): deterministic health checks with a JSON envelope — beat pending, state freshness, outbox size/age, release drift, dream log, journal spool, LaunchAgent status. Until it ships, the recipes above are the doctor.