dream

# Dream - Memory Consolidation Deterministic memory maintenance: detect stale entries, merge duplicates, resolve contradictions, rebuild the MEMORY.md index. All pruning decisions are based on verifiable checks (file exists? function exists? duplicate content?), not LLM judgment. ## Argument Resolution ```python DRY_RUN = "--dry-run" in "$ARGUMENTS" # Preview changes without writing ``` ## Overview Memory files accumulate across sessions. Over time they develop problems: - **Stale references** — memories pointing to files, functions, or classes that no longer exist - **Duplicates** — multiple memories covering the same topic with overlapping content - **Contradictions** — newer memories superseding older ones without cleanup - **Index drift** — MEMORY.md index out of sync with actual memory files This skill fixes all four problems using deterministic checks only. **Finish line.** Done means: MEMORY.md is rebuilt from the surviving files and passes the STEP 5.5 verify, the STEP 6 report is printed, and STEPs 7 to 9 have each run or printed a one-line skip reason. Budget: one read pass over the discovered memory files (STEP 1), one MEMORY.md rewrite (STEP 5) plus the STEP 5.5 trailer rewrite when the index is over budget, and no subagents (every check is deterministic); stop and report at the finish line or the first cap, whichever comes first. > **Cadence (CC 2.1.142+):** Reactive compaction now sizes its first summarize attempt to the actual overflow, so long sessions stall mid-turn far less often. The "run nightly" cadence can relax toward "run when memory files accumulate" — consolidation is no longer needed to head off compaction inefficiency. --- ## STEP 1: Discover Memory Files ```python # Find the memory directory (agent-specific or project-level) # Agent memory lives in: .claude/agent-memory/<agent-id>/ # Project memory lives in: .claude/projects/<hash>/memory/ # Also check: .claude/memory/ memory_dirs = [] Glob(pattern=".claude/agent-memory/*/MEMORY.md") Glob(pattern=".claude/projects/*/memory/MEMORY.md") Glob(pattern=".claude/memory/MEMORY.md") # For each discovered MEMORY.md, glob all *.md files in that directory for dir in memory_dirs: Glob(pattern=f"{dir}/../*.md") # All memory files alongside MEMORY.md ``` Read every discovered memory file. Parse frontmatter (`name`, `description`, `type`) and body content. Build an in-memory inventory: ``` inventory = [{ "path": "/abs/path/to/file.md", "name": frontmatter.name, "type": frontmatter.type, # user, feedback, project, reference "description": frontmatter.description, "body": body_text, "file_refs": [], # extracted file paths "symbol_refs": [], # extracted function/class names "topics": [], # key phrases for duplicate detection }] ``` --- ## STEP 2: Detect Staleness For each memory file, extract references and verify they still exist. ### 2a: File Path References Extract paths that look like file references (patterns: paths with `/` and file extensions, backtick-wrapped paths): ```python # Regex-like extraction from body text: # - Paths containing / with common extensions: .py, .ts, .tsx, .js, .json, .md, .yaml, .yml, .sh # - Backtick-wrapped paths: `src/something/file.ts` # - Quoted paths in frontmatter descriptions ``` **Classify each ref's SCOPE before verifying it.** `Glob` only sees the current repo, so a path that lives anywhere else can never match and would otherwise be scored as missing. A memory about `~/.claude` hooks, a homebrew cask, a cmux config, or another repo is not stale just because this repo does not contain it. ```python def scope(ref): # Anything rooted outside the working repo is UNVERIFIABLE, not missing. if ref.startswith(("~", "/", "$")): return "UNVERIFIABLE" if ref.startswith(("http://", "https://")): return "UNVERIFIABLE" if re.match(r'^[A-Za-z0-9_.-]+/', ref) and not (REPO / ref.split("/")[0]).exists(): return "UNVERIFIABLE" # first segment is not a real top-level dir here return "REPO_RELATIVE" verifiable = [r for r in file_refs if scope(r) == "REPO_RELATIVE"] external = [r for r in file_refs if scope(r) == "UNVERIFIABLE"] missing = [] for ref in verifiable: Glob(pattern=ref) # If no match → missing.append(ref) ``` **The staleness ratio is computed over `verifiable` ONLY.** `external` refs are recorded for the report and never counted toward pruning. A memory with zero verifiable refs is `EVERGREEN` no matter how many external paths it names. ### 2b: Symbol References Extract function/class names (patterns: `function_name()`, `ClassName`, `def function_name`): ```python for symbol in symbol_refs: Grep(pattern=symbol, path=".", output_mode="files_with_matches", head_limit=1) # If no match → mark as STALE_SYMBOL_REF ``` ### 2c: Staleness Classification | Finding | Classification | Action | |---------|---------------|--------| | **Zero VERIFIABLE refs** (none, or all UNVERIFIABLE) | EVERGREEN | Keep | | All verifiable refs valid, all symbols found | FRESH | Keep | | Some verifiable refs missing | PARTIALLY_STALE | Flag for review | | All verifiable refs missing AND all symbols missing | FULLY_STALE | Prune candidate | Only memories classified as FULLY_STALE are auto-pruned. PARTIALLY_STALE memories are reported but kept — the user decides. ### 2d: Prune guards — checked AFTER classification, before any delete `FULLY_STALE` is necessary but **not sufficient** to delete. Every guard below downgrades to PARTIALLY_STALE (kept + flagged). These exist because memory files are **not in git**: a wrong delete is silent and unrecoverable, so the asymmetry always favours keeping. ```python GUARD_DAYS = 14 for m in list(fully_stale_files): reason = None # 1. Preferences do not decay because a path moved. if m["type"] == "user": reason = "type:user is never auto-pruned" # 2. A feedback/reference memory carries a LESSON; the file paths in it are # illustrations, not a manifest. Its worth does not expire when an # illustrative path moves, and ref-extraction is lossy anyway (it catches # `file.ts` but misses `file.ts:186` and `functionName()`). Only project # memories — which track live work against concrete files — are eligible # to go fully stale on ref death. elif m["type"] in ("feedback", "reference"): reason = f"type:{m['type']} value is the lesson, not its file refs" # 3. Recently written memories describe the present, whatever their refs say. elif (now - m["mtime"]) < GUARD_DAYS * 86400: reason = f"modified within {GUARD_DAYS}d" # 4. A memory that exists to prevent a regression must outlive the code it cites. elif re.search(r'\b(do not|don\'t|never|avoid)\b', m["body"], re.I): reason = "carries a do-not/never directive" if reason: m["classification"] = "PARTIALLY_STALE" m["kept_reason"] = f"prune-guard: {reason}" fully_stale_files.remove(m) partially_stale_files.append(m) ``` Guard 3 is the subtle one. `reference_cmux_scroll_blank_research` said *"RESOLVED; do NOT re-suggest `tui:fullscreen` on cmux"* and every path it cited had moved. Deleting it reintroduces exactly the regression it was written to prevent. A memory whose value is a prohibition is at its most useful precisely when the original code is gone. ### STEP 2.5: Consult-gate (#2351) — never prune a memory that's still being used Closing the VERIFY loop: a deletion must survive the question *"was this actually consulted?"*. A memory whose external refs all vanished (FULLY_STALE) but that the agent keeps looking up is still load-bearing — its **refs** are stale, its **knowledge** is live. So before pruning, read `.claude/logs/memory-consult.jsonl` (written by `memory-validator` on every `mcp__memory__search_nodes`/`open_nodes`/`read_graph`) and **downgrade any recently-consulted FULLY_STALE memory to PARTIALLY_STALE** (kept + flagged, not auto-deleted). ```python import json, time from pathlib import Path def recently_consulted_terms(days=14): log = Path(".claude/logs/memory-consult.jsonl") if not log.exists(): return set() cutoff = time.time() - days * 86400 terms = set() for line in log.read_text().splitlines(): try: e = json.loads(line) except ValueError: continue # best-effort: skip malformed lines # open_nodes carries exact entity names; search carries a query string terms.update(n.lower() for n in e.get("names", [])) if e.get("query"): terms.update(w.lower() for w in e["query"].split() if len(w) > 2) return terms consulted = recently_consulted_terms() for m in list(fully_stale_files): slug = Path(m["path"]).stem.lower() name = (m.get("name") or "").lower() if name in consulted or any(c in slug or slug in c for c in consulted): m["classification"] = "PARTIALLY_STALE" m["kept_reason"] = "consult-gate: looked up in the last 14 days (#2351)" fully_stale_files.remove(m) partially_stale_files.append(m) ``` This is conservative by design — fuzzy term matching errs toward **keeping** a maybe-consulted memory rather than deleting a live one. The Step 6 report records each consult-gated keep (the "did it matter?" audit the loop was missing). If the log is absent (consult instrumentation not yet exercised), the gate is a no-op and pruning proceeds as before. --- ## STEP 3: Detect Duplicates Compare memories pairwise within the same directory. Two memories are duplicates when: 1. **Same type** (both `feedback`, both `project`, etc.) 2. **Overlapping topic** — 60%+ of significant words (excluding stopwords) appear in both bodies 3. **Same subject** — `name` or `description` fields reference the same concept ```python stopwords = {"the", "a", "an", "is", "are", "was", "were", "be", "been", "have", "has", "had", "do", "does", "did", "will", "would", "could", "should", "may", "might", "can", "shall", "to", "of", "in", "for", "on", "with", "at", "by", "from", "as", "into", "through", "during", "before", "after", "this", "that", "it", "not", "no", "but", "or", "and", "if", "then", "than", "so"} def significant_words(text): words = set(text.lower().split()) - stopwords return {w for w in words if len(w) > 2} def overlap_ratio(words_a, words_b): if not words_a or not words_b: return 0.0 intersection = words_a & words_b smaller = min(len(words_a), len(words_b)) return len(intersection) / smaller if smaller > 0 else 0.0 # For each pair with same type: # if overlap_ratio >= 0.6 → DUPLICATE pair # Keep the NEWER file (by filesystem mtime), prune the older ``` --- ## STEP 4: Resolve Contradictions Contradictions occur when two memories of the same type make opposing claims about the same subject. Detection: 1. **Same type + same topic** (overlap >= 0.4 but < 0.6 — related but not duplicate) 2. **Negation signals** — one body contains negation of the other's assertion: - "do X" vs "do not X" / "don't X" / "never X" - "use X" vs "avoid X" / "stop using X" - "prefer X" vs "prefer Y" (for same decision domain) ```python negation_pairs = [ ("do ", "do not "), ("do ", "don't "), ("use ", "avoid "), ("use ", "stop using "), ("prefer ", "don't prefer "), ("always ", "never "), ] # For each pair flagged as contradictory: # Keep the NEWER file (more recent decision supersedes) # Prune the older file ``` --- ## STEP 5: Execute Changes (or Dry Run) ### Dry Run Mode (`--dry-run`) If `--dry-run` flag is present, skip all writes. Output the full report (Step 6) with `[DRY RUN]` prefix and list what WOULD be changed: ``` [DRY RUN] Would delete: .claude/agent-memory/foo/stale_old_path.md (FULLY_STALE) [DRY RUN] Would delete: .claude/agent-memory/foo/duplicate_auth.md (DUPLICATE of auth_patterns.md) [DRY RUN] Would delete: .claude/agent-memory/foo/old_preference.md (CONTRADICTED by new_preference.md) [DRY RUN] Would rebuild: .claude/agent-memory/foo/MEMORY.md (3 entries removed, 12 remaining) ``` ### Live Mode ```python # Move, never rm: memory files are not in git (see references/safe-deletes.md) trash = f"{memory_dir}/.trash/{date.today().isoformat()}" Bash(command=f"mkdir -p '{trash}'") # 1. FULLY_STALE files, 2. DUPLICATE (older), 3. CONTRADICTED (older) to_remove = [s["path"] for s in fully_stale_files] to_remove += [d["older"]["path"] for d in duplicate_pairs] to_remove += [c["older"]["path"] for c in contradiction_pairs] for path in to_remove: Bash(command=f"mv '{path}' '{trash}/'") # 4. Rebuild MEMORY.md index from surviving files (exclude .trash/ from the walk) ``` ### Rebuild MEMORY.md Read all surviving `.md` files (excluding MEMORY.md itself). Generate the index: ```markdown # <Directory Name> Memory