aim-lore-hygiene

# aim-lore-hygiene — Per-Operator Lore File Hygiene Compacts always-injected sanctum files (LORE.md, MEMORY.md) under the BP-159 ~200-line cap so rules don't "get lost in the noise" and adherence stays high. **FILE-content compaction only** — NOT Qdrant-point purging by age (that is `aim-purge`, a different domain — PLAN-028 P0-3). The deterministic mechanics (line-counting, marker classification, structural compaction, archiving) live in `scripts/lore_hygiene.py`, invoked **by path**. This file decides *when* to run, *which* files, and *how to read the plan*. ## When to use - A sanctum file crosses ~80% of its cap (160/200 lines — the compaction trigger). - A scheduled hygiene pass (session-stop hook or cron). - The operator asks to prune or compact lore, or flags a stale/wrong entry. ## Steps 1. Resolve the sanctum path: `{project-root}/_ai-memory/sanctum/{agent_id}/`. The scan governs every present class — `LORE.md`/`MEMORY.md` (compact), `PERSONA.md` (Evolution-Log section-cap), `CREED.md`/`BOND.md` (cap-check only). 2. **Dry-run audit (DEFAULT — never mutates):** `python3 scripts/lore_hygiene.py <sanctum-path>` Reports per file: line count, % of cap, and the proposed prune / archive / dedup actions with a projected post-compaction line count. 3. **Review the plan** — the verifiable intermediate output. Confirm prunes are genuinely superseded/contradicted, archives are stale-but-meaningful, and the projected count lands at or under cap. If the plan reports a residual over-cap, a manual/LLM semantic-summarization pass is needed (the script will not auto-truncate recall-value content). 4. **Apply only after review:** `python3 scripts/lore_hygiene.py <sanctum-path> --apply` Each mutated file gets a timestamped `.bak` sidecar first; archived entries move to `references/lore-archive/<FILE>.archive.md` with a one-line pointer left in the hot file. Add `--qdrant --group-id <project>` to also push archived entries to the Qdrant cold tier (best-effort, opt-in). 5. **Verify:** re-run the dry-run audit. Every file should be ≤ cap (or carry an explicit residual flag), and each archived entry should have a one-line pointer in the hot file. A second `--apply` on an already-clean file is a no-op. ## What the script acts on (structure-aware, keep-when-uncertain) The script acts **only on genuine content entries** — bullets, paragraphs, and table content rows. It is structure-aware: every structural or ambiguous construct is opaque passthrough, copied byte-for-byte and **never classified, deduped, split, or rewritten**: - **Code fences** — the entire block (```` ``` ````/`~~~`, including the info string and everything between the delimiters) is one opaque unit. A marker token *inside* a fence is example text, never an entry; an unterminated fence keeps its remainder opaque. - **Tables** — the header row and its `|---|` separator are structural (a tagged header is never classified, so a separator is never orphaned). Only the *content* rows below them are classified. - **Thematic breaks** (`---`, `***`, `___`) and other delimiters — structural, never deduped. - **Anything else not confidently structural-or-content** (blockquotes, indented code, raw HTML, odd constructs) — **kept opaque**. A raw-HTML block is gathered from its start line to the blank-line terminator (inner lines need not start with `<`), so a multi-line `<div>…</div>` stays whole. This is the keep-when-uncertain posture: because the skill mutates the operator's own memory, when it cannot confidently classify a block it keeps it intact rather than risk corrupting it. ## How entries are classified Within genuine content entries, the script is marker-driven (it never guesses an entry is low-utility): - **Prune (delete):** `[superseded]`, `[contradicted]`, `[wrong]`, `[obsolete]`, `[prune]`, a `[expired:YYYY-MM-DD]` TTL that has passed, or a full-entry strikethrough (a single `~~...~~` span covering all of the entry's content). - **Archive (cold tier + pointer):** `[stale]`, `[archive]`. For a non-table entry a one-line pointer is left in the hot file; for a **table content row** the row is dropped in place (the table stays well-formed) and its content still moves to the cold tier — no inline pointer is injected into the table body. - **Dedup:** exact-duplicate entries **within the same `## section`** are collapsed, keeping the first; identical text under different sections is kept (distinct records). Table rows (header, separator, content) are never deduped, so a second same-schema table keeps its header and separator. - **Keep:** everything else. **Markers must be anchored in the LEADING position.** A marker counts only as a **leading tag** — right after the bullet/number prefix (`- [superseded] …`) or in the first cell of a table row (`| [superseded] row | … |`). A marker merely *mentioned in prose* ("Tag an entry `[superseded]` when…") **or one that merely ends a line of prose** ("a fact retired at end of life `[obsolete]`") is **ignored and kept**, never pruned. (Trailing-tag anchoring was removed in cycle-2 — it pruned prose ending in a marker token; leading-only is the single, unambiguous, data-safe convention.) Tag entries during a Pulse, or when the operator says "that's not right anymore", so the next hygiene pass acts on them safely. > **Crash-rerun note (TGT-2).** Cold-tier appends are deduplicated by exact block > string. A same-day crash-rerun (hot file not yet rewritten) is fully idempotent. A > *cross-day* crash-rerun writes a second, differently-dated archive block for the > same content — accepted behavior that preserves a per-day audit trail and never > loses content. ## Governed classes & per-class strategy When run on a sanctum dir the script governs each present class by its declared `Contract` (caps from TASK-077 A2; the `Contract` model is the **shared** `pov/lib/governance/contract.py`, the same one `aim-tracking-rotate` uses): - **LORE.md / MEMORY.md — compact.** Marker-driven prune/archive/dedup (above). - **PERSONA.md — section-cap.** The `## Evolution Log` section keeps its last 10 entries; older rows relocate losslessly to the cold archive with one pointer left below the table. Symbolic section anchor only — never line numbers; the rest of PERSONA is not rotated. - **CREED.md / BOND.md — check-only.** Cap is reported but identity directives are **never auto-relocated**. An over-cap CREED/BOND makes the run exit non-zero so closeout HALTs — relocate by hand. ## Session-close gate (`--check`) `python3 scripts/lore_hygiene.py --check <sanctum-path>` is the read-only session-close gate (companion to `aim-tracking-rotate --check` for oversight files). It never mutates: it exits non-zero with a SYSTEM FAILURE block per **blocking** over-cap sanctum file (CREED/BOND line+KB, PERSONA `## Evolution Log` over the last-10 entry cap) and a remedy command, so closeout is blocked until each blocking class is within cap. The LORE/MEMORY compact-class file-SIZE cap is **reporting-only** (A2: full files stay on disk at full size; file rotation is tag-driven, not a closeout blocker) — an over-size LORE/MEMORY prints a WARNING but never blocks closeout. Wired into session-close step-04. ## Conservation proof (prove-then-commit) Every relocation is proven 0-lost **before** anything is written: the script builds the content multiset over the hot file and the cold archive and proves, via the shared `conservation` module, that every line not intentionally prune/dedup-removed survives in the would-be hot file **or** the would-be archive (and that the archive never shrinks). If the proof fails the run aborts with a non-zero exit and **leaves every original byte-identical** — no `.bak`, no partial write. The `.bak` sidecar is written only once the proof passes. ## `--cap` on a directory (global override) Scanning a sanctum dir uses each hot file's own cap (LORE.md/MEMORY.md → 200). Passing `--cap N` **overrides every file in that dir with the single value N** — a global override, not a per-file map (use it to audit one explicit file, e.g. `--cap 300` for a project-root `CLAUDE.md`). `--cap` must be a positive integer. Per-file caps when scanning a dir remain a possible future option (not built). ## References - Per-file caps, the ~80% trigger, and the LORE section schema: `references/caps.md` - The prune-vs-archive decision rule (BP-159 §6): `references/decision-rule.md`