skill-distill

# Distill Skill (Direct Execution) Direct execution skill for memory vault analysis and maintenance. Handles the `/distill` command's sub-modes: the seven existing hygiene sub-modes (`report`/`purge`/`merge`/`compress`/`refine`/`gc`/`auto`) plus five new or redefined telemetry-sourced sub-modes (`--revise`/`--meta`/`--review`/`--learn`/`--dream`). Memory *creation* is a separate concern owned by the sibling `skill-learn` skill. ## Context References Reference (do not load eagerly): - Path: `@.memory/memory-index.json` - Machine-queryable memory index - Path: `@.claude/context/project/memory/distill-usage.md` - Usage guide - Path: `@.claude/context/project/memory/telemetry-guardrails.md` - Binding design constraints for the telemetry-sourced sub-modes (four-tier source model, evaluator-outside-the-loop rule, six failure modes, `gen_ai.*` borrowing rule, cross-repo invocation discipline) - Path: `@.claude/skills/skill-learn/SKILL.md` - the sibling skill this file was split from. Its "Validate-on-Read" and "JSON Index Maintenance" sections are the shared procedures this file's body cites as "above" or "below" throughout -- both skills operate on the same `.memory/memory-index.json`, and those two procedures were not duplicated by the split; they live in `skill-learn/SKILL.md` only. --- ## Mode: distill Memory vault distillation: scoring, health reporting, and maintenance operations. Invoked by `/distill` command with `mode=distill`. ### Prerequisites **Validate-on-Read**: Before scoring, run the validate-on-read procedure from the "Validate-on-Read" section above to ensure `memory-index.json` is consistent with the filesystem. If stale, regenerate using the "JSON Index Maintenance" procedure before proceeding. ### Sub-Mode Dispatch | Sub-Mode | Description | Status | |----------|-------------|--------| | `report` | Generate health report with scoring | Available | | `purge` | Tombstone stale/zero-retrieval memories | Available | | `merge` | Combine memories with duplicate score > 0.6 | Available | | `compress` | Summarize memories with size penalty > 0.5 | Available | | `refine` | Improve memory quality (keywords, tags) | Available | | `gc` | Hard-delete tombstoned memories past grace period | Available | | `auto` | Automated distillation (Tier 1 refine only) | Available | | `revise` | Event-and-OTel-correlated memory refactoring proposals | Available | | `meta` | Cross-repo agent-system improvement proposals | Available | | `review` | Read-only ad hoc inquiry over the vault and all four source tiers | Available | | `learn` | Retroactive batch harvest across already-completed tasks | Available | | `dream` | Speculative direction-finding over `history.jsonl`'s recurring themes | Available | All 12 sub-modes are now available. No placeholder responses needed. ### Scoring Engine The scoring engine computes a composite maintenance score for each memory in the vault. Higher scores indicate memories that are better candidates for maintenance operations. #### Input Read all entries from `.memory/memory-index.json` after validate-on-read. Each entry provides: - `created` (ISO date) - `modified` (ISO date) - `last_retrieved` (ISO date or null) - `retrieval_count` (number) - `token_count` (number) - `keywords` (array of strings) #### Component 1: Staleness Score (weight: 0.3) Measures how long since the memory was last useful. ``` days_since_last = days_between(today, last_retrieved or created) staleness = min(1.0, days_since_last / 90) # FSRS adjustment: reduce staleness for actively retrieved old memories if retrieval_count > 0 AND days_since_created > 60: staleness = max(0, staleness - 0.3) ``` - Range: 0.0 (fresh) to 1.0 (90+ days stale) - FSRS adjustment rewards memories that have proven useful over time #### Component 2: Zero-Retrieval Penalty (weight: 0.25) Penalizes memories that have never been retrieved after a grace period. ``` if topic starts_with "email/preferences/": zero_retrieval = 0.0 # reserved-namespace exemption (email-to-memory-preferences.md design # §5.1) -- these memories are intentionally never read back by # /research /plan /implement auto-retrieval (see memory-retrieve.sh's # topic-prefix pre-filter), so a zero retrieval_count is expected, not # a staleness signal elif retrieval_count == 0 AND days_since_created > 30: zero_retrieval = 1.0 else: zero_retrieval = 0.0 ``` - Binary: 0.0 (has retrievals, too new, or `email/preferences/*` exempt) or 1.0 (never retrieved, older than 30 days, and not in the exempt namespace) #### Component 3: Size Penalty (weight: 0.2) Penalizes oversized memories that may benefit from compression. ``` size_penalty = max(0, (token_count - 600) / 600) ``` - Range: 0.0 (600 tokens or fewer) to unbounded (linear above 600) - A 1200-token memory scores 1.0; a 300-token memory scores 0.0 #### Component 4: Duplicate Score (weight: 0.25) Measures keyword overlap with the most similar other memory in the vault. ``` for each other_memory in vault: overlap = |memory.keywords intersect other_memory.keywords| / |memory.keywords| duplicate = max(overlap across all other memories) ``` - Range: 0.0 (no keyword overlap) to 1.0 (complete keyword subset) - Uses Jaccard-like ratio: intersection size divided by the memory's own keyword count #### Composite Score ``` composite = (staleness * 0.3) + (zero_retrieval * 0.25) + (size_penalty * 0.2) + (duplicate * 0.25) composite = clamp(composite, 0, 1) ``` - Weights sum to 1.0 (0.3 + 0.25 + 0.2 + 0.25) - Range: 0.0 (healthy memory) to 1.0 (strong maintenance candidate) #### Topic-Cluster Grouping Group memories by topic cluster for the health report. The cluster key is the first path segment of the memory's `topic` field: ``` cluster_key = topic.split("/")[0] # Example: # topic "python/libs/requests" -> cluster "python" # topic "lua/patterns" -> cluster "lua" # topic "" or null -> cluster "uncategorized" ``` ### Maintenance Candidate Classification Based on composite scores, classify each memory: | Composite Score | Classification | Recommended Action | |-----------------|----------------|-------------------| | >= 0.7 | Purge candidate | Remove (--purge) | | >= 0.5 | Merge/compress candidate | Merge duplicates (--merge) or compress (--compress) | | >= 0.3 | Review candidate | May benefit from refinement (--refine) | | < 0.3 | Healthy | No action needed | Additionally, flag specific conditions: - `duplicate > 0.6` -> Merge candidate regardless of composite - `size_penalty > 0.5` -> Compress candidate regardless of composite - `zero_retrieval == 1.0` -> Review for relevance ### Health Report Template The `report` sub-mode generates a formatted health report displayed to the user. Template: ``` ## Memory Vault Health Report **Generated**: {today} **Vault**: .memory/ --- ### Overview | Metric | Value | |--------|-------| | Total memories | {total_count} | | Total tokens | {total_tokens} | | Average tokens/memory | {avg_tokens} | | Oldest memory | {oldest_date} ({oldest_id}) | | Newest memory | {newest_date} ({newest_id}) | --- ### Category Distribution | Category | Count | Tokens | Avg Score | |----------|-------|--------|-----------| | {category_1} | {count} | {tokens} | {avg_composite} | | {category_2} | {count} | {tokens} | {avg_composite} | | ... | ... | ... | ... | --- ### Topic Clusters | Cluster | Memories | Avg Staleness | Avg Duplicate | |---------|----------|---------------|---------------| | {cluster_1} | {count} | {avg_staleness} | {avg_duplicate} | | {cluster_2} | {count} | {avg_staleness} | {avg_duplicate} | | ... | ... | ... | ... | --- ### Retrieval Statistics | Metric | Value | |--------|-------| | Never retrieved | {never_retrieved_count} ({never_retrieved_pct}%) | | Retrieved 1-3 times | {low_retrieval_count} | | Retrieved 4+ times | {high_retrieval_count} | | Most retrieved | {most_retrieved_id} ({most_retrieved_count} times) | --- ### Maintenance Candidates #### Purge Candidates (score >= 0.7) {purge_list or "None"} #### Merge Candidates (duplicate > 0.6) {merge_list or "None"} #### Compress Candidates (size > 0.5) {compress_list or "None"} #### Review Candidates (score 0.3-0.7) {review_list or "None"} --- ### Health Score **Score**: {health_score}/100 **Status**: {status_emoji} {status_label} Formula: `100 - (purge_count * 3) - (merge_count * 5) - (compress_count * 2)` | Threshold | Status | |-----------|--------| | 80-100 | Healthy | | 60-79 | Manageable | | 40-59 | Concerning | | 0-39 | Critical | --- ### Recommended Actions {action_list based on candidates found} ``` #### Health Score Formula ``` health_score = 100 - (purge_count * 3) - (merge_count * 5) - (compress_count * 2) health_score = clamp(health_score, 0, 100) ``` Where: - `purge_count` = number of memories with composite score >= 0.7 - `merge_count` = number of memories with duplicate score > 0.6 - `compress_count` = number of memories with size_penalty > 0.5 #### Health Status Thresholds | Score Range | Status | Description | |-------------|--------|-------------| | 80-100 | healthy | Vault is well-maintained | | 60-79 | manageable | Some maintenance recommended | | 40-59 | concerning | Significant maintenance needed | | 0-39 | critical | Urgent maintenance required | These thresholds mirror `repository_health.status` vocabulary in state.json. ## Shared Sub-Mode Skeleton Every mutating `/distill` sub-mode below (`purge`, `gc`, `merge`, `compress`, `refine`, `auto`, and the telemetry-sourced sub-modes added after them) follows the same seven-step shape. This section states that shape once, with named, generic placeholders; each sub-mode's own section below states only its deltas from this skeleton -- its specific candidate logic, prompts, execution steps, and log payload -- rather than restating the shape itself. This generalizes a convention this file already used once for the `dream` section's `### Overlap Scoring` cross-reference ("reference the section by name -- do not restate or fork the formula") into a file-wide rule. 1. **Edge Case Checks** -- Validate preconditions before identifying candidates (e.g. run validate-on-read, confirm a minimum count of eligible memories). If a precondition fails, display a specific message and return early without further action. 2. **Candidate Identification** -- Compute the sub-mode's specific candidate set from scored or otherwise-derived memory data. If a shared dependency like validate-on-read or the Scoring Engine is used, cite it by name rather than re-deriving it. 3. **Dry-Run** -- When `--dry-run` is active, display what the sub-mode would do (the specific candidate list, with sub-mode-relevant fields) and return early. No file is modified. 4. **Interactive Selection (MANDATORY STOP)** -- Present candidates via `AskUserQuestion` (`multiSelect: true` for any sub-mode selecting among multiple candidates). **This step is non-negotiable in every sub-mode that mutates the vault: no mutation may proceed without an explicit, user-confirmed selection at this step.** If no candidates exist, or the user selects none, display a specific message and exit without changes. 5. **Execution** -- Apply the confirmed operation. This step's actual content is the most sub-mode-specific of the seven and is stated in full in each sub-mode's own section -- it is