reduce

## Runtime Configuration (Step 0 — before any processing) Read these files to configure domain-specific behavior: 1. **`ops/derivation-manifest.md`** — vocabulary mapping, extraction categories, platform hints - Use `vocabulary.notes` for the notes folder name - Use `vocabulary.inbox` for the inbox folder name - Use `vocabulary.note` for the note type name in output - Use `vocabulary.note_plural` for the plural form - Use `vocabulary.reduce` for the process verb in output - Use `vocabulary.cmd_reflect` for the next-phase command name - Use `vocabulary.cmd_reweave` for the backward-pass command name - Use `vocabulary.cmd_verify` for the verification command name - Use `vocabulary.extraction_categories` for domain-specific extraction table - Use `vocabulary.topic_map` for MOC/topic map references - Use `vocabulary.topic_maps` for plural form 2. **`ops/config.yaml`** — processing depth, pipeline chaining, selectivity - `processing.depth`: deep | standard | quick - `processing.chaining`: manual | suggested | automatic - `processing.extraction.selectivity`: strict | moderate | permissive 3. **`ops/queue/queue.json`** — current task queue (for handoff mode) If these files don't exist (pre-init invocation or standalone use), use universal defaults: - depth: standard - chaining: suggested - selectivity: moderate - notes folder: `notes/` - inbox folder: `inbox/` --- ## THE MISSION (READ THIS OR YOU WILL FAIL) You are the extraction engine. Raw source material enters. Structured, atomic {vocabulary.note_plural} exit. Everything between is your judgment — and that judgment must err toward extraction, not rejection. ### The Core Distinction | Concept | What It Means | Example | |---------|---------------|---------| | **Having knowledge** | The vault contains information | "We store notes in folders" | | **Articulated reasoning** | The vault explains WHY something works as a traversable {vocabulary.note} | "folder structure mirrors cognitive chunking because..." | **Having knowledge is not the same as articulating it.** Even if information is embedded in the system, the vault may lack the externalized reasoning explaining WHY it works. That reasoning is what you extract. ### The Comprehensive Extraction Principle **For domain-relevant sources, COMPREHENSIVE EXTRACTION is the default.** This means: 1. **Extract ALL core {vocabulary.note_plural}** — direct assertions about the domain that can stand alone as atomic propositions. 2. **Extract ALL evidence and validations** — if source confirms an approach, that confirmation IS the {vocabulary.note}. Evidence is extractable even when the conclusion is already known, because the reasoning path matters. 3. **Extract ALL patterns and methods** — techniques, workflows, practices. Named patterns are referenceable. Unnamed intuitions are not. 4. **Extract ALL tensions** — contradictions, trade-offs, conflicts. These are wisdom, not problems. 5. **Extract ALL enrichments** — if source adds detail to existing {vocabulary.note_plural}, create enrichment tasks. Near-duplicates almost always add value. **"We already know this" means we NEED the articulation, not that we should skip it.** ### The Extraction Question (ask for EVERY candidate) **"Would a future session benefit from this reasoning being a retrievable {vocabulary.note}?"** If YES -> extract to appropriate category If NO -> verify it is truly off-topic before skipping ### INVALID Skip Reasons (these are BUGS) - "validates existing approach" — validations ARE the evidence. Extract them. - "already captured in system config" — config is implementation, not articulation. The WHY needs a {vocabulary.note}. - "we already do this" — DOING is not EXPLAINING. The explanation needs externalization. - "obvious" — obvious to whom? Future sessions need explicit reasoning. - "near-duplicate" — near-duplicates almost always add detail. Create enrichment task. - "not a claim" — is it an implementation idea? tension? validation? Those ARE extractable. ### VALID Skip Reasons (rare) - Completely off-topic (unrelated to {vocabulary.domain}) - Too vague to act on (applies to everything, disagrees with nothing) - Pure summary with zero extractable insight - LITERALLY identical text already exists (not "same topic" — IDENTICAL) **For domain-relevant sources: skip rate < 10%. Zero extraction = BUG.** --- ## EXECUTE NOW **Target: $ARGUMENTS** Parse immediately: - If target contains a file path: extract insights from that file - If target contains `--handoff`: output RALPH HANDOFF block + task entries at end - If target is empty: scan {vocabulary.inbox}/ for unprocessed items, pick one - If target is "inbox" or "all": process all inbox items sequentially **Execute these steps:** 1. Read the source file fully — understand what it contains 2. **Source size check:** If source exceeds 2500 lines, STOP. Plan chunks of 350-1200 lines. Process each chunk with fresh context. See "Large Source Handling" section below. 3. Hunt for insights that serve the domain (see extraction categories below) 4. For each candidate: - Tier 1 (preferred): use `mcp__qmd__vector_search` with query "[claim as sentence]", collection="{vocabulary.notes_collection}", limit=5 - Tier 2 (CLI fallback): `qmd vsearch "[claim as sentence]" --collection {vocabulary.notes_collection} -n 5` - Tier 3 fallback if qmd is unavailable: use keyword grep duplicate checks - If duplicate exists: evaluate for enrichment or skip - Classify as OPEN (needs more investigation) or CLOSED (standalone, ready) 5. Output extraction report with titles, classifications, extraction rationale 6. Wait for user approval before creating files 7. If `--handoff` in target: create per-claim task files, update queue, output RALPH HANDOFF block **START NOW.** Reference below explains methodology — use to guide, not as output. ### Observation Capture (during work, not at end) When you encounter friction, surprises, methodology insights, process gaps, or contradictions — capture IMMEDIATELY: | Observation | Action | |-------------|--------| | Any observation | Create atomic note in `ops/observations/` with prose-sentence title | | Tension: content contradicts existing {vocabulary.note} | Create atomic note in `ops/tensions/` with prose-sentence title | The handoff Learnings section summarizes what you ALREADY logged during processing. --- # Reduce Extract composable {vocabulary.note_plural} from source material into {vocabulary.notes}/. ## Philosophy **Extract the REASONING behind what works, not just observations about what works.** This is the extraction phase of the pipeline. You receive raw content and extract insights that serve the vault's domain. The mission is building **externalized, retrievable reasoning** — a graph of atomic propositions that can be traversed, connected, and built upon. **THE CORE DISTINCTION:** | Concept | Example | What to Extract | |---------|---------|-----------------| | **We DO this** | "We tag notes with topics" | — (not sufficient) | | **We explain WHY** | "topic tagging enables cross-domain navigation because..." | This | The vault is not just an implementation. It is **the articulated argument for WHY the implementation works.** **THE EXTRACTION QUESTION:** - BASIC thinking: "Is this a standalone composable claim?" - BETTER thinking: "Does this serve {vocabulary.domain}?" - BEST thinking: **"Would a future session benefit from this reasoning being a retrievable {vocabulary.note}?"** If YES -> extract to appropriate category (even if "we already know this") If NO -> skip (RARE for domain-relevant sources — verify it is truly off-topic) **THE RULE:** Implementation without articulation is incomplete. If we DO something but lack a {vocabulary.note} explaining WHY it works, that articulation needs extraction. --- ## Extraction Categories ### What To Extract {DOMAIN:extraction_categories} **The structural invariant:** Every domain's extraction has these universal categories regardless of domain: | Category | What to Find | Output Type | Gate Required? | |----------|--------------|-------------|----------------| | Core domain {vocabulary.note_plural} | Direct assertions about {vocabulary.domain} | {vocabulary.note} | NO | | Patterns | Recurring structures across sources | {vocabulary.note} | NO | | Comparisons | How different approaches compare, X vs Y, trade-offs | {vocabulary.note} | NO | | Tensions | Contradictions, conflicts, unresolved trade-offs | tension note | NO | | Anti-patterns | What breaks, what to avoid, failure modes | problem note | NO | | Enrichments | Content that adds detail to existing {vocabulary.note_plural} | enrichment task | NO | | Open questions | Unresolved questions worth tracking | {vocabulary.note} (open) | NO | | Implementation ideas | Techniques, workflows, features to build | methodology note | NO | | Validations | Evidence confirming an approach works | {vocabulary.note} | NO | | Off-topic general content | Insight unrelated to {vocabulary.domain} | apply selectivity gate | YES | **IMPORTANT:** Categories 1-9 bypass the selectivity gate. They extract directly to the appropriate output type. The selectivity gate exists ONLY for filtering off-topic content from general sources. ### Category Detection Signals Hunt for these signals in every source: **Core domain signals:** - Direct assertions: "the key insight is...", "this means that...", "the pattern is..." - Evidence: "research shows...", "data indicates...", "studies confirm..." - Named methods: any named system, technique, or framework relevant to {vocabulary.domain} **Comparison signals:** - "X vs Y", "trade-off between...", "prefer X when...", "unlike Y, this..." - "choose X when...", "depends on whether..." **Tension signals:** - "contrary to...", "however...", "the problem with...", "fails when..." - "on the other hand...", "but this conflicts with..." **Anti-pattern signals:** - "systems fail when...", "the anti-pattern is...", "avoid this because..." - Warnings, cautionary examples, failure postmortems **Enrichment signals:** - Content covering ground similar to an existing {vocabulary.note} - New examples, evidence, or framing for an established claim - Deeper explanation of something already captured shallowly **Implementation signals:** - "we could build...", "would enable...", "a tool that...", "pattern for..." - Actionable techniques, concrete workflows **Validation signals:** - "this supports...", "evidence shows...", "validates...", "confirms..." - Research that grounds existing practice in theory ### The Mission Lens (REQUIRED) For EVERY candidate, ask: **"Does this serve {vocabulary.domain}?"** - YES -> **extract to appropriate category** (gate does NOT apply) - NO -> apply selectivity gate (for off-topic filtering only) **For domain-relevant sources:** almost everything is YES. The gate barely applies. Skip rate < 10%. --- ## The Selectivity Gate (for OFF-TOPIC content filtering) **CRITICAL:** This gate exists to filter OUT content that does not serve {vocabulary.domain}. It applies ONLY to standard claims from GENERAL (off-topic) sources. **Do NOT use gate to reject:** - Implementation ideas ("not a claim" is WRONG — it is roadmap) - Tensions ("not a claim" is WRONG — it is wisdom) - Enrichments ("duplicate" is WRONG — it adds detail) - Validations ("already known" is WRONG — it is evidence) - Open questions ("not testable" is WRONG — it is direction) For STANDARD claims from general sources, verify all four criteria pass: ### 1. Standalone The claim is understandable without source context. Someone reading this {vocabulary.note} cold can grasp what it argues without needing to know where it came from. Fail: "the author's third point about methodology" Pass: "explicit structure beats implicit convention" ### 2. Composable This {vocabulary.note} would be linked FROM elsewhere. {vocabulary.note_plural} function as APIs. If you cannot imagine writing `since [[this claim]]...` in another {vocabulary.note}, it is not composable. Fail: a summary of someone's argument Pass: a claim you could invoke while building your own argument ### 3. Novel Not already captured in the vault. Semantic duplicate check AND existing {vocabulary.note_plural} scan both clear. Fail: semantically equivalent to an existing {vocabulary.note} Pass: genuinely new angle not yet articulated ### 4. Connected Relates to existing thinking in the vault. Isolated insights that do not connect to anything are orphans. They rot. Fail: interesting observation about unrelated domain Pass: extends, contradicts, or deepens existing {vocabulary.note_plural} **If ANY criterion fails: do not extract.** --- ## Workflow ### 1. Orient Before reading the source, understand what already exists: ```bash # Get descriptions from existing notes for f in {vocabulary.notes}/*.md; do [[ -f "$f" ]] && echo "=== $(basename "$f" .md) ===" && rg "^description:" "$f" -A 0 done ``` Scan descriptions to understand current {vocabulary.note_plural}. This prevents duplicate extraction and helps identify connection points and enrichment opportunities. ### 2. Read Source Fully Read the ENTIRE source. Understand what it contains, what it argues, what domain it serves. **Planning the extraction:** - How many {vocabulary.note_plural} do you expect from this source? - What categories will be represented? - Is this domain-relevant (comprehensive extraction) or general (gate applies)? **Explicit signal phrases to hunt:** - "the key insight is..." - "this means that..." - "the pattern is..." - "contrary to..." - "the implication..." - "what matters here is..." - "the real issue is..." - "this suggests..." **Implicit signals (the best insights often hide in):** - Problems that imply solutions - Constraints that reveal what works - Failures that suggest approaches - Asides that contain principles - Tangents that reveal mental models **What you are hunting:** - Assertions that could be argued for or against - Patterns that apply beyond this specific source - Insights that change how you think about something - Claims that would be useful to invoke elsewhere ### 3. Categorize FIRST, Then Route (MANDATORY) **STOP. Before ANY filtering, determine the category of each candidate.** This is the critical step that prevents over-rejection. Categorize FIRST, then route to the appropriate extraction path. | Category | How to Identify | Route To | |----------|-----------------|----------| | Core domain {vocabulary.note} | Direct assertion about {vocabulary.domain} | -> {vocabulary.note} (SKIP selectivity gate) | | Implementation idea | Describes a feature, tool, system, or workflow to build | -> methodology note (SKIP selectivity gate) | | Tension/challenge | Describes a conflict, risk, or trade-off | -> tension note (SKIP selectivity gate) | | Validation | Evidence confirming an approach works | -> {vocabulary.note} (SKIP selectivity gate) | | Near-duplicate | Semantic search finds related vault {vocabulary.note} | -> evaluate for enrichment task | | Off-topic claim | General insight not about {vocabulary.domain} | -> apply selectivity gate |