extract

name	extract
description	Extract structured knowledge from source material. Two modes — `extract` (default) for comprehensive atomic claim extraction, and `analyze` for design proposals where one impact-analysis note replaces claim decomposition. A relevance filter ("does this change how we build our project?") applies to all modes. For `extract` mode on domain-relevant sources, skip rate must be below 10%. Triggers on "/extract", "/extract [file]", "extract insights", "mine this", "process this".
version	1.0
generated_from	arscontexta-v1.6
user-invocable	true
allowed-tools	Read, Write, Grep, Glob, mcp__qmd__vsearch
context	fork
model	opus

Runtime Configuration (Step 0 — before any processing)

Read these files to configure domain-specific behavior:

1. arscontexta/ops/derivation-manifest.md — vocabulary mapping, extraction categories, platform hints

   • Use vocabulary.notes for the notes folder name
   • Use vocabulary.inbox for the inbox folder name
   • Use vocabulary.note for the note type name in output
   • Use vocabulary.note_plural for the plural form
   • Use vocabulary.reduce for the process verb in output
   • Use vocabulary.cmd_reflect for the next-phase command name
   • Use vocabulary.cmd_reweave for the backward-pass command name
   • Use vocabulary.cmd_verify for the verification command name
   • Use vocabulary.extraction_categories for domain-specific extraction table
   • Use vocabulary.topic_map for MOC/topic map references
   • Use vocabulary.topic_maps for plural form

2. arscontexta/ops/config.yaml — processing depth, pipeline chaining, selectivity

   • processing.depth: deep | standard | quick
   • processing.chaining: manual | suggested | automatic
   • processing.extraction.selectivity: strict | moderate | permissive

3. arscontexta/ops/queue/queue.json — current task queue (for handoff mode)

If these files don't exist (pre-init invocation or standalone use), use universal defaults:

• depth: standard
• chaining: suggested
• selectivity: moderate
• notes folder: notes/
• inbox folder: arscontexta/inbox/

---

THE MISSION (READ THIS OR YOU WILL FAIL)

You are the extraction engine. Raw source material enters. Structured, atomic notes exit. Everything between is your judgment — and that judgment must err toward extraction, not rejection.

The Core Distinction

Concept	What It Means	Example
Having knowledge	The vault contains information	"We store notes in folders"
Articulated reasoning	The vault explains WHY something works as a traversable note	"folder structure mirrors cognitive chunking because..."

Having knowledge is not the same as articulating it. Even if information is embedded in the system, the vault may lack the externalized reasoning explaining WHY it works. That reasoning is what you extract.

The Comprehensive Extraction Principle

For domain-relevant sources, COMPREHENSIVE EXTRACTION is the default. This means:

1. Extract ALL core notes — direct assertions about the domain that can stand alone as atomic propositions.

2. Extract ALL evidence and validations — if source confirms an approach, that confirmation IS the note. Evidence is extractable even when the conclusion is already known, because the reasoning path matters.

3. Extract ALL patterns and methods — techniques, workflows, practices. Named patterns are referenceable. Unnamed intuitions are not.

4. Extract ALL tensions — contradictions, trade-offs, conflicts. These are wisdom, not problems.

5. Extract ALL enrichments — if source adds detail to existing notes, create enrichment tasks. Near-duplicates almost always add value.

"We already know this" means we NEED the articulation, not that we should skip it.

The Extraction Question (ask for EVERY candidate)

"Would a future session benefit from this reasoning being a retrievable note?"

If YES -> extract to appropriate category If NO -> verify it is truly off-topic before skipping

INVALID Skip Reasons (these are BUGS)

• "validates existing approach" — validations ARE the evidence. Extract them.
• "already captured in system config" — config is implementation, not articulation. The WHY needs a note.
• "we already do this" — DOING is not EXPLAINING. The explanation needs externalization.
• "obvious" — obvious to whom? Future sessions need explicit reasoning.
• "near-duplicate" — near-duplicates almost always add detail. Create enrichment task.
• "not a claim" — is it an implementation idea? tension? validation? Those ARE extractable.

VALID Skip Reasons (rare)

• Completely off-topic (unrelated to llm-do)
• Too vague to act on (applies to everything, disagrees with nothing)
• Pure summary with zero extractable insight
• LITERALLY identical text already exists (not "same topic" — IDENTICAL)

For domain-relevant sources: skip rate < 10%. Zero extraction = BUG.

---

EXECUTE NOW

Target: $ARGUMENTS

Parse immediately:

• If target contains a file path: extract insights from that file
• If target contains --handoff: output RALPH HANDOFF block + task entries at end
• If target is empty: scan arscontexta/inbox/ for unprocessed items, pick one
• If target is "inbox" or "all": process all inbox items sequentially

Execute these steps:

1. Read the source file fully — understand what it contains
2. Source size check: If source exceeds 2500 lines, STOP. Plan chunks of 350-1200 lines. Process each chunk with fresh context. See "Large Source Handling" section below.
3. Hunt for insights that serve the domain (see extraction categories below)
4. For each candidate:
   • Use mcp__qmd__vsearch with query "[claim as sentence]", collection="docs-notes", limit=5 to check for duplicates
   • If duplicate exists: evaluate for enrichment or skip
   • Classify as OPEN (needs more investigation) or CLOSED (standalone, ready)
5. Output extraction report with titles, classifications, extraction rationale
6. Wait for user approval before creating files
7. If --handoff in target: create per-claim task files, update queue, output RALPH HANDOFF block

START NOW. Reference below explains methodology — use to guide, not as output.

Processing Mode Detection

Check the task file (if invoked via queue) for processing_mode:

Mode	Behavior	Skip to Section
extract (default)	Full atomic claim extraction — existing pipeline	Continue below
analyze	One analysis note on impact to our project	Jump to Analyze Mode section

If no task file or no processing_mode field, default to extract.

Relevance Filter Preamble (applies to ALL modes)

Before extracting any claim or writing any analysis, apply this filter:

"Does this change how we think about or build this project?"

• YES → proceed with extraction/analysis
• NO → discard

This filter is the first gate for ALL content in ALL modes. It replaces the assumption that all domain-adjacent content is worth capturing.

For extract mode: Discard off-topic claims (existing category 10), but ALSO discard claims about third-party internals unless they directly inform our design. A claim about how another project implements its internals is not automatically relevant just because we use that project. Ask: "Does knowing this change what WE build?"

For analyze mode: The entire document is filtered through relevance. Don't summarize what the document proposes — surface only what matters for our project. The output is an impact analysis, not a summary.

Observation Capture (during work, not at end)

When you encounter friction, surprises, methodology insights, process gaps, or contradictions — capture IMMEDIATELY:

Observation	Action
Any observation	Create atomic note in arscontexta/ops/observations/ with prose-sentence title
Tension: content contradicts existing note	Create atomic note in arscontexta/ops/tensions/ with prose-sentence title

The handoff Learnings section summarizes what you ALREADY logged during processing.

---

Reduce

Extract composable notes from source material into docs/notes/.

Philosophy

Extract the REASONING behind what works, not just observations about what works.

This is the extraction phase of the pipeline. You receive raw content and extract insights that serve the vault's domain. The mission is building externalized, retrievable reasoning — a graph of atomic propositions that can be traversed, connected, and built upon.

THE CORE DISTINCTION:

Concept	Example	What to Extract
We DO this	"We tag notes with topics"	— (not sufficient)
We explain WHY	"topic tagging enables cross-domain navigation because..."	This

The vault is not just an implementation. It is the articulated argument for WHY the implementation works.

THE EXTRACTION QUESTION:

• BASIC thinking: "Is this a standalone composable claim?"
• BETTER thinking: "Does this serve llm-do?"
• BEST thinking: "Would a future session benefit from this reasoning being a retrievable note?"

If YES -> extract to appropriate category (even if "we already know this") If NO -> skip (RARE for domain-relevant sources — verify it is truly off-topic)

THE RULE: Implementation without articulation is incomplete. If we DO something but lack a note explaining WHY it works, that articulation needs extraction.

---

Extraction Categories

What To Extract

See arscontexta/ops/derivation-manifest.md extraction_categories section

The structural invariant: Every domain's extraction has these universal categories regardless of domain:

Category	What to Find	Output Type	Gate Required?
Core domain notes	Direct assertions about llm-do	note	NO
Patterns	Recurring structures across sources	note	NO
Comparisons	How different approaches compare, X vs Y, trade-offs	note	NO
Tensions	Contradictions, conflicts, unresolved trade-offs	tension note	NO
Anti-patterns	What breaks, what to avoid, failure modes	problem note	NO
Enrichments	Content that adds detail to existing notes	enrichment task	NO
Open questions	Unresolved questions worth tracking	note (open)	NO
Implementation ideas	Techniques, workflows, features to build	methodology note	NO
Validations	Evidence confirming an approach works	note	NO
Off-topic general content	Insight unrelated to llm-do	apply selectivity gate	YES

IMPORTANT: Categories 1-9 bypass the selectivity gate. They extract directly to the appropriate output type. The selectivity gate exists ONLY for filtering off-topic content from general sources.

Category Detection Signals

Hunt for these signals in every source:

Core domain signals:

• Direct assertions: "the key insight is...", "this means that...", "the pattern is..."
• Evidence: "research shows...", "data indicates...", "studies confirm..."
• Named methods: any named system, technique, or framework relevant to llm-do

Comparison signals:

• "X vs Y", "trade-off between...", "prefer X when...", "unlike Y, this..."
• "choose X when...", "depends on whether..."

Tension signals:

• "contrary to...", "however...", "the problem with...", "fails when..."
• "on the other hand...", "but this conflicts with..."

Anti-pattern signals:

• "systems fail when...", "the anti-pattern is...", "avoid this because..."
• Warnings, cautionary examples, failure postmortems

Enrichment signals:

• Content covering ground similar to an existing note
• New examples, evidence, or framing for an established claim
• Deeper explanation of something already captured shallowly

Implementation signals:

• "we could build...", "would enable...", "a tool that...", "pattern for..."
• Actionable techniques, concrete workflows

Validation signals:

• "this supports...", "evidence shows...", "validates...", "confirms..."
• Research that grounds existing practice in theory

The Mission Lens (REQUIRED)

For EVERY candidate, ask: "Does this serve llm-do?"

• YES -> extract to appropriate category (gate does NOT apply)
• NO -> apply selectivity gate (for off-topic filtering only)

For domain-relevant sources: almost everything is YES. The gate barely applies. Skip rate < 10%.

---

The Selectivity Gate (for OFF-TOPIC content filtering)

CRITICAL: This gate exists to filter OUT content that does not serve llm-do. It applies ONLY to standard claims from GENERAL (off-topic) sources.

Do NOT use gate to reject:

• Implementation ideas ("not a claim" is WRONG — it is roadmap)
• Tensions ("not a claim" is WRONG — it is wisdom)
• Enrichments ("duplicate" is WRONG — it adds detail)
• Validations ("already known" is WRONG — it is evidence)
• Open questions ("not testable" is WRONG — it is direction)

For STANDARD claims from general sources, verify all four criteria pass:

1. Standalone

The claim is understandable without source context. Someone reading this note cold can grasp what it argues without needing to know where it came from.

Fail: "the author's third point about methodology" Pass: "explicit structure beats implicit convention"

2. Composable

This note would be linked FROM elsewhere. notes function as APIs. If you cannot imagine writing since [[this claim]]... in another note, it is not composable.

Fail: a summary of someone's argument Pass: a claim you could invoke while building your own argument

3. Novel

Not already captured in the vault. Semantic duplicate check AND existing notes scan both clear.

Fail: semantically equivalent to an existing note Pass: genuinely new angle not yet articulated

4. Connected

Relates to existing thinking in the vault. Isolated insights that do not connect to anything are orphans. They rot.

Fail: interesting observation about unrelated domain Pass: extends, contradicts, or deepens existing notes

If ANY criterion fails: do not extract.

---

Workflow

1. Orient

Before reading the source, understand what already exists:

# Get descriptions from existing notes
for f in docs/notes/*.md; do
  [[ -f "$f" ]] && echo "=== $(basename "$f" .md) ===" && rg "^description:" "$f" -A 0
done

Scan descriptions to understand current notes. This prevents duplicate extraction and helps identify connection points and enrichment opportunities.

2. Read Source Fully

Read the ENTIRE source. Understand what it contains, what it argues, what domain it serves.

Planning the extraction:

• How many notes do you expect from this source?
• What categories will be represented?
• Is this domain-relevant (comprehensive extraction) or general (gate applies)?

Explicit signal phrases to hunt:

• "the key insight is..."
• "this means that..."
• "the pattern is..."
• "contrary to..."
• "the implication..."
• "what matters here is..."
• "the real issue is..."
• "this suggests..."

Implicit signals (the best insights often hide in):

• Problems that imply solutions
• Constraints that reveal what works
• Failures that suggest approaches
• Asides that contain principles
• Tangents that reveal mental models

What you are hunting:

• Assertions that could be argued for or against
• Patterns that apply beyond this specific source
• Insights that change how you think about something
• Claims that would be useful to invoke elsewhere

3. Categorize FIRST, Then Route (MANDATORY)

STOP. Before ANY filtering, determine the category of each candidate.

This is the critical step that prevents over-rejection. Categorize FIRST, then route to the appropriate extraction path.

Category	How to Identify	Route To
Core domain note	Direct assertion about llm-do	-> note (SKIP selectivity gate)
Implementation idea	Describes a feature, tool, system, or workflow to build	-> methodology note (SKIP selectivity gate)
Tension/challenge	Describes a conflict, risk, or trade-off	-> tension note (SKIP selectivity gate)
Validation	Evidence confirming an approach works	-> note (SKIP selectivity gate)
Near-duplicate	Semantic search finds related vault note	-> evaluate for enrichment task
Off-topic claim	General insight not about llm-do	-> apply selectivity gate

CRITICAL: Implementation ideas, tensions, validations, and domain notes do NOT need to pass the 4-criterion selectivity gate. The gate is for off-topic filtering ONLY.

Why this matters: The selectivity gate was designed for filtering general insights. But implementation ideas ("build a trails feature"), tensions ("optimization vs readability trade-off"), and validations ("research confirms our approach") are DIFFERENT output types that serve different purposes. Applying the selectivity gate to them is a category error.

4. Semantic Search for Duplicates and Enrichment

For each candidate, run duplicate detection:

mcp__qmd__vsearch  query="[proposed claim as sentence]"  collection="docs-notes"  limit=5

Why vsearch (vector semantic) instead of keyword search: Duplicate detection is where keyword search fails hardest. A claim about "friction in systems" will not find "resistance to change" via keyword matching even though they may be semantic duplicates. Vector search (~5s) catches same-concept-different-words duplicates that keyword search misses entirely. For a batch of 30-50 candidates, this adds ~3 minutes total — worth it to catch duplicates early rather than discovering them during /connect.

Scores are signals, not decisions. For ANY result with a relevant title or snippet:

1. READ the full note
2. Compare: is this the SAME claim in different words?
3. Ask: "What does source add that existing note lacks?"

The Enrichment Judgment (DEFAULT TO ENRICHMENT):

Situation	Action
Exact text already exists	SKIP (truly identical — RARE)
Same claim, different words, source adds nothing	SKIP (verify by re-reading existing note)
Same claim, source has MORE detail/examples/framing	-> ENRICHMENT TASK (update existing note)
Same topic, DIFFERENT claim	-> EXTRACT as new note, flag for cross-linking
Related mechanism, different scope	-> EXTRACT as new note, flag for cross-linking

DEFAULT TO ENRICHMENT. If source mentions the same topic, it almost certainly adds something. Truly identical content is RARE.

MANDATORY protocol when semantic search finds overlap:

1. READ the existing note fully (not just title/description)
2. Ask: "What does source ADD that existing note LACKS?"
   • New examples -> ENRICHMENT
   • Deeper framing -> ENRICHMENT
   • Citations/evidence -> ENRICHMENT
   • Different angle -> ENRICHMENT
   • Concrete implementation -> ENRICHMENT
   • Literally identical -> skip (RARE)
3. If source adds ANYTHING: CREATE ENRICHMENT TASK
4. Only skip if source adds literally NOTHING new (verify this claim)

Near-duplicates are opportunities, not rejections. Creating enrichment tasks is CORRECT behavior. If you are skipping near-duplicates without enrichment tasks, you are probably wrong.

5. Classify Each Extraction

Every extracted candidate gets classified:

• CLOSED — standalone claim, design decision, ready for processing as-is
• OPEN — needs more investigation, testable hypothesis, requires evidence

Classification affects downstream handling but does NOT affect whether to extract. Both open and closed candidates get extracted.

6. Present Findings

Report what you found by category. Include counts:

Extraction scan complete.

SUMMARY:
- notes: N
- implementation ideas: N
- tensions: N
- enrichment tasks: N
- validations: N
- open questions: N
- skipped: N
- TOTAL OUTPUTS: N

---

CLAIMS (notes):
1. [claim as sentence] — connects to [[existing note]]
2. [claim as sentence] — extends [[existing note]]
...

IMPLEMENTATION IDEAS (methodology notes):
1. [feature/pattern] — what it enables, why it matters
...

TENSIONS (tension notes):
1. [X vs Y] — the conflict, why it matters
...

ENRICHMENT TASKS (update existing notes):
1. [[existing note]] — source adds [what is missing]
...

SKIPPED (truly nothing to add):
- [description] — why nothing extractable

Wait for user approval before creating files. Never auto-extract.

7. Extract (With User Approval)

For each approved note:

a. Craft the title

The title IS the claim. Express the concept in exactly the words that capture it.

Test: "this note argues that [title]"

• Must make grammatical sense
• Must be something you could agree or disagree with
• Composability over brevity — a full sentence is fine if the concept requires it
• Lowercase with spaces
• No punctuation that breaks filesystems: . * ? + [ ] ( ) { } | \ ^

Good: "explicit structure beats implicit convention for agent navigation" Good: "small differences compound through repeated selection" Bad: "context management strategies" (topic label, not a claim)

b. Write the note

---
description: [~150 chars elaborating the claim, adds info beyond title]
type: [claim | methodology | problem | learning | tension]
created: YYYY-MM-DD
[domain-specific fields from derivation-manifest]
---

# [prose-as-title proposition]

[Body: 150-400 words showing reasoning]

Use connective words: because, but, therefore, which means, however.
Acknowledge uncertainty where appropriate.
Consider the strongest counterargument.
Show the path to the conclusion, not just the conclusion.

---

Source: [[source filename]]

Relevant Notes:
- [[related claim]] — [why it relates: extends, contradicts, builds on]

Topics:
- [[relevant index]]

c. Verify before writing

• Title passes the claim test ("this note argues that [title]")
• Description adds information beyond the title (not a restatement)
• Body shows reasoning, not just assertion
• At least one relevant note connection identified
• At least one index link
• Source attribution present

d. Create the file

Write to: docs/notes/[title].md

---

Analyze Mode

When processing_mode: analyze is set in the task file, use this section instead of the standard extraction workflow.

Analyze mode is for external design proposals, RFCs, and specs — documents that propose changes to ANOTHER project where we need to assess impact on OUR project. The goal is not to decompose the document into atomic claims, but to understand it as a whole and surface what matters for us.

When Analyze Mode Applies

• External API proposals (e.g., a traits/personality system for an LLM provider)
• RFCs or specs from upstream dependencies
• Design documents from projects we integrate with
• Large PRs or proposals that affect our architecture

What Analyze Mode Does NOT Do

• Decompose into atomic claims
• Apply the < 10% skip rate mandate (this mandate is for extract mode only)
• Create numbered claim files
• Apply the comprehensive extraction principle

Analyze Mode Workflow

1. Read the source fully — understand what it proposes, its scope, its architecture.

2. Read the task file — it contains a summary, guiding questions, and related existing notes (populated by /seed).

3. Apply the relevance filter — for every section of the document, ask: "Does this change how we think about or build our project?" Most sections of an external proposal will NOT pass this filter. That's correct.

4. Write one analysis note answering:

• What does this proposal mean for our project?
• Which of our existing design decisions does it affect?
• What should we watch for as this evolves?
• What existing notes need updating?

5. Connect — link the analysis note to affected existing notes. Update those notes if the proposal changes their context.

Analyze Mode Output

One analysis note in docs/notes/, following the standard note schema:

---
description: [what this proposal means for our project — ~150 chars]
type: claim
created: YYYY-MM-DD
---

# [impact claim as prose title]

[Analysis body — what the proposal means for us, not a summary of the proposal itself.
Focus on: implications for our architecture, changes to assumptions in existing notes,
new constraints or opportunities, what to watch for.]

---

Source: [[source filename]]

Relevant Notes:
- [[affected note 1]] — [how this proposal changes the context]
- [[affected note 2]] — [what needs updating]

Topics:
- [[relevant index]]

Optionally, if the analysis reveals that specific existing notes need updating, create enrichment tasks or update the notes directly (with user approval).

Analyze Mode Report

Extraction scan complete (analyze mode).

SUMMARY:
- Analysis notes: 1
- Existing notes affected: N
- Enrichment tasks: N (if any)

ANALYSIS:
[title of analysis note] — [one-sentence summary]

AFFECTED NOTES:
- [[note 1]] — [how affected]
- [[note 2]] — [how affected]

ENRICHMENT TASKS (if any):
- [[existing note]] — source changes context for [aspect]

Wait for user approval before creating files.

---

Large Source Handling

For sources exceeding 2500 lines: chunk processing is MANDATORY.

Context degrades as it fills. A single-pass extraction of a 3000-line source will miss insights in the later sections because your attention has degraded by the time you reach them. Chunking ensures each section gets fresh attention.

Chunking Strategy

Source Size	Chunk Count	Chunk Size	Rationale
2500-4000 lines	3-4 chunks	700-1200 lines	Standard chunking
4000-6000 lines	4-5 chunks	800-1200 lines	Balanced attention
6000+ lines	5+ chunks	1000-1500 lines	Prevent context overflow

Chunk boundaries: Split at natural section breaks (headings, topic transitions). Never split mid-paragraph or mid-argument. A chunk should be a coherent unit of content.

Processing Depth Adaptation

Depth (from config)	Chunking Behavior
deep	Fresh context per chunk (spawn subagent per chunk if platform supports). Maximum quality.
standard	Process chunks sequentially in current session. Reset orientation between chunks.
quick	Larger chunks (1500-2000 lines). Fewer, faster passes.

Cross-Chunk Coordination

When processing in chunks:

1. Keep a running list of extracted notes across chunks
2. Later chunks check against earlier chunks' extractions (not just existing vault notes)
3. Cross-chunk connections get flagged for /connect
4. The final extraction report covers ALL chunks combined

The anti-pattern: Processing chunk 3 and extracting a duplicate of something already extracted in chunk 1 because you lost track. Maintain the running list.

---

Enrichment Detection

When source content adds value to an EXISTING note rather than creating a new one, create an enrichment task instead.

When to Create Enrichment Tasks

Signal	Action
Source has better examples for an existing note	Enrichment: add examples
Source has deeper framing or context	Enrichment: strengthen reasoning
Source has citations or evidence	Enrichment: add evidence base
Source has a different angle on the same claim	Enrichment: add perspective
Source has concrete implementation details	Enrichment: add actionable specifics

Enrichment Task Format

Each enrichment task specifies:

• Target: Which existing note to enrich (by title)
• What to add: Specific content from the source
• Why: What the existing note lacks that this adds
• Source lines: Where in the source the enrichment content is found

The enrichment default: When in doubt between "new note" and "enrichment to existing note", lean toward enrichment. The existing note already has connections, index placement, and integration. Adding to it compounds existing value.

---

Quality Gates

Red Flags: Extraction Too Tight (THE COMMON FAILURE MODE)

If you catch yourself doing ANY of these, STOP IMMEDIATELY and recalibrate:

The Cardinal Sins (NEVER do these)

1. "validates existing approach" as skip reason

   • WRONG: "This just confirms what we do, skip"
   • RIGHT: Validations ARE valuable. Extract as note with evidence framing.
   • WHY: Future sessions need to see WHY an approach is validated, not just that it works.

2. "already captured in system config" as skip reason

   • WRONG: "We already have this in our config, skip"
   • RIGHT: Extract "session handoff creates continuity without persistent memory"
   • WHY: Config is implementation. notes explain WHY it works.

3. "we already do this" as skip reason

   • WRONG: "We use wiki links, this is obvious, skip"
   • RIGHT: Extract the reasoning that explains WHY it works
   • WHY: DOING is not EXPLAINING. The reasoning needs externalization.

4. "obvious" or "well known" as skip reason

   • WRONG: "Everyone knows structure helps, skip"
   • RIGHT: Extract the specific, named, referenceable claim
   • WHY: Named patterns are referenceable. Unnamed intuitions are not.

5. Treating near-duplicates as skips instead of enrichments

   • WRONG: "Similar to existing note, skip"
   • RIGHT: Create enrichment task to add source's details to existing note
   • WHY: Near-duplicates almost always add framing, examples, or evidence.

Other Red Flags

• Rejecting implementation ideas as "not claims" (they ARE extractable as methodology notes)
• Rejecting tensions as "not claims" (they become tension notes)
• Zero extraction from a domain-relevant source (the source IS about your domain)
• Rejecting open questions as "not testable" (directions guide future work)
• Applying the 4-criterion gate to non-standard-claim categories (gate is for off-topic filtering)
• Skip rate > 10% on domain-relevant sources (most domain content should extract to SOME category)

The Test

Before skipping ANYTHING, ask: "Would a future session benefit from this being a retrievable note?"

If YES -> extract (even if "we already know this") If NO -> verify it is truly off-topic or literally identical to existing content

Red Flags: Extraction Too Loose

• Extracting vague observations with no actionable content
• Creating notes without articulating vault connection
• Titles that are topics, not claims ("knowledge management" instead of "knowledge management fails without active maintenance")
• Body text that is pure summary without reasoning

Calibration Check (REQUIRED Before Finishing)

STOP before outputting results. Count your outputs by category:

notes extracted: ?
implementation ideas: ?
tensions: ?
enrichment tasks: ?
validations: ?
open questions: ?
truly skipped: ?
TOTAL: ?

Expected yields by source size:

Source Size	Expected Outputs	Skip Rate
~100 lines	5-10 outputs	varies by relevance
~350 lines	15-30 outputs	< 10% for domain-relevant
~500+ lines	25-50+ outputs	< 10% for domain-relevant
~1000+ lines	40-70 outputs	< 5% for domain-relevant

Zero extraction from a domain-relevant source is a BUG.

If your total outputs are significantly below these ranges, you are over-filtering.

Selectivity Adaptation

Processing selectivity adapts based on arscontexta/ops/config.yaml:

Selectivity (config)	Gate Behavior	Skip Rate Target
strict	4-criterion gate applies to ALL claims including domain-relevant	Higher skip rate acceptable
moderate (default)	Gate applies only to off-topic content. Domain-relevant bypasses gate	< 10% for domain sources
permissive	Gate barely applies. Extract nearly everything, heavy enrichment	< 5% overall

Strict mode is for mature vaults where noise reduction matters more than coverage. Permissive mode is for new vaults building initial density. Moderate is the default — comprehensive extraction for domain content, selective for off-topic.

Mandatory Review If Low Yield

Go back through candidates you marked as "duplicate" or "rejected":

1. Did any "duplicates" have source content that enriches existing notes?

   • YES -> convert to enrichment task (DEFAULT TO ENRICHMENT)
   • NO -> verify by re-reading existing note FULLY

2. Did any "rejected" items describe features to build?

   • YES -> extract as implementation idea
   • NO -> verify it is truly unactionable

3. Did any "rejected" items describe conflicts or challenges?

   • YES -> extract as tension note
   • NO -> verify it is truly vague

4. Did any "rejected" items provide evidence for existing approaches?

   • YES -> extract as validation claim
   • NO -> verify it does not support existing methodology

5. Did any "rejected" items suggest questions worth investigating?

   • YES -> extract as open question note
   • NO -> verify it is not worth tracking

Do not proceed with handoff until low yield is investigated.

---

Note Design Reference

Titles

Titles are claims that work as prose when linked:

since [[explicit structure beats implicit convention]], the question becomes...
the insight is that [[small differences compound through repeated selection]]
because [[capture speed beats filing precision]], we separate the two...

The claim test: "this note argues that [title]"

Example	Passes?
quality requires active judgment	yes: "argues that quality requires active judgment"
knowledge management	no: "argues that knowledge management" (incomplete)
small differences compound through selection	yes: "argues that small differences compound through selection"
tools for thought	no: "argues that tools for thought" (incomplete)

Description

One field. ~150 characters. Must add NEW information beyond the title — scope, mechanism, or implication.

Bad (restates title): "quality is important in knowledge work" Good (adds mechanism + implication): "when creation becomes trivial, maintaining signal-to-noise becomes the primary challenge — selection IS the work"

The description is progressive disclosure: title says WHAT the claim is, description says WHY it matters or HOW it works. If the description just rephrases the title, it wastes context and provides no filter value.

Body

Show reasoning. Use connective words. Acknowledge uncertainty.

Bad:

Quality matters. When creation is easy, curation becomes the work.

Good:

The easy part is capture. We bookmark things, save screenshots, clip articles we never open again. The hard part is doing something with it all. Automation makes this worse because generation is now trivial — anyone can produce endless content. So the constraint shifts from production to selection. Since [[structure without processing provides no value]], the question becomes: who does the selecting?

Characteristics:

• Conversational flow (because, but, therefore)
• Shows path to conclusion
• Acknowledges where thinking might be wrong
• Considers strongest objection
• Invokes other notes as prose

Section Headings

Headings serve navigation, not decoration. Use when agents would benefit from grepping the outline.

Always use headings for:

• Tension notes (sections: Quick Test, When Each Pole Wins, Dissolution Attempts, Practical Applications)
• index notes (sections: Synthesis, Core Ideas, Tensions, Explorations Needed, Agent Notes)
• Implementation patterns with discrete steps
• Notes exploring multiple facets of a concept (>1000 words AND distinct sub-topics)

Use prose without headings for:

• Single flowing arguments under ~1000 words
• Notes where transitions like "since [[X]]..." already carry structure

Footer

---

Source: [[source filename]]

Relevant Notes:
- [[related claim]] — extends this by adding the temporal dimension

Topics:
- [[relevant index]]

The relationship context explains WHY to follow the link:

• Bad: "-- related"
• Good: "-- contradicts by arguing for explicit structure"
• Good: "-- provides the foundation this challenges"

---

The Composability Test

Before finalizing ANY note, verify:

1. Standalone Sense If you link to this note from another context, will it make sense without reading three other notes first?

2. Specificity Could someone disagree with this claim? Vague notes cannot be built on.

3. Clean Linking Would linking to this note drag unrelated content along? If yes, the note covers too much.

When to skip: content does not pass all four selectivity criteria (off-topic content only) When to split: multiple distinct claims in one extraction When to sharpen: claim too vague, title is label not statement

---

Research Provenance

When the source file contains provenance metadata (source_type, research_prompt, research_server, generated), preserve the chain:

• Each created note's Source footer links to the source file
• The source file's YAML contains the research prompt
• The chain: research query -> inbox file -> /extract -> docs/notes

If source has source_type in frontmatter, this is research-generated content — handle with extra care for attribution.

Provenance fields to preserve:

Field	Purpose
source_type	How this content was generated
research_prompt	The query or directive that produced this content
research_server	Which research tool was used
generated	When the research was produced

The research_prompt is the most critical field — it captures the intellectual context that shaped what was returned. Knowing "I searched for X because I was exploring Y" is part of the knowledge graph.

---

Example: What Good Extraction Looks Like

Example 1: 300-line domain-relevant source

Source: 300-line research document directly relevant to llm-do

Scan found: ~45 items across sections

Extraction results:

• 12 core notes
• 6 implementation ideas -> methodology notes
• 4 tensions -> tension notes
• 5 enrichment tasks -> update existing notes
• 3 validations -> notes
• 3 skipped (too vague to act on)

Total: 30 outputs, 3 skipped (~9% skip rate)

Example 2: 100-line general article

Source: 100-line article with partial relevance to llm-do

Extraction results:

• 4 core notes
• 1 enrichment task
• 2 skipped (off-topic)
• 3 skipped (too vague)

Total: 5 outputs, 5 skipped (50% skip rate — acceptable for general source)

Contrast: WRONG Behavior

• 45 candidates -> 0 outputs (everything "rejected as duplicate or not a claim")
• Treating implementation ideas as "not claims" and skipping
• Treating tensions as "not claims" and skipping
• Treating near-duplicates as skips instead of enrichment tasks
• Skip rate > 10% on a domain-relevant source

---

Critical

Never auto-extract. Always present findings and wait for user approval.

When in doubt, extract. For domain-relevant sources, err toward capturing. Implementation ideas, tensions, validations, open questions, and near-duplicates all have value — they become different output types, not rejections.

The principle: the goal is to capture everything relevant to llm-do. For domain-relevant sources, that is MOST of the content. The selectivity gate exists for OFF-TOPIC filtering, not for rejecting on-mission content that happens to have a different form.

Remember:

• Implementation ideas are NOT "not claims" — they are roadmap
• Tensions are NOT "not claims" — they are wisdom
• Enrichments are NOT "duplicates" — they add detail
• Validations are NOT "already known" — they are evidence
• Open questions are NOT "not testable" — they are guidance

For domain-relevant sources: skip rate < 10%. Zero extraction = BUG.

---

Handoff Mode (--handoff flag)

When invoked with --handoff, this skill handles queue management for orchestrated execution. This includes creating per-claim task files and updating the task queue.

Detection: Check if $ARGUMENTS contains --handoff.

Per-Claim Task Files (REQUIRED in handoff mode)

After extraction, for EACH claim, create a task file in arscontexta/ops/queue/:

Filename: {source}-NNN.md where:

• {source} is the source basename (from the extract task)
• NNN is the claim number, starting from next_claim_start in the extract task file

Example: If article-name.md task has next_claim_start: 010, claims are:

• article-name-010.md, article-name-011.md, etc.

Why unique names: Claim filenames must be unique across the entire vault. Claim numbers are global and never reused across batches. The pattern {source}-NNN.md ensures every claim file is uniquely identifiable even after archiving.

Structure:

---
claim: "[the claim as a sentence]"
classification: closed | open
source_task: [source-basename]
semantic_neighbor: "[related note title]" | null
---

# Claim NNN: [claim title]

Source: [[source filename]] (lines NNN-NNN)

## Reduce Notes

Extracted from [source_task]. This is a [CLOSED/OPEN] claim.

Rationale: [why this claim was extracted, what it contributes]

Semantic neighbor: [if found, explain why DISTINCT not DUPLICATE]

---

## Create
(to be filled by create phase)

## /connect
(to be filled by /connect phase)

## /revisit
(to be filled by /revisit phase)

## /review
(to be filled by /review phase)

Enrichment Task Files (REQUIRED in handoff mode)

For each ENRICHMENT detected, create a task file in arscontexta/ops/queue/:

Filename: {source}-EEE.md where:

• {source} is the source basename (same as claims)
• EEE is the enrichment number, continuing from where claims left off

Example: If claims are 010-015, enrichments start at 016.

Why unique names: Enrichments share the numbering system with claims. Both use the global next_claim_start counter. This ensures every task file is uniquely identifiable across the entire vault.

Structure:

---
type: enrichment
target_note: "[[existing note title]]"
source_task: [source-basename]
addition: "what to add from source"
source_lines: "NNN-NNN"
---

# Enrichment EEE: [[existing note title]]

Source: [[source filename]] (lines NNN-NNN)

## Reduce Notes

Enrichment for [[existing note title]]. Source adds [what it adds].

Rationale: [why this enriches rather than duplicates]

---

## Enrich
(to be filled by enrich phase)

## /connect
(to be filled by /connect phase)

## /revisit
(to be filled by /revisit phase)

## /review
(to be filled by /review phase)

Queue Updates (REQUIRED in handoff mode)

After creating task files, update arscontexta/ops/queue/queue.json:

1. Mark the extract task as "status": "done" with completion timestamp
2. For EACH claim, add ONE queue entry:

{
  "id": "claim-NNN",
  "type": "claim",
  "status": "pending",
  "target": "[claim title]",
  "classification": "closed|open",
  "batch": "[source-basename]",
  "file": "[source-basename]-NNN.md",
  "created": "[ISO timestamp]",
  "current_phase": "create",
  "completed_phases": []
}

3. For EACH enrichment, add ONE queue entry:

{
  "id": "enrich-EEE",
  "type": "enrichment",
  "status": "pending",
  "target": "[existing note title]",
  "source_detail": "[what to add]",
  "batch": "[source-basename]",
  "file": "[source-basename]-EEE.md",
  "created": "[ISO timestamp]",
  "current_phase": "enrich",
  "completed_phases": []
}

Critical queue rules:

• ONE entry per claim (NOT one per phase) — phase progression is tracked via current_phase and completed_phases
• type is "claim" or "enrichment" — these are the task's single queue entries
• Every task MUST have "file" pointing to its uniquely-named task file
• Every task MUST have "batch" identifying which source batch it belongs to
• Task IDs use claim-NNN or enrich-EEE format with the global claim number
• Claim numbers are global and never reused across batches
• current_phase starts at "create" for claims, "enrich" for enrichments
• The orchestrator advances phases through the configured phase_order sequence

Claim Numbering

• Start from next_claim_start value in the extract task file (set by /seed)
• /seed calculated this by checking the queue and archive for the highest existing claim number
• Example: if highest claim in vault is 009, next_claim_start will be 010
• Claim numbers are GLOBAL and never reused across batches
• Enrichments continue the same numbering sequence after claims

Handoff Output Format

After creating files and updating queue, output:

=== RALPH HANDOFF: reduce ===
Target: [source file]

Work Done:
- Extracted N claims from [source]
- Created claim files: {source}-NNN.md through {source}-NNN.md
- Created M enrichment files: {source}-EEE.md through {source}-EEE.md (if any)
- Duplicates skipped: [list or "none"]
- Semantic neighbors flagged for cross-linking: [list or "none"]

Files Modified:
- arscontexta/ops/queue/{source}-NNN.md (claim files)
- arscontexta/ops/queue/{source}-EEE.md (enrichment files, if any)
- arscontexta/ops/queue/queue.json (N claim tasks + M enrichment tasks, 1 entry each)

Learnings:
- [Friction]: [description] | NONE
- [Surprise]: [description] | NONE
- [Methodology]: [description] | NONE
- [Process gap]: [description] | NONE

Queue Updates:
- Mark: {source} done
- Create: claim-NNN entries (1 per claim, current_phase: "create")
- Create: enrich-EEE entries (1 per enrichment, current_phase: "enrich", if any)
=== END HANDOFF ===

Critical: The handoff mode adds queue management ON TOP of the standard reduce workflow. Do the full extraction workflow first, then create task files, update queue, and output handoff.

Queue Update (Interactive Execution)

When running interactively (NOT via orchestrator), YOU must execute the queue updates. The orchestrator parses the handoff block and handles this automatically, but interactive sessions do not.

After completing extraction, update the queue:

# Get timestamp
TIMESTAMP=$(date -u +"%Y-%m-%dT%H:%M:%SZ")

# Mark extract task done (replace TASK_ID with actual task ID)
jq '(.tasks[] | select(.id=="TASK_ID")).status = "done" | (.tasks[] | select(.id=="TASK_ID")).completed = "'"$TIMESTAMP"'"' arscontexta/ops/queue/queue.json > tmp.json && mv tmp.json arscontexta/ops/queue/queue.json

The handoff block's "Queue Updates" section is not just output — it is your own todo list when running interactively.

---

Skill Selection Routing

When processing content, route to the correct skill:

Task Type	Required Skill	Why
New content to process	/extract	Extraction requires quality gates
note just created	//connect	New notes need connections
After connecting	//revisit	Old notes need updating
Quality check	//review	Combined verification gate
System health	/health	Systematic diagnostics

Pipeline Chaining

After extraction completes, output the next step based on arscontexta/ops/config.yaml pipeline chaining mode:

• manual: Output "Next: /connect [created notes]" — user decides when to proceed
• suggested: Output next step AND add each created note to arscontexta/ops/queue/queue.json with current_phase: "create" and completed_phases: []
• automatic: Queue entries created and processing continues immediately via orchestration

The chaining output uses domain-native command names from the derivation manifest.