docs-notes-retrieval

name	docs-notes-retrieval
description	Builds local-first note vault retrieval for Obsidian, markdown notebooks, and NotebookLM-adjacent exports. Use when packaging notes into LLM-ready context.
compatibility	Portable core. Works on Claude Code and Codex.
version	1.1
last_validated	2026-07-11T00:00:00.000Z

Notes Retrieval

Quick Start — Concrete Commands

Three common scenarios using the scripts in scripts/.

Scenario 1: Vault inventory

Before indexing or packaging, understand what is in the vault.

# Full inventory as JSON — one record per note with title, frontmatter, wikilinks, tags, word count, mtime
python scripts/scan_vault.py inventory /path/to/vault > vault-inventory.json

# Tag summary sorted by note count
python scripts/scan_vault.py tags /path/to/vault

# CSV for spreadsheet review
python scripts/scan_vault.py inventory /path/to/vault --format csv > vault-inventory.csv

Scenario 2: Packaged context for an LLM session

Build a markdown bundle ready to paste into a Claude or GPT session.

# Keyword search — include all notes matching "weekly review", budget 40k chars
python scripts/build_context_pack.py /path/to/vault \
    --query "weekly review" \
    --max-chars 40000 > context-pack.md

# Specific notes with heading-level chunking
python scripts/build_context_pack.py /path/to/vault \
    --notes "Projects/Alpha.md" "Projects/Beta.md" \
    --chunk-strategy heading \
    --max-chars 60000 > project-alpha-pack.md

# Most recent notes first, paragraph chunking, written to file
python scripts/build_context_pack.py /path/to/vault \
    --query "architecture decision" \
    --order-by recency \
    --chunk-strategy paragraph \
    --out session-context.md

Scenario 3: Orphan note cleanup

Find and review notes with no inbound or outbound wikilinks — usually stale or misplaced.

# List orphans as JSON
python scripts/scan_vault.py orphans /path/to/vault

# Pipe into jq to see just paths and word counts
python scripts/scan_vault.py orphans /path/to/vault | jq '.[] | {path, word_count}'

# CSV for manual triage
python scripts/scan_vault.py orphans /path/to/vault --format csv > orphans.csv

Orphan cleanup workflow:

1. Run orphans to get the list.
2. Review word count: orphans < 50 words are likely stubs — archive or delete.
3. Orphans > 200 words with no links may be valuable but unconnected — add wikilinks or a backlink from a map-of-content note.
4. Re-run orphans after cleanup to verify the count dropped.

Judgment call before acting on the list: scan_vault.py orphans only detects [[wikilink]] topology. In vaults that organize by folder hierarchy or tags instead of wikilinks (common in Logseq-style or PARA-folder vaults), a genuinely well-connected note can show up as a false-positive orphan. Cross-check folder placement and tags before archiving anything the tool flags — treat the orphan list as a triage queue, not a deletion list.

---

Use this skill to build or operate local-first retrieval pipelines for note vaults, markdown knowledge bases, and notebook-style exports.

This skill covers:

• Obsidian-style markdown vaults with frontmatter, wikilinks, tags, and aliases
• Local notebook exports and source packs prepared for NotebookLM-style workflows
• Raw ingest folders such as transcript exports that feed a canonical note system
• Read-only extraction of note text, metadata, backlinks, and neighborhood context
• Packaging note sets into Markdown or retrieval-ready context for LLMs, agents, and MCP tools
• Optional write-back routing of summaries, actions, and decisions into canonical notes when the user explicitly wants synchronized notes

Quick Reference

Need	Default path	Notes
Inspect a vault before indexing	references/obsidian-and-local-vaults.md	Inventory note types, frontmatter conventions, attachments, and private areas first
Run vault inventory from the CLI	scripts/scan_vault.py inventory	Extracts path, title, frontmatter, wikilinks, tags, word count, mtime
Find orphan notes (no links)	scripts/scan_vault.py orphans	Returns notes with no inbound or outbound wikilinks
List all tags with counts	scripts/scan_vault.py tags	Sorted by note count descending
Package notes for LLM context	scripts/build_context_pack.py + references/context-packaging-patterns.md	Keyword or explicit-note selection; whole/heading/paragraph chunking; char budget
Understand chunk strategies and token budgets	references/context-packaging-patterns.md	Char→token ratios, ordering, deduplication, attribution format
Avoid retrieval mistakes	references/retrieval-anti-patterns.md	8 anti-patterns with fixes
Structure a vault as durable agent memory	references/obsidian-and-local-vaults.md	Separate session memory, canonical notes, and ingest artifacts instead of one flat note dump
Turn a vault into an operating memory loop	references/obsidian-and-local-vaults.md	Keep notes canonical and treat transcripts or exports as ingest inputs
Use AI-owned wiki (solo, high-volume capture)	references/ai-curated-wiki-pattern.md	Folder layout, invariants, compounding loop, scale heuristic, snapshots
Decide note-vault retrieval vs generic RAG	../ai-rag/SKILL.md + this skill	Use this skill for note-shape and packaging rules; use ai-rag for retrieval architecture
Turn note access into a repeatable tool	../agents-mcp/SKILL.md	Build an MCP wrapper only after the local note workflow is clear

Supported Inputs

Default support in v1:

• markdown note vaults
• Obsidian frontmatter and wikilinks
• folder-based personal or team knowledge bases
• NotebookLM-adjacent exported note packs, summaries, and source bundles
• transcript or export folders that are later summarized into canonical notes
• read-only local file trees prepared for agent retrieval

Explicitly out of scope in v1:

• direct NotebookLM API automation
• mutating or auto-rewriting the source vault by default
• OCR-heavy attachment pipelines
• closed-source vendor note apps without an export path

Three-Layer Operating Memory Pattern

For agent workflows, separate these three layers — collapsing them into one pool is the most common vault anti-pattern:

Layer	Role	Examples
Session memory	Orients the agent; always loaded, tiny	Onboarding notes, repo CLAUDE.md, orientation files
Knowledge graph	Persists and connects knowledge	Canonical notes with frontmatter, wikilinks, stable destinations
Ingest pipeline	Evidence and raw material	Transcripts, exports, recordings, raw captures

AI-Curated Wiki Variant (Ganim Pattern)

The three-layer default assumes humans curate the canonical layer. A second mode exists where the AI owns the curated layer entirely and the human only appends to a junk-drawer source folder. Use this variant for solo operators capturing source material faster than they can hand-organize it.

Full details, invariants, compounding loop, monthly health check, scale heuristic (~100 articles / ~400K words before vector RAG is needed), image-aware ingest, and snapshot strategy: references/ai-curated-wiki-pattern.md.

ASCII Flow

Notes request
  |
  v
Classify source shape
  |-- Obsidian vault
  |-- markdown folder
  |-- notebook export
  |-- transcript / raw ingest dump
  |-- mixed notes repo
  |
  v
Separate layers
  |-- session memory -----> tiny always-loaded orientation
  |-- canonical notes ----> durable graph with metadata + links
  |-- raw ingest ---------> transcripts, exports, recordings
  |
  v
Inventory metadata + topology
  |-- frontmatter
  |-- wikilinks / backlinks
  |-- tags / aliases
  |-- private folders / attachments
  |
  v
Choose output
  |-- Markdown context pack ----> scripts/build_context_pack.py
  |-- vault report -------------> scripts/scan_vault.py
  |-- retrieval architecture ---> ai-rag
  |-- repeatable tool ----------> agents-mcp
  |
  v
Package with stable attribution and budget limits

Default Workflow

1. Identify the source shape: Obsidian vault, markdown folder, notebook export, transcript dump, or mixed notes repo.
2. Separate canonical notes from raw ingest areas before indexing or packaging.
3. Inventory note conventions before indexing: frontmatter keys, aliases, tags, task syntax, attachments, private folders.
4. Extract note text and metadata separately.
5. Preserve note relationships: wikilinks, backlinks, tags, path hierarchy, and recent edits.
6. If external transcripts or exports feed the vault, route extracted summaries, decisions, and actions into stable note destinations instead of treating raw dumps as the memory layer.
7. Only then choose whether the downstream consumer needs direct Markdown bundles, a search index, or an MCP server.

Packaging Rules

Rule	Why it matters
A note ≠ a chunk — preserve whole-note metadata even when chunking	Path, title, aliases, timestamps are retrieval signals
Keep canonical notes and raw ingest artifacts separate	A transcript dump is not the operating note system
Prefer note titles that read like claims or decisions	Helps an agent triage relevance before opening the file
Use maps-of-content / home notes for navigation	Semantic search alone cannot recover the right abstraction level
Separate private, shared, and project-specific spaces before retrieval	Prevents accidental cross-space leakage
Default to local-first, read-only flows	Only add write-back when the user explicitly needs it
If notes are one corpus among many, route to ai-rag	This skill covers note-shape; ai-rag covers retrieval architecture

Judgment Calls

These are the calls a non-expert tends to get wrong. Full detail and the anti-pattern behind each is in references/retrieval-anti-patterns.md.

• Full-text/grep beats semantic search for most personal vaults. Notes are small, high-lexical-overlap, and often need exact-phrase or exact-title recall (a decision title, a project name, a specific date). Reach for embeddings only once keyword/tag/path search demonstrably under-recalls — typically past a few hundred notes, or when queries are conceptual ("what did I conclude about X") rather than lexical. Below that, a vector index adds latency and infra without improving grounding.
• Citation integrity is non-negotiable. Never let a downstream agent cite a note without a resolvable path + mtime. If a wikilink target does not resolve to an actual file, that is a stop-and-fix signal — flag it in the pack footer, do not silently drop it or silently invent a path.
• An orphan is a triage signal, not a verdict. See the vault-hygiene note in Scenario 3 above — orphan detection is topology-specific (wikilinks only) and produces false positives in folder- or tag-organized vaults.
• Status metadata beats recency for trust, not just freshness. A note edited yesterday but marked status: draft is less trustworthy than a status: active note from three months ago. Filter on status before recency, not instead of it.

When To Use This Skill

Use this skill when the user asks:

• "Search my Obsidian vault."
• "Prepare my notes for an LLM."
• "Package markdown notebooks into retrieval-ready context."
• "How should I structure note metadata, links, and summaries for AI?"
• "I want Claude to keep my Obsidian vault up to date from transcripts or exports."
• "I have NotebookLM-style exports or source packs and want a reusable local workflow."

Navigation

Scripts

• scripts/scan_vault.py - vault scanner: inventory, tags, orphans subcommands; JSON/CSV output
• scripts/build_context_pack.py - context pack builder: keyword/path selection, chunk strategies, char budget

References

• references/obsidian-and-local-vaults.md - vault conventions, metadata, and safe inventory rules
• references/context-packaging-patterns.md - chunk strategies, token budgets, ordering, deduplication, attribution
• references/retrieval-anti-patterns.md - 8 anti-patterns with fixes (blind chunking, ignoring topology, draft notes, etc.)
• references/ai-curated-wiki-pattern.md - AI-owned wiki variant: folder layout, invariants, compounding loop, scale heuristic, snapshots
• data/sources.json - official and community sources for note-vault retrieval patterns

Related Skills

• ../ai-rag/SKILL.md - retrieval architecture, ranking, and evaluation
• ../agents-mcp/SKILL.md - MCP wrappers for repeated note retrieval workflows
• ../agents-memory/SKILL.md - repo-native memory and long-lived agent context

Fact-Checking

• Start from data/sources.json for source-app behavior and community patterns.
• Verify current vendor export paths and app capabilities before citing volatile details.
• If web access is unavailable, mark product-specific claims as unverified.

Learnings Loop

Before applying this skill on a non-trivial task, read learnings.consolidated.md in this directory (and learnings.md if present).

After applying it, if you encountered a pattern worth remembering, a mistake worth preventing, or a domain fact that surprised you, append one dated bullet to learnings.md via agents-skills-feedback-loop/scripts/append_learning.py. Do not modify SKILL.md itself.