vault-ingest

name	vault-ingest
version	0.2.0
description	Ingest sources into the Obsidian wiki vault. Reads a source, extracts entities and concepts, creates or updates wiki pages, cross-references, and logs the operation. Supports files, URLs, and batch mode. Triggers on: ingest, process this source, add this to the wiki, read and file this, batch ingest, ingest all of these, ingest this url.
type	skill
targets	["claude-code"]
category	{"primary":"context-management"}

wiki-ingest: Source Ingestion

Read the source. Write the wiki. Cross-reference everything. A single source typically touches 8-15 wiki pages.

Syntax standard: Write all Obsidian Markdown using proper Obsidian Flavored Markdown. Wikilinks as [[Note Name]], callouts as > [!type] Title, embeds as ![[file]], properties as YAML frontmatter. If the kepano/obsidian-skills plugin is installed, prefer its canonical Obsidian Markdown reference for syntax guidance. Otherwise, follow the guidance in this skill.

---

Delta Tracking

Before ingesting any file, check raw/.manifest.json to avoid re-processing unchanged sources.

# Check if manifest exists
[ -f raw/.manifest.json ] && echo "exists" || echo "no manifest yet"

Manifest format (create if missing):

{
  "sources": {
    "raw/articles/article-slug-2026-04-08.md": {
      "hash": "abc123",
      "ingested_at": "2026-04-08",
      "pages_created": ["wiki/sources/article-slug.md", "wiki/entities/Person.md"],
      "pages_updated": ["wiki/index.md"]
    }
  }
}

Before ingesting a file:

1. Compute a hash: md5sum [file] | cut -d' ' -f1 (or sha256sum on Linux).
2. Check if the path exists in .manifest.json with the same hash.
3. If hash matches, skip. Report: "Already ingested (unchanged). Use force to re-ingest."
4. If missing or hash differs, proceed with ingest.

After ingesting a file:

1. Record {hash, ingested_at, pages_created, pages_updated} in .manifest.json.
2. Write the updated manifest back.

Skip delta checking if the user says "force ingest" or "re-ingest".

---

URL Ingestion

Trigger: user passes a URL starting with https://.

Steps:

1. Fetch the page using WebFetch.
2. Clean (optional): if defuddle is available (which defuddle 2>/dev/null), run defuddle [url] to strip ads, nav, and clutter. Typically saves 40-60% tokens. Fall back to raw WebFetch output if not installed.
3. Derive slug from the URL path (last segment, lowercased, spaces→hyphens, strip query strings).
4. Save to raw/articles/[slug]-[YYYY-MM-DD].md with a frontmatter header:

   ---
   source_url: [url]
   fetched: [YYYY-MM-DD]
   ---
   

5. Proceed with Single Source Ingest starting at step 2 (file is now in raw/).

---

Image / Vision Ingestion

Trigger: user passes an image file path (.png, .jpg, .jpeg, .gif, .webp, .svg, .avif).

Steps:

1. Read the image file using the Read tool. Claude can process images natively.
2. Describe the image contents: extract all text (OCR), identify key concepts, entities, diagrams, and data visible in the image.
3. Save the description to raw/images/[slug]-[YYYY-MM-DD].md:

   ---
   source_type: image
   original_file: [original path]
   fetched: YYYY-MM-DD
   ---
   # Image: [slug]
   
   [Full description of image contents, transcribed text, entities visible, etc.]
   

4. Copy the image to _attachments/images/[slug].[ext] if it's not already in the vault.
5. Proceed with Single Source Ingest on the saved description file.

Use cases: whiteboard photos, screenshots, diagrams, infographics, document scans.

---

Single Source Ingest

Trigger: user drops a file into raw/ or pastes content.

Steps:

1. Read the source completely. Do not skim.
2. Discuss key takeaways with the user. Ask: "What should I emphasize? How granular?" Skip this if the user says "just ingest it."
3. Create source summary in wiki/sources/. Use the source frontmatter schema from references/frontmatter.md.
4. Create or update entity pages for every person, org, product, and repo mentioned. One page per entity.
5. Create or update concept pages for significant ideas and frameworks.
6. Update relevant domain page(s) and their _index.md sub-indexes.
7. Update wiki/overview.md if the big picture changed.
8. Update wiki/index.md. Add entries for all new pages.
9. Update wiki/hot.md with this ingest's context.
10. Append to wiki/log.md (new entries at the TOP):

    ## [YYYY-MM-DD] ingest | Source Title
    - Source: `raw/articles/filename.md`
    - Summary: [[Source Title]]
    - Pages created: [[Page 1]], [[Page 2]]
    - Pages updated: [[Page 3]], [[Page 4]]
    - Key insight: One sentence on what is new.
    

11. Check for contradictions. If new info conflicts with existing pages, add > [!contradiction] callouts on both pages.
12. Move the original out of the inbox. After the wiki pages are written, move the source file from raw/ to _sources/<category>/ (choose the category subfolder by kind — articles/, research/, specs/, notes/, ...), or to _archive/ if it is large (>100 KB) or low-value. raw/ is an inbox and should end empty once everything is processed.
13. Sync the qkb index. Run qkb update so BM25, graph link extraction, and the question-cache used by vault-query-qkb pick up everything you just wrote. See Index Sync below.

---

Batch Ingest

Trigger: user drops multiple files or says "ingest all of these."

Steps:

1. List all files to process. Confirm with user before starting.
2. Process each source following the single ingest flow. Defer cross-referencing between sources until step 3.
3. After all sources: do a cross-reference pass. Look for connections between the newly ingested sources.
4. Update index, hot cache, and log once at the end (not per-source).
5. Report: "Processed N sources. Created X pages, updated Y pages. Here are the key connections I found."
6. Sync the qkb index once at the end (not per-source). See Index Sync below.

Batch ingest is less interactive. For 30+ sources, expect significant processing time. Check in with the user after every 10 sources.

---

Context Window Discipline

Token budget matters. Follow these rules during ingest:

• Read wiki/hot.md first. If it contains the relevant context, don't re-read full pages.
• Read wiki/index.md to find existing pages before creating new ones.
• Read only 3-5 existing pages per ingest. If you need 10+, you are reading too broadly.
• Use PATCH for surgical edits. Never re-read an entire file just to update one field.
• Keep wiki pages short. 100-300 lines max. If a page grows beyond 300 lines, split it.
• Use search (/search/simple/) to find specific content without reading full pages.

---

Contradictions

[!note] Custom callout dependency The [!contradiction] callout type used below is a custom callout defined in .obsidian/snippets/vault-colors.css (auto-installed by /wiki scaffold). It renders with reddish-brown styling and an alert-triangle icon when the snippet is enabled. If the snippet is missing, Obsidian falls back to default callout styling, so the page still works without the visual flourish. See [[skills/wiki/references/css-snippets.md]] for the four custom callouts (contradiction, gap, key-insight, stale).

When new info contradicts an existing wiki page:

On the existing page, add:

> [!contradiction] Conflict with [[New Source]]
> [[Existing Page]] claims X. [[New Source]] says Y.
> Needs resolution. Check dates, context, and primary sources.

On the new source summary, reference it:

> [!contradiction] Contradicts [[Existing Page]]
> This source says Y, but existing wiki says X. See [[Existing Page]] for details.

Do not silently overwrite old claims. Flag and let the user decide.

---

Naming Conventions

Obsidian resolves wikilinks by matching the exact filename (without extension). Mismatched names create broken links.

Page Type	Filename Convention	Example	Wikilink
Entity	Title Case with spaces	Universal Weather.md	[[Universal Weather]]
Concept	Title Case with spaces	Missing Middle.md	[[Missing Middle]]
Domain	Title Case with spaces	Competitive Landscape.md	[[Competitive Landscape]]
Comparison	Title Case with spaces	Us vs Universal Weather.md	[[Us vs Universal Weather]]
Question	Title Case with spaces	Research NOTAM Reform.md	[[Research NOTAM Reform]]
Source	kebab-case (slug)	competitor-research-deep-dive.md	[[competitor-research-deep-dive|Display Name]]
Meta	lowercase	index.md, log.md, hot.md	[[index]], [[log]], [[hot]]

Why the split? Entities, concepts, domains, comparisons, and questions are referenced frequently with natural-language wikilinks ([[Missing Middle]]). The filename must match exactly. Source summaries are referenced less frequently and use pipe syntax for display names ([[some-slug|Readable Title]]), so kebab-case slugs are fine.

The rule: If the wikilink text is the page name people type, the filename must match it exactly. If the wikilink always uses pipe syntax, kebab-case is acceptable.

---

Index Sync

After every ingest (single or batch), the qkb retrieval index needs to know about the new/changed pages. Run this once at the end of the ingest:

qkb update

What this does:

• Walks the vault and diffs file content hashes against the index — only changed/new files get re-processed.
• Rebuilds BM25 entries for changed docs so lexical search picks them up.
• Auto-runs graph link internally — extracts [[wikilinks]], ![[embeds]], and frontmatter types into the typed graph (LINKS_TO / EMBEDS / REFERENCES edges). This is what the --graph retrieval mode in vault-query-qkb relies on.
• Idempotent: re-running with no vault changes is a near-instant no-op.

Latency: typically 1–5 seconds for a handful of new pages, 30–60 seconds for a batch of 50+ new files. Print the line to the user so they can confirm freshness.

Vector embeddings (the slow step) — NOT automatic

qkb update only refreshes lexical + graph state. New vector embeddings for the new docs are a separate, slower step (uses local LLM, 5–15 minutes for 100+ docs):

qkb embed

Do NOT run embed automatically inside the ingest skill. Two reasons:

1. Lexical + graph alone is usually enough. The reranker in qkb query scores against fresh chunk text from the FTS5 index regardless of vector freshness, so newly-ingested docs still surface correctly on most queries. They're slightly weaker on pure-semantic matches until embeddings catch up.
2. It's heavy. A user doing 20 small ingests across a session shouldn't trigger 20 model-load + embed cycles.

Instead, after the ingest report, suggest to the user:

"Lexical + graph synced. Run qkb embed when you want the new pages' vectors caught up (5-15 min, optional)."

When qkb update fails

Don't block the ingest report on this. If qkb update returns non-zero:

• Log the error briefly: "qkb index sync failed: . Ingest itself succeeded; retry with qkb update once resolved."
• Continue with the rest of the ingest output.

Common causes: qkb not on PATH (user hasn't installed @agent-ops/qkb globally), GraphQLite extension missing, or a transient SQLite lock from a running MCP daemon.

---

What Not to Do

• raw/ is an inbox: after processing, move the original to _sources/<category>/ (or _archive/), leaving raw/ empty. Do not treat raw/ as permanent storage.
• Do not modify the archived originals in _sources/ or _archive/. Those are the immutable source documents.
• Do not create duplicate pages. Always check the index and search before creating.
• Do not skip the log entry. Every ingest must be recorded.
• Do not skip the hot cache update. It is what keeps future sessions fast.
• Do not skip the qkb index sync. Future vault-query-qkb calls will miss what you just wrote.
• Do not use kebab-case filenames for entities, concepts, domains, comparisons, or questions. These must match wikilink text exactly (Title Case with spaces).
• Do not run qkb embed automatically inside an ingest. Suggest it; let the user trigger.