sb-ingest

name	sb-ingest
description	Ingest a new source document into the LLM Wiki. Use when asked to "ingest", "add source", "process document", "import article", "file source". Follows interactive interview workflow.

LLM Wiki Ingest

Process a new source document and integrate it into the wiki.

Prerequisites

1. Read AGENTS.md first to understand conventions

Step 1: Check for New Sources

Before asking user, run the status check:

uv run scripts/list_raw.py --wiki-root .

This shows which raw files are already ingested vs. new. Present the list:

• New files: ready to ingest
• Ingested files: already processed

Ask user: "Which file would you like to ingest?" or proceed with first new file.

Step 2: Identify Source

Confirm with user:

• Source file path (e.g., raw/articles/attention-is-all-you-need.md)
• Source category (articles, papers, podcasts, books, notes)

If no source specified, use the first new file found.

Step 2: Read Source

Read the entire source file. Extract:

• Title (from frontmatter or first heading)
• All entities mentioned (people, places, organizations)
• All concepts/ideas discussed
• Key claims and arguments

Step 3: Check Existing Wiki

Scan wiki/ directory for existing pages:

• List all current page slugs (from filenames)
• Read relevant pages to detect potential contradictions

Step 4: Interactive Interview

Present the following form to user. LLM auto-fills Q1-Q3 from source + wiki scan. Q4 is suggested. Q5 requires user input.

## Ingest Interview: [source-filename]

### 1. Key Entities (auto-extracted)
The following entities were found. Add any missing:
[entity list]

### 2. Core Concepts (auto-extracted)
The following concepts were identified. Add any missing:
[concept list]

### 3. Contradictions (auto-detected)
Potential conflicts with existing wiki:
[contradictions or "None detected"]

### 4. Emphasis (suggested)
What should we highlight vs. gloss over?
[suggested based on source content]

### 5. Personal Notes (your input)
Your observations, reactions, gaps you noticed?
[user fills in]

Wait for user to complete the form before proceeding.

Step 5: Present Plan

After receiving user input, show the plan:

## Ingest Plan

### Source Summary Page
- File: `wiki/sources/[slug].md`
- Sections: Summary, Key Points, Source Details

### Entity Pages to Create/Update
- `wiki/entities/[slug].md` (create/update for each)

### Concept Pages to Create/Update
- `wiki/concepts/[slug].md` (create/update for each)

### Synthesis Notes
- From: [user's personal notes + emphasis]
- File: TBD based on content

### Cross-References to Update
- Pages linking to new entities/concepts

---
Proceed? (yes/adjust/cancel)

Wait for user approval.

Step 6: Create Pages (Breadth-First)

Create ALL pages before writing content to any:

1. Create stub pages for all entities
2. Create stub pages for all concepts
3. Create source summary page
4. Create synthesis note (if applicable)
5. Only then: fill content into all stubs

This ensures all wikilinks can resolve.

Step 7: Run Post-Edit Script

After all pages created:

uv run scripts/ingest_post.py --source [source-slug] --pages [comma-separated-slugs]

Script will:

• Validate all wikilinks in new pages
• Create stubs for any missing target pages (bootstrap mode) OR reject broken links (strict mode)
• Normalize link casing to kebab-case
• Update wiki/index.md
• Append to wiki/log.md

Step 8: Report

Report to user:

• Pages created/updated
• Stubs auto-generated (if any)
• Links normalized (if any)
• Log entry created

Wikilink Rules

• Always use kebab-case: [[neural-networks]], NOT [[Neural Networks]]
• Always link to known pages: if creating a link, the target page must exist
• Use explicit links: [[slug]] not [[slug|Display Text]] unless necessary
• No invented slugs: if referencing a concept not yet in wiki, create the page first

Exit Criteria

• All pages created with proper frontmatter and sections
• All wikilinks valid (checked by post-edit script)
• wiki/index.md updated
• wiki/log.md appended
• User can view results in Obsidian or file browser

Error Handling

If post-edit script reports broken links:

1. LLM must fix them before completing
2. Re-run post-edit script
3. Report final status

If source file not found:

• Ask user for correct path
• Do not proceed without valid source