kb-ingest

name	kb-ingest
description	Ingest a new source into the knowledge base — processes a markdown file or URL into wiki pages with summaries, entity/concept updates, and cross-references. Use when the user wants to add, ingest, import, or process a source, article, URL, paper, or document into their knowledge base. Also triggers on "kb-ingest" or "/kb-ingest".

kb-ingest

Ingest a new source into the knowledge base. Accepts a local file path, a URL, or no argument (lists unprocessed sources).

Arguments

• <path> — Local markdown file to ingest.
• <url> — URL to fetch and ingest.
• (no argument) — List unprocessed raw sources and let the user pick.

Step 1: Resolve KB

Run the resolver in mutating mode:

python3 ~/.config/kb/scripts/kb_resolve.py --mode mutating

Capture the returned KB root path. If resolution fails, abort with the error message.

Step 2: Acquire Source

If URL provided:

1. Fetch the content using WebFetch.
2. Generate a filename: raw/<slug>-<YYYY-MM-DD>-<8-char-hash>.md
   • <slug>: lowercase, hyphenated version of the page title (max 60 chars).
   • <hash>: first 8 characters of SHA-256 of the URL.
3. Check if that file already exists. If so, abort: "Source already ingested. Use kb-reingest to update."
4. Write the fetched content to the raw file. Prepend a metadata header:

---
url: <original-url>
fetched: <YYYY-MM-DD>
title: "<page title>"
---

<fetched content>

If file path provided:

1. Read the file.
2. Copy it to raw/ with a collision-resistant name: raw/<slug>-<YYYY-MM-DD>-<8-char-hash>.md
   • <hash>: first 8 characters of SHA-256 of the file content.
3. Check if that file already exists. If so, abort: "Source already ingested."

If no argument:

1. Run the unprocessed scanner:

python3 ~/.config/kb/scripts/kb_unprocessed.py "{{KB_ROOT}}"

2. Present the list to the user. Let them pick one. Set that as the source file.

Step 3: Read and Analyze Source

1. Read the full raw source file.
2. Identify key takeaways (2-3 bullets). Present them to the user for confirmation before proceeding.

Step 4: Contradiction Check

Run the related-content finder against the source content:

python3 ~/.config/kb/scripts/kb_find_related.py "{{KB_ROOT}}" --query "{{SUMMARY_OF_SOURCE}}" --top-k 10

Read the top related pages. Check for factual contradictions or conflicting recommendations against the new source. If contradictions are found:

• Warn the user with specific page pairs and conflicting claims.
• Ask whether to proceed (contradictions will be recorded in frontmatter).

Step 5: Create/Update Wiki Pages

Summary page

Write wiki/summaries/<slug>.md with full frontmatter:

---
title: "Summary: {{SOURCE_TITLE}}"
type: summary
source_ids:
  - raw/{{RAW_FILENAME}}
confidence: 0.7
stability: evolving
human_reviewed: false
created: {{DATE}}
last_verified: {{DATE}}
supersedes: []
contradictions: []    # populated if Step 4 found issues
tags: []              # extracted from content
---

Include: key points, important quotes, takeaways. Use [[wikilinks]] to link to entity and concept pages.

Entity pages

For each entity (person, org, tool, project) identified in the source:

• If a wiki page already exists in wiki/entities/: update it.
  • Add the new source_ids entry.
  • Merge new information (never remove existing content).
  • Update last_verified.
  • Skip if human_reviewed: true — log a warning instead.
• If no page exists: create wiki/entities/<entity-slug>.md with full frontmatter (type: entity, confidence: 0.6).

Concept pages

For each concept (idea, pattern, framework) identified:

• Same create-or-update logic as entities.
• Use type: concept, initial confidence: 0.6.
• Link to related entities via [[wikilinks]].

Step 6: Regenerate Index

python3 ~/.config/kb/scripts/kb_index.py "{{KB_ROOT}}"

Step 7: Append Log

Add a row to log.md:

| {{DATE}} | ingest | Ingested {{RAW_FILENAME}} → {{N}} wiki pages created/updated |

Step 8: Update Registry

python3 ~/.config/kb/scripts/kb_registry.py update-access --name "{{KB_NAME}}"

Final Output

Print:

• Source file path
• Pages created (list with paths)
• Pages updated (list with paths)
• Contradictions found (if any)
• Human-reviewed pages skipped (if any)
• Suggestion: "Run kb-lint to verify KB health"