knowledge-base

name

knowledge-base

description

Create and maintain a structured, AI-oriented knowledge base. Supports four modes: first-time creation, incremental ingestion (from inbox, conversation, or URL), auto-reorganization, and health check (lint). Trigger when users mention: creating a knowledge base, organizing knowledge, digesting documents, processing inbox, adding knowledge, wiki management, organizing materials, ingesting articles, fetching content from URLs, updating the knowledge base, health check, lint, check knowledge base, etc. Even casual phrases like "help me organize this", "save this for later", "there's new stuff in inbox", "check my knowledge base for problems", "is my wiki healthy" should trigger this skill.

Knowledge Base Manager

Manage a structured knowledge base designed for AI consumption. The core idea: humans feed raw materials, AI digests and maintains everything.

Knowledge Base Structure

<kb-root>/
├── README.md            # For humans: purpose, structure, usage guide
├── _surface.md          # Root index (flat phase: lists every page; categorized phase: lists only categories)
├── _rules.md            # Operation rules (human-maintained, LLM read-only)
├── log.md               # Append-only operation log
├── inbox/               # Raw materials waiting to be digested
│   └── _processed/      # Originals moved here after digestion (never deleted)
└── wiki/                # All knowledge pages (maintained by LLM)
    ├── page_a.md        # Flat phase: pages sit directly under wiki/
    └── crypto/          # Categorized phase: pages grouped into subfolders
        ├── _index.md    # Sub-index: one-line index of every page in this category
        └── page_b.md

Core mechanism — hierarchical index:

The knowledge base has two phases, transitioning automatically as content grows:

• Flat phase (< 20 wiki files): _surface.md lists every page directly. LLM reads _surface.md → loads the target page.
• Categorized phase (after reorganization): _surface.md lists only category folders. Each subfolder has its own _index.md. LLM reads _surface.md → picks a category → reads that _index.md → loads the target page.

This keeps _surface.md short no matter how large the wiki grows.

---

Step 1: Detect Mode

First, confirm the knowledge base path. If the user hasn't specified one, ask: "Where should the knowledge base live?"

Once you have the path, detect the current mode:

Pre-check: If the user explicitly asks for a health check or lint (e.g. "check my wiki", "is my knowledge base healthy"), go directly to Mode 4: Lint regardless of inbox state or other signals.

Otherwise, follow this priority:

1. Path doesn't exist or directory is empty → First-time creation (fresh)
2. Path exists with files but no _surface.md → First-time creation (migrate existing docs)
3. Structure is complete + inbox/ has files → Incremental mode (digest from inbox)
4. Structure is complete + user provided content or URL in conversation → Incremental mode (from conversation/URL)

---

Mode 1: First-Time Creation

Fresh Knowledge Base

Execute in order:

1. Create directory structure (inbox/, wiki/)
2. Generate README.md (use template below, replace knowledge base name)
3. Generate _rules.md (use template below)
4. Create _surface.md with header line only
5. Create log.md with first entry
6. Tell the user: "Knowledge base created. Drop materials into inbox/ or tell me what to organize."

Migrate Existing Documents

The directory already has files but isn't in knowledge base format:

1. Create any missing structure files (_surface.md, _rules.md, README.md, log.md, inbox/, wiki/)
2. Read every existing file one by one — do not skip any
3. Classify each file:
   • Already organized knowledge (clear topic, structured) → move to wiki/, add index line to _surface.md
   • Raw material (unstructured notes, downloaded articles) → move to inbox/, then digest using incremental mode
   • Rules or documentation → extract useful content into _rules.md or README.md
4. After all files are processed, update log.md

---

Mode 2: Incremental Ingestion

Detect Source

Check in this priority order:

1. User gave a URL in conversation → Source C (fetch from URL)
2. User gave text content in conversation → Source B (conversation content)
3. Neither of the above → check if inbox/ has files → Source A (inbox digestion)
4. Inbox is also empty → tell user: "Nothing to process. Drop materials in inbox/ or give me content directly."

Source A: Digest from Inbox

1. List all files in inbox/
2. Process each file:
   • Read the full content
   • Extract core knowledge, remove redundancy
   • Check the current index to find related existing pages:
     • Flat phase: read _surface.md
     • Categorized phase: read _surface.md to identify the relevant category, then read that category's _index.md
   • Related page exists → update the existing wiki page with new information
   • No related page → create a new page:
     • Flat phase: in wiki/
     • Categorized phase: in the appropriate subfolder. If the content doesn't fit any existing category, create a new subfolder with its own _index.md and add a new category entry to _surface.md
   • Update the index:
     • Flat phase: update _surface.md
     • Categorized phase: update the subfolder's _index.md (add entry, update the category summary and last-updated line), and update the category line in _surface.md if the description needs adjusting
   • Bubble-up check: after updating a wiki page, re-read the parent index (_index.md or _surface.md) and verify its category summary still accurately reflects the content. If not, update it. Knowledge flows bottom-up: page changes propagate upward through the index hierarchy
   • Move the processed file to inbox/_processed/ (never delete originals — they serve as the source of truth if re-ingestion is needed)
   • Append a record to log.md

Source B: From Conversation

User provided text content directly in the conversation:

1. Understand the content, extract core knowledge
2. Check the index to find related existing pages (same flat/categorized logic as Source A)
3. Create new page or update existing page
4. Update the appropriate index (_surface.md or subfolder _index.md) and log.md. Run bubble-up check (same as Source A)

Source C: From URL

User provided a link:

1. Use WebFetch to retrieve the page content
2. Based on content length:
   • Short (< 3000 words) → save original to inbox/ as backup, extract knowledge into wiki/
   • Long (≥ 3000 words) → summarize, write only core content to wiki/, keep original URL in page frontmatter
   • Fetch failed → tell user "Can't access this link", ask them to paste the content manually
3. Move the saved original from inbox/ to inbox/_processed/ (same as Source A — keep the traceability chain)
4. Update the appropriate index (_surface.md or subfolder _index.md) and log.md. Run bubble-up check (same as Source A)

---

Mode 3: Auto-Reorganization

After every incremental operation, check the file count in wiki/:

File count	Action
< 20	Do nothing. Keep flat structure.
20 – 50	Suggest to user: "wiki/ now has X files. Want me to organize them into subfolders by topic?"
> 50	Strongly recommend: "wiki/ has X files — this is getting hard to navigate. I strongly suggest organizing by topic."

Reorganization steps (must get user consent first):

1. Read all wiki page titles and _surface.md content
2. Cluster by topic, propose 3–7 subfolder names
3. Show the plan to the user, e.g.:

   Proposed organization:
   - crypto/ (12 files): on-chain analysis, token research
   - tools/ (8 files): tool guides, API docs
   - research/ (6 files): paper summaries, study notes
   

4. After user confirms:
   • Create subfolders
   • Create _index.md in each new subfolder (migrate the relevant entries from _surface.md)
   • Move files into their subfolders
   • Rewrite _surface.md: replace individual page entries with category-level entries pointing to each subfolder's _index.md
   • Update cross-references ([[]] links) in wiki pages to reflect new paths
   • Record the reorganization in log.md

Category splitting: If any single subfolder exceeds 30 pages, suggest splitting it into sub-categories (e.g. crypto/ → crypto/defi/, crypto/mev/). The same reorganization rules apply: show the plan, get user consent, then execute. Each new sub-category gets its own _index.md, and the parent _index.md is rewritten to list sub-categories instead of individual pages.

Important: Any operation that moves, renames, or restructures files requires showing the plan and getting explicit user approval before execution.

---

Mode 4: Lint (Health Check)

When the user asks to check the knowledge base health:

Pre-check: First verify the knowledge base structure exists at the given path. If the path doesn't exist, has no _surface.md, or is missing wiki/, tell the user: "No valid knowledge base found at this path. Want me to create one?" Do not proceed with lint checks on a non-existent or incomplete structure.

If the structure is valid, run all checks below and present a report.

Check 1: Orphan Pages

Wiki pages that exist as files but have no entry in any index (_surface.md or _index.md).

• Scan all .md files under wiki/ (excluding _index.md files)
• For each file, verify it appears in _surface.md (flat phase) or the parent folder's _index.md (categorized phase)
• Report any unindexed pages

Check 2: Broken References

Index entries or [[]] cross-references that point to pages that don't exist.

• Read _surface.md and all _index.md files, check every link target exists as a file
• Read all wiki pages, check every [[page-name]] reference resolves to an existing file
• Report any broken links

Check 3: Duplicate / Overlapping Pages

Multiple pages covering the same topic, potentially with conflicting information.

• Read _surface.md (and _index.md files if categorized) to get all page titles and descriptions
• Identify pages with very similar titles or descriptions
• For suspected duplicates, read the first 2–3 paragraphs of each page to confirm whether they genuinely overlap (title similarity alone is not reliable — e.g. "DeFi Basics" and "DeFi Introduction" might be the same, or might cover different aspects)
• Report confirmed overlapping pairs, noting any contradictions between them

Check 4: Stale Pages

Pages with outdated content based on their updated frontmatter date.

• Read frontmatter of all wiki pages
• Flag pages where updated is more than 90 days ago
• Report with the date, so the user can decide what to refresh

Check 5: Index Consistency

Mismatches between the index and the actual file structure.

• In categorized phase: verify every subfolder in wiki/ has an _index.md
• Verify page counts in _surface.md category entries match actual file counts
• Check for empty subfolders

Presenting the Report

After running all checks, present results to the user in this format:

Knowledge Base Health Report
=============================

Orphan Pages (not in any index):
- wiki/forgotten_note.md

Broken References:
- _surface.md → wiki/deleted_page.md (file missing)
- wiki/solana_mev.md → [[old_page]] (target missing)

Potential Duplicates:
- wiki/defi_basics.md ↔ wiki/defi_intro.md (overlapping content about DeFi fundamentals)

Stale Pages (not updated in 90+ days):
- wiki/old_research.md (last updated: 2025-12-01)

Index Issues:
- wiki/crypto/ has no _index.md
- _surface.md says crypto/ has 15 pages, actually has 18

✅ No issues: [list any checks that passed clean]

After the report, ask the user: "Want me to fix any of these issues?" Then only act on what the user approves. Fixes may include:

• Adding orphan pages to the index
• Removing broken references or creating the missing pages
• Merging duplicate pages (show a merged draft first)
• Updating the updated date after reviewing stale pages
• Fixing index counts and adding missing _index.md files

Record the lint operation in log.md.

---

File Format Specifications

Wiki Page Format

---
source: <source description or URL>
source_file: <path to original file in inbox/_processed/, if applicable>
created: <YYYY-MM-DD>
updated: <YYYY-MM-DD>
---

# Page Title

Body content...

## Related Pages
- [[related-page-name]]

The source_file field creates a traceability chain back to the original material, enabling re-ingestion if the digest was incomplete or incorrect.

_surface.md Format

Flat phase (pages listed directly):

# Surface Index

- **Title A** — one-line description → [details](wiki/filename_a.md)
- **Title B** — one-line description → [details](wiki/filename_b.md)

Categorized phase (only categories listed):

# Surface Index

- **crypto/** (15 pages) — on-chain analysis, tokens, DeFi → [sub-index](wiki/crypto/_index.md)
- **tools/** (8 pages) — tool guides, API docs → [sub-index](wiki/tools/_index.md)
- **research/** (6 pages) — paper summaries, study notes → [sub-index](wiki/research/_index.md)

_index.md Format (subfolder sub-index)

Each subfolder in categorized phase has an _index.md. It includes a short category summary (1–2 sentences giving the LLM enough context to judge relevance without reading individual pages) and a recent-updates line:

# crypto/

> Covers on-chain analysis, DeFi mechanics, and MEV strategies. Key trend: MEV revenue is being compressed by competition.

- **Solana MEV** — MEV extraction strategies on Solana → [details](solana_mev.md)
- **Uniswap V3** — concentrated liquidity mechanics → [details](uniswap_v3.md)

**Last updated**: [2026-04-13] Added Solana MEV page

Paths in _index.md are relative to the subfolder. Update the summary and last-updated line each time a page in this category is created or modified.

log.md Format

## [2026-04-13] ingest | Digested an article about XX
- Created: wiki/xx.md
- Updated: _surface.md

## [2026-04-13] reorganize | Organized wiki files into subfolders
- Created subfolders: crypto/, tools/
- Moved 15 files

---

Template: README.md

Use when creating a fresh knowledge base (replace bracketed content):

# [Knowledge Base Name]

## What is this
A structured knowledge base maintained by AI.

## Structure
| File/Directory | Purpose |
|----------------|---------|
| _surface.md | AI index — one-line summary of every entry |
| _rules.md | Operation rules, editable by humans |
| log.md | Full operation history |
| inbox/ | Drop raw materials here for AI to digest |
| wiki/ | Knowledge pages organized by AI |

## How to use
1. Drop materials into `inbox/`, then tell AI "process inbox"
2. Or give AI content/links directly in conversation
3. AI will digest, classify, and update the index automatically

Template: _rules.md

Use when creating a fresh knowledge base:

# Operation Rules

## Ingest
1. Read the full source material
2. Extract core knowledge, remove redundancy
3. Create or update wiki pages
4. Update _surface.md index
5. Append log.md record

## Query
1. Read _surface.md to find relevant entries
2. If categorized: read the relevant subfolder's _index.md
3. Load only the needed wiki pages
4. Synthesize information and answer

## Page Standards
- Each page focuses on one topic
- Use H1 for the title
- Must have frontmatter (source, created, updated)
- Cross-reference related pages with [[]]
- Every new page must be indexed (in _surface.md during flat phase, or in the subfolder's _index.md during categorized phase)