obsidian-kb

name	obsidian-kb
description	Use this skill when the user wants to build a knowledge base for their project, create an Obsidian wiki from project data, analyze project structure for knowledge management, or mentions "obsidian-kb", "knowledge base", or "compile wiki".

/obsidian-kb — Build a Project Knowledge Base

Analyze any project's structure and build a custom knowledge base: tiered context packs for LLM consumption + an Obsidian wiki for human browsing. Inspired by Karpathy's LLM Knowledge Base workflow, adapted for LLM-first consumption.

Usage

/obsidian-kb init                # Analyze project, propose architecture
/obsidian-kb build               # Generate compiler, commands, run initial compile
/obsidian-kb rebuild             # Re-run init + build (for when project structure changes)

---

Phase 1: init — Analyze & Design

Step 1: Scan the Project

Scan the project directory tree. Respect .gitignore if it exists. Collect:

1. Data files by format:

   • .jsonl — JSON Lines (line-delimited records)
   • .json — JSON documents or arrays
   • .yaml / .yml — YAML documents
   • .csv / .tsv — Tabular data
   • .toml — Structured configuration
   • .md with YAML frontmatter — Structured markdown

2. Project signals — detect what kind of project this is:

   • package.json or tsconfig.json → JavaScript/TypeScript software project
   • setup.py or pyproject.toml or requirements.txt → Python software project
   • Cargo.toml → Rust project
   • go.mod → Go project
   • .bib or papers/ or references/ → Research/academic project
   • JSONL files with name + description + domain fields → Knowledge graph project
   • clients/ or projects/ directories → Consulting/portfolio project
   • docs/ or documentation/ → Documentation-heavy project
   • data/ with structured files → Data-centric project

3. Existing compilation — check if data/compiled/ or data/kb-architecture.yaml already exist. If so, note them and ask if the user wants to rebuild or update.

Display: "Scanned [N] directories, found [M] data files across [K] formats. Project type: [detected type]."

Step 2: Sample and Classify Knowledge Atoms

For each data file found:

1. Read 3-5 sample records (first 3-5 lines for JSONL, first 3-5 entries for JSON arrays, etc.)
2. Identify what each record represents — its knowledge atom type:
   • What is the primary identifier field? (name, title, id, path, etc.)
   • What is the description/content field?
   • What fields create relationships to other records? (references, links, tags, categories)
   • What grouping dimension exists? (domain, category, module, type, directory)

For software projects without structured data files, the knowledge atoms come from the code structure:

• Components/modules (from src/ directory structure)
• API endpoints (from route definitions)
• Architecture decisions (from docs/adr/ or decisions/ directories)
• Configuration (from config files)
• Dependencies (from package manifests)

For markdown-heavy projects, extract frontmatter fields as the schema.

Display a table:

Source: data/knowledge-graph/frameworks.jsonl
  Format: JSONL (22 records)
  Atom type: framework
  Key field: name
  Description: description
  Group field: domain
  Relationship fields: entangled_with, related_frameworks
  Sample: "Knowledge Gradient Model" (intelligence_systems)

Source: data/reading-list/index.jsonl
  Format: JSONL (50 records)
  Atom type: reading
  Key field: title
  Description: core_arguments
  Group field: domain
  Relationship fields: framework_connections
  Sample: "Thinking in Systems" (intelligence_systems)

Step 3: Map Relationships

For each pair of knowledge atom types, identify:

1. Explicit references: Field A in type X contains names/IDs from type Y
   • Example: concept.related_frameworks → framework.name
2. Shared groupings: Both types share a grouping field with the same values
   • Example: framework.domain = concept.domain
3. Hierarchical containment: Type X is a child/component of type Y
   • Example: API endpoint belongs to a module

Build a relationship map:

framework <-> concept (via related_frameworks)
framework <-> reading (via framework_connections)
framework <-> framework (via entangled_with)
concept -- framework (shared domain)
signal -> framework (text mention)

Step 4: Design Tier Structure

Based on the atoms and relationships, design a three-tier compiled context pack structure:

Tier 0 — Index (always loaded, ~1000-1500 tokens)

• One-line entry per knowledge atom across ALL types
• Purpose: routing. "What exists in this knowledge base?"
• Output: data/compiled/index.md

Tier 1 — Group Context (loaded per operation, ~3000-6000 tokens each)

• One file per group value (domain, category, module, directory)
• Contains: all atoms in that group with full descriptions, cross-references, relationship summaries
• Purpose: working context for a specific area
• Output: data/compiled/[group_dimension]/[group_value].md
• If no natural grouping exists: use a single "overview" file per atom type

Tier 2 — Entity Deep Context (loaded on demand, ~1000-2000 tokens each)

• One file per entity in the richest atom type (the type with most relationships)
• Contains: the entity + summaries of all connected entities + history + related sources
• Purpose: deep work on a specific item
• Output: data/compiled/[atom_type]/[slug].md
• Only generate Tier 2 for atom types with rich interconnections. Simple lists don't need deep context pages.

Design the wiki structure in parallel:

• One wiki section per atom type
• Wikilinks between related atoms: [[section/slug|Display Name]]
• Frontmatter tags for filtering in Obsidian
• Graph color groups: one color per section

Step 5: Propose Architecture

Present the full architecture to the user:

=== Knowledge Base Architecture Proposal ===

Project: [project name from directory or package.json]
Type: [detected type]

Sources:
  [list of source files with atom types and record counts]

Tier Structure:
  Tier 0 (Index): [what it contains]
  Tier 1 ([group dimension]): [N groups] — [group names]
  Tier 2 ([entity type]): [N entities]

Wiki Structure:
  wiki/
    [section 1]/ — [atom type] ([count] pages)
    [section 2]/ — [atom type] ([count] pages)
    ...
    index.md — Map of Content

Estimated pages: [total]
Estimated compile time: fast (< 5 seconds)

Proceed with build?

Wait for user approval. If the user suggests changes, modify the architecture. Once approved, save to data/kb-architecture.yaml:

project_name: "[name]"
project_type: "[type]"
analyzed_at: "[ISO timestamp]"
sources:
  - path: "[relative path]"
    format: "[jsonl|json|yaml|csv|md|toml]"
    atom_type: "[type name]"
    count: [N]
    key_field: "[field name]"
    description_field: "[field name]"
    group_field: "[field name or null]"
    relationship_fields: ["[field1]", "[field2]"]
tiers:
  tier0:
    output: "data/compiled/index.md"
    token_budget: 1500
  tier1:
    group_dimension: "[field name]"
    groups: ["[group1]", "[group2]"]
    output_pattern: "data/compiled/[dimension]/{group}.md"
    token_budget_per_group: 5000
  tier2:
    atom_type: "[primary type]"
    output_pattern: "data/compiled/[type]/{slug}.md"
    token_budget_per_entity: 1500
wiki:
  root: "wiki/"
  sections:
    - name: "[section name]"
      atom_type: "[type]"
      color: "[hex color]"

---

Phase 2: build — Generate & Compile

Step 1: Read Architecture Spec

Read data/kb-architecture.yaml. If it doesn't exist, run Phase 1 first.

Step 2: Generate the Compiler

Write scripts/compile-kb.py — a Python script tailored to this project. The compiler must:

1. Load functions for each source format:

   • load_jsonl(path) — read line-delimited JSON
   • load_json(path) — read JSON file (handle both objects and arrays)
   • load_yaml(path) — read YAML (requires import yaml or pip install pyyaml)
   • load_csv(path) — read CSV with csv.DictReader
   • load_markdown(path) — extract YAML frontmatter from markdown files
   • Only include loaders for formats actually used in this project

2. Compile Tier 0: Read all sources, extract key + one-line description per atom, write data/compiled/index.md. Include summary stats (counts per type, group distribution).

3. Compile Tier 1: For each group value, collect all atoms in that group across all types. Write full descriptions, cross-references, and relationship summaries. Respect token budget by truncating descriptions proportionally if the group is too large.

4. Compile Tier 2: For each entity in the primary atom type:

   • Write the entity's full data
   • Look up and include summaries of connected entities (from relationship fields)
   • Include any other atom types that reference this entity
   • Include history/evolution if available

5. Wiki compilation (behind --human flag):

   • One page per atom with YAML frontmatter and wikilinks
   • Group overview pages
   • Index page (Map of Content)
   • .obsidian/ config with graph coloring from architecture spec

6. Freshness tracking: Write data/compiled/freshness.json with compiled_at timestamp and source file modification times.

7. Idempotent: Always fully regenerates data/compiled/. Never appends.

The compiler should be:

• Self-contained (no external dependencies beyond Python stdlib, unless YAML sources require PyYAML)
• Runnable via python3 scripts/compile-kb.py (context packs) or python3 scripts/compile-kb.py --human (also wiki)
• Fast (< 5 seconds for typical projects)

Reference pattern: Use the BrunoTwin compiler at scripts/compile-kb.py in that project as the structural reference — same load_jsonl, slugify, tier functions, freshness.json pattern. But write the code fresh for this project's specific sources and schema.

Step 3: Generate Obsidian Config

Create wiki/.obsidian/ with:

app.json:

{
  "showLineNumber": true,
  "strictLineBreaks": false,
  "readableLineLength": true,
  "showFrontmatter": false
}

graph.json: Generate color groups from the architecture spec's wiki sections. Each section gets a distinct color. Use the "path:[section]" query pattern.

Step 4: Generate /kb Command

Write .claude/commands/kb.md tailored to this project. It should support:

/kb                              # Show stats and health
/kb compile                      # Run python3 scripts/compile-kb.py
/kb search [query]               # Grep across all compiled pages
/kb [atom_type] [name]           # Load Tier 2 context for a specific entity
/kb gaps                         # Find orphaned atoms and missing relationships
/kb recent                       # Show recent changes to source data

The command instructions should tell Claude to:

1. Check data/compiled/freshness.json for staleness
2. Load from data/compiled/ (not raw source files) for queries
3. Fall back to raw source files for detail beyond what compiled packs contain
4. Recompile when sources have changed

Step 5: Generate /lint-kb Command

Write .claude/commands/lint-kb.md with project-specific health checks:

Universal checks (work for any project):

1. Broken references — relationship fields pointing to atoms that don't exist
2. Orphaned atoms — atoms with no incoming or outgoing relationships
3. Missing bidirectional links — if A references B, does B reference A?
4. Group coverage — each group should have minimum 3 atoms
5. Staleness — compiled output older than source data
6. Empty fields — atoms missing descriptions or key fields

Scoring: health = 100 - (critical * 10) - (warnings * 2)

The lint command should:

1. Read all raw source files (JSONL/JSON/YAML/CSV)
2. Run all checks
3. Report findings as CRITICAL / WARNING / INFO
4. Support --fix for mechanical fixes (backlinks, staleness)
5. Support --suggest for LLM-generated suggestions (missing connections, new atom candidates)

Step 6: Run Initial Compile

Execute python3 scripts/compile-kb.py --human to generate both context packs and wiki.

Report:

=== Knowledge Base Built ===

Context Packs (data/compiled/):
  Tier 0: 1 file (~[N] tokens)
  Tier 1: [M] files (~[K] tokens avg)
  Tier 2: [P] files (~[Q] tokens avg)

Obsidian Wiki (wiki/):
  [total] pages across [sections] sections
  Open in Obsidian: File > Open Vault > [project]/wiki

Commands installed:
  /kb — query, search, compile
  /lint-kb — health check

Run /lint-kb to check knowledge base health.

---

Phase 3: rebuild — Re-analyze

When the user runs /obsidian-kb rebuild:

1. Re-scan the project (new sources may have been added)
2. Compare against existing data/kb-architecture.yaml
3. Show what changed (new sources, removed sources, schema changes)
4. Propose updated architecture
5. On approval, regenerate compiler and recompile

---

Design Principles

1. The LLM is both producer and consumer. Context packs are the primary output. The Obsidian wiki is a human-readable export.

2. Architecture spec is the contract. kb-architecture.yaml bridges analysis and generation. Users can edit it. It documents what the KB tracks.

3. Freshness over rebuilding. The compiler checks source timestamps. Slash commands check freshness.json. Recompile only when sources change.

4. The compiler is project-specific code, not a template instantiation. You write the compiler fresh each time, tailored to the project's actual schema and relationships. The BrunoTwin compiler is your reference for structure, but the content is unique per project.

5. Obsidian conventions matter. Wikilinks use [[path/slug|Display Name]] format. Frontmatter uses standard YAML. Tags are lowercase. Graph coloring uses path: queries. These make the wiki work well in Obsidian without plugins.