know-base

name

know-base

description

Build and maintain a local markdown knowledge base from raw source materials by ingesting files, scaffolding source notes, synthesizing concept pages, rebuilding derived indexes, and running integrity checks. Use when Codex needs to turn a folder of research notes, articles, papers, exports, or mixed markdown/text documents into a structured wiki-like knowledge base with `raw/`, `sources/`, `concepts/`, `indexes/`, `outputs/`, and `state/`.

Know-Base

Use this skill to create or maintain a local markdown knowledge base with a deterministic filesystem layout and LLM-authored synthesis layered on top of raw evidence.

Workflow

1. Establish the knowledge base root.

   • Use the user-provided root if one exists.
   • Otherwise initialize the canonical directories: raw/, sources/, concepts/, indexes/, outputs/, assets/, and state/.

2. Ingest source material.

   • Use scripts/run_pipeline.py with --inputs to run the full pipeline in one step, or call each script individually.
   • Use scripts/ingest_sources.py to copy or register input files into raw/ and update state/raw_manifest.json.
   • Keep raw/ immutable after ingest. If a source changes, ingest it again instead of rewriting the old file in place.

3. Compile source notes.

   • Use scripts/compile_kb.py to scaffold missing sources/*.md, create default derived index files, and write state/compile_plan.json.
   • Let the compiler prefill deterministic metadata from the raw source, including title, authors, tags, dates, and structural hints when available.
   • Use the templates in references/templates.md.

4. Synthesize source notes with AI.

   • Use scripts/source_synthesis.py queue to build state/source_synthesis_queue.json.
   • Use scripts/source_synthesis.py prompt or scripts/source_synthesis.py context for each queued note.
   • Use scripts/source_synthesis.py synthesize for a deterministic first-pass draft before manual or AI refinement when appropriate.
   • Rewrite sources/*.md so the note is grounded, TODO markers are removed, and the frontmatter reflects the evidence.

5. Synthesize concept pages.

   • Use scripts/concept_synthesis.py queue to build state/concept_synthesis_queue.json.
   • Use scripts/concept_synthesis.py prompt or scripts/concept_synthesis.py context for one concept candidate, or scripts/concept_synthesis.py synthesize for a deterministic first-pass draft.
   • By default, synthesize skips concept notes already marked fresh. Pass --force to overwrite them.
   • Create concepts/*.md only when at least two sources justify the concept, or when a single source introduces a clearly reusable concept.
   • Link every concept claim back to one or more source notes.
   • Merge overlapping concepts instead of creating near-duplicates.

6. Rebuild indexes and outputs.

   • Use scripts/rebuild_indexes.py after editing sources/ or concepts/.
   • Treat indexes/ as derived output. Regenerate it instead of editing it manually.
   • Create outputs/ artifacts only after the underlying source and concept notes are grounded.

7. Validate before stopping.

   • Run scripts/lint_kb.py to catch duplicate ids, broken links, missing frontmatter, and stale raw hashes.
   • Lint also warns when a concept note's related_sources point to a source note with a stale hash.
   • Fix integrity issues before calling the knowledge base ready.

Operating Rules

• Treat raw/ as evidence, not as a writable note layer.
• Treat sources/ as one-note-per-source normalization.
• Treat concepts/ as cross-source synthesis.
• Keep every non-trivial conclusion traceable to one or more source notes.
• Prefer incremental updates. Do not rewrite the entire vault when only a few sources changed.
• Mark unknown metadata explicitly rather than inventing it.
• Prefer markdown links to local files so the lint and index scripts can verify them deterministically.

Resource Guide

• references/schema.md
  • Read this when you need the canonical directory layout, frontmatter fields, status values, or file responsibilities.
• references/templates.md
  • Read this when creating or revising sources/*.md, concepts/*.md, or outputs/*.md.
• references/source_synthesis.md
  • Read this when performing the AI-driven rewrite of sources/*.md.
• references/concept_synthesis.md
  • Read this when grouping concepts from source notes and drafting concepts/*.md.
• scripts/run_pipeline.py
  • Use this to run the full pipeline in order. Accepts --inputs, --from-step, --limit, and --force-concepts.
• scripts/ingest_sources.py
  • Use this to initialize the knowledge base folders, copy input files into raw/, and update the manifest.
• scripts/compile_kb.py
  • Use this to scaffold missing source notes, extract deterministic raw metadata, mark stale notes, and create default derived indexes.
• scripts/source_synthesis.py
  • Use this to build the source-synthesis queue, emit structured prompts or context bundles, and auto-draft source notes.
• scripts/concept_synthesis.py
  • Use this to build concept queues, emit concept synthesis prompts or context bundles, and auto-draft concept notes.
• scripts/rebuild_indexes.py
  • Use this to regenerate topic, author, timeline, and glossary indexes from the note corpus.
• scripts/lint_kb.py
  • Use this to validate frontmatter, links, ids, raw-source integrity, unresolved TODO markers, and concept notes that reference stale sources.
• state/domain_keywords.json (optional)
  • A JSON array of lowercase strings added to the built-in concept-extraction keyword list. Create this file to improve concept extraction for domain-specific vocabularies.

Output Standard

• Ground summary and synthesis work in raw/ and sources/, not in memory alone.
• Leave the knowledge base in a rerunnable state: derived files regenerated, validation clean, and status fields updated.
• Surface uncertainty plainly. If the corpus is sparse or contradictory, write that into the note instead of smoothing it away.