thoth-codebase-map

name	thoth-codebase-map
description	Orientation map for the thoth codebase — what to read first, what each module does, the vault page-type model, and the Hindsight semantic-index reality. Use when starting work on thoth and you need to know where things live or which file owns a responsibility, before diving into a feature or bug.

thoth codebase map

thoth is a self-hostable personal knowledge-management appliance: messages/files go to a Slack bot, an LLM pipeline classifies + curates + files them as Markdown notes in an Obsidian vault (git-backed), and Claude reaches the vault through an MCP server.

Read these first

• CLAUDE.md — the working rules (notably: no backward-compat / migration prose).
• SPEC.md — the authoritative spec; sections are referenced throughout the code.
• docs/explanations/architecture.md — the high-level pipeline diagrams.
• docs/adr/ (a.k.a. docs/explanations/decisions/) — the binding design decisions. ADR 0005 = the folder/page-type model; check here before changing classification, folders, or the curate contract.

Two invariants you must preserve

1. The vault is canonical. The semantic index, the SQLite state, the LLM calls are all disposable, rebuildable projections of the plain-Markdown git repo. Never make the vault depend on a derived store; anything authored must live in the vault, not only in an index. (This is why a page's one-line summary lives in page frontmatter, not only in index.md.)
2. The tool surface is closed. The appliance LLM has no shell and no arbitrary filesystem access — only a fixed set of validated, path-confined tools. Don't add a tool that can escape the vault or run commands.

Finding your way around src/thoth/

Most names say what they are (slack_app/, query/, summary/, lint/, budget/, config/, state/, git_sync.py, mcp_server/) — ls src/thoth/ plus each module's docstring is the authoritative, never-stale map, so this skill deliberately does not restate it (a hand-kept file list just drifts). The larger subsystems are packages: the package __init__ is the public surface (import from the package, never a submodule) and its docstring is the map of the submodules; underscore-prefixed submodules are private. The "Code layout" section of docs/explanations/architecture.md has the full layering and per-package submodule table. Only the non-obvious ones, where the name doesn't reveal the role:

• extract.py vs analyse/ — extract.py fetches external content (Firecrawl page→Markdown, Whisper transcription); the analyse/ package does vision / OCR / PDF analysis. Easy to reach for the wrong one — and note ingest/analyse.py is a third thing: the ingest pass that calls the analyse/ package's Analyser.
• llm/ — the single Anthropic seam (LLM.complete in llm/client.py); also owns the PKM persona and the curate file-plan contract + validation. The classify/curate prompts live with their passes in ingest/classify.py and ingest/curate.py. Change LLM behavior via the prompt — not with output post-processing.
• hindsight.py — HTTP client to the standalone hindsight-api server (see the foot-gun below); the reindex_from_vault/ package rebuilds that index from the vault.
• ingest/ — the bounded 8-pass capture pipeline (persist raw → classify → capture_raw → fetch_candidates → curate → retain → commit → report); raw is persisted before any LLM call so nothing is lost on restart. One submodule per pass group (raw_capture, classify, curate, finalise), composed by pipeline.py.
• intent.py — the cheap Haiku intent gate (see behavior note below).

Vault page-type model (ADR 0005 + ADR 0013)

Four flat content folders plus machinery. Every content page carries the universal set title,type,created,updated,source,tags,summary,personal (personal is a real boolean defaulted to false at write time; tags are descriptive topics only — view-critical facets are properties):

• Reference pages — entities/, notes/, memories/ (lifecycle-free). Memories add memory_date (falls back to created).
• Action pages — actions/ (todos, errands AND the media queue). Carry kind: task|media|errand + status: todo|in_progress|done|cancelled (one lifecycle; a media item is kind: media with media_type/url), optionally due_date/priority. Glossed with a summary like every other content page; surfaced by the _bases dashboards.
• inbox/ — durable pre-curate holding pages (machinery set: sha256, no tags). raw/ — immutable sources.
• index.md is a static 5-section Home dashboard (callouts embedding actions.base/triage.base views; reference.base is a link line); agents never write to it. log.md is the activity log.

Intent-gate behavior (useful when testing): a plain declarative fact / concept / person / memory classifies as a reference page; phrasing like "read / watch / buy / book / meet / remember to X" classifies as an action (kind task/media/ errand). Every content page gets a summary; to exercise the reference path, the test capture must be declarative, not a task.

Hindsight reality (foot-gun)

Hindsight is a fact-extraction engine (entities/observations/chunks for a query), not token-chunking, and it exposes no page-level embeddings or pairwise cosine. Features framed as "reuse Hindsight embeddings + a cosine threshold" (near-duplicate detection, idea-mining similarity) are not buildable as written — they need a separate embedding-access decision first. The index is a rebuildable projection; recall is the last/most-expensive query pass.

The two retrieval modalities are complementary, not redundant. QueryEngine.answer (behind pkm_search) blends three passes: grep over curated folders, wikilink graph-follow, and semantic recall via Hindsight. Recall always gets a vote when enabled — it runs concurrently in a worker thread and the candidate sets are fused by Reciprocal Rank Fusion (RRF_K=60, fuse on rank; #143 / ADR 0012, implemented in query/_blend.py) — there is no "only when grep looks thin" gate any more. grep = exact / authoritative / always-fresh on the vault bytes; Hindsight = semantic / vocabulary-bridging but derived, lossy, eventually-consistent (it holds LLM-extracted facts, not text — exact tokens like IDs/filenames may not survive, and a just-written or budget-deferred page is grep-only until reindex). Don't treat grep as redundant.