| name | knowledge-pipeline |
| description | Operate the local-first Weblyfe knowledge pipeline for ingesting documents and workspace content, embedding locally, semantic search, and graph exploration. |
knowledge-pipeline Skill
Purpose
Operate the local-first Weblyfe knowledge pipeline: ingest documents and Google Drive / Notion content, embed locally, search by meaning, and surface it as a 3D graph in Mission Control. Built 2026-06-03. Codifies the patterns so future sessions and fleet agents do not re-derive them.
Where it lives
~/clawd/projects/knowledge-pipeline/. Store at .data/knowledge.db (SQLite, gitignored). Drive cache .data/drive-cache/, Notion cache .data/notion-cache/.
Core principle (from the adversarial review)
Local-first is REAL: SQLite + local embeddings are canonical. Pinecone is a deferred optional sync target, never the critical path. No client data is embedded until per-tenant isolation is hard. tenant is fail-closed (NOT NULL, scoped search never returns another tenant or UNASSIGNED).
Quick reference
| Task | Command |
|---|
| Ingest a local folder | python3 ingest.py <dir> --tenant weblyfe-internal |
| Search | python3 query.py "your question" --tenant weblyfe-internal -k 5 |
| Sync a Google Drive folder | python3 sync_drive.py <driveFolderId> then ingest .data/drive-cache/<id> |
| Export Notion (streaming) | NOTION_API_KEY=... python3 -u sync_notion.py then ingest .data/notion-cache --tenant notion |
Stack
- Embeddings: local Ollama
bge-m3 (1024-dim). ollama pull bge-m3. Helper kp/embed.py hits localhost:11434/api/embed. Nothing leaves the machine.
- Store: SQLite + brute-force numpy cosine (
kp/store.py). Corpus is small (tens of thousands of chunks) so this is fast and avoids fragile native installs on Python 3.14. sqlite-vec / Chroma is the upgrade path.
- Parse: PyMuPDF (fitz, preinstalled) for PDF, plain read for md/txt (
kp/parse.py). iter_files skips node_modules/.git/.next/etc.
- Incremental: files are skipped by md5 on re-run.
Gotchas
- Python 3.14 (
/opt/homebrew/bin/python3) has a broken pyexpat for native builds; chromadb/sentence-transformers installs are fragile. Stick to ollama + SQLite + numpy.
- Notion
/v1/search returns thousands of page objects for a large workspace. Stream (write per page) instead of collecting all first, or it appears to hang.
- Notion rate limit is ~3 req/s; keep a 0.34s pause per request.
- nohup buffers stdout; use
python3 -u for live progress.
Mission Control integration
The 3D graph lives at MC Memory > Graph (reagraph WebGL, already installed). src/app/api/memory/graph/route.ts surfaces the pipeline DB as a knowledge cluster (GROUP BY path, cap 350 nodes). The Graph tab was gated !isLocal; unlocked in local mode in memory-browser-panel.tsx. Override the DB path with env KNOWLEDGE_DB_PATH.
Next phases (not built)
Per-tenant namespaces then client folders; FastAPI read-only for MC; audio (faster-whisper) + OCR ingest; entity/relation knowledge graph behind a flag; Pinecone remote sync; clustering/analysis; Obsidian export; voice "ask the KB".
