pdf-brain

# PDF Brain — Research → Practical System Moves Use this skill when the user wants evidence-backed synthesis from the docs library (600+ books, PDFs, long-form references), not generic web summarization. ## Pipeline v2 (ADR-0234) The docs pipeline uses a staged artifact chain: - **Extraction**: opendataloader-pdf → structured markdown with headings, tables, reading order - **Chunking**: markdown-native heading detection, no overlap, hierarchical section + snippet chunks - **Embeddings**: nomic-embed-text via ollama GPU (768-dim, retrieval-tuned, pre-computed at ingest) in `docs_chunks_v2` collection - **Artifacts**: durable on NAS at `${DOCS_ARTIFACTS_DIR}/{docId}/` — `.md`, `.meta.json`, `.chunks.jsonl`; resolve the machine-local value from `~/.config/system-bus.env` - **Summaries**: LLM-generated per-document summaries in `.meta.json` ## When to Use Trigger cues (explicit or implied): - "research this" / "from the library" / "from the books" - "pdf brain" / "correlate this to our system" - "what does the research say" / "what do the books say" - "expand this into practical ideas" ## Retrieval Workflow ### CLI path (preferred for interactive sessions) There is no separate `pdf-brain` binary. Use `joelclaw docs …`. A PATH alias `pdf-brain` → `joelclaw docs` may exist for old muscle memory. ```bash # Search across all books — semantic by default (nomic 768-dim) joelclaw docs search "distributed consensus" --limit 8 # Search within a specific book joelclaw docs search "consensus" --doc designing-dataintensive-applications-39cc0d1842a5 # Expand a chunk into surrounding context joelclaw docs context <chunk-id> --mode snippet-window --before 2 --after 2 # Get the full parent section joelclaw docs context <chunk-id> --mode parent-section # Get neighboring sections for broad context joelclaw docs context <chunk-id> --mode section-neighborhood --neighbors 2 # Read the full structured markdown of a book joelclaw docs markdown <doc-id> # Get document summary + taxonomy metadata joelclaw docs summary <doc-id> ``` ### API path (for programmatic access or docs-api consumers) ``` GET /search?q=distributed+consensus&semantic=true&expand=true&assemble=true GET /docs/:docId/toc GET /docs/:docId/markdown GET /docs/:docId/summary GET /chunks/:chunkId ``` The docs-api runs on k8s at `docs-api:3838` (Bearer auth required). ### Context expansion strategy The library supports progressive context expansion: 1. **Search** → chunk-level hits with heading_path and snippet 2. **snippet-window** → 2 chunks before/after for local context 3. **parent-section** → the full section containing the snippet 4. **section-neighborhood** → adjacent sections for broader flow 5. **markdown** → the complete structured book text Start narrow, expand only when needed. Don't dump full books into context. ## Evidence Synthesis ### Build an evidence ledger While reading, keep a compact ledger: - `doc` (title) - `chunk-id` - `claim` (one sentence) - `relevance` (why it matters to this problem) Never output synthesis without traceable evidence. ### Convert evidence into principles Turn each claim into an operational principle in imperative form: - "Treat partial failure as normal." - "Fail fast at dependency boundaries." - "Prefer idempotent replay-safe remediation loops." Avoid vague advice. Each principle must imply a technical behavior. ### Correlate to joelclaw philosophy Map principles to existing joelclaw operating rules: - single source of truth - silent failures are bugs - Inngest durability + retries - CLI-first agent interface - observability required at every step - skill/doc updates when reality changes ### Translate into action For each principle, produce: 1. **Concrete change** (file/service/config path) 2. **Validation gate** (exact command) 3. **Failure signal** (what proves it did not work) 4. **Rollback or containment move** ## Taxonomy The library is classified via SKOS taxonomy: - `jc:docs:programming` (systems, languages, architecture) - `jc:docs:business` (creator economy) - `jc:docs:education` (learning science, pedagogy) - `jc:docs:design` (game, systems, product) - `jc:docs:marketing`, `jc:docs:strategy`, `jc:docs:ai`, `jc:docs:operations` Use `--concept jc:docs:programming:systems` to narrow by domain. Use `joelclaw docs status` to see facet counts per concept. ## Rules - Do not fabricate quotes or claims. - Always cite chunk IDs for non-obvious assertions. - Do not output "book report" fluff. Translate to operations. - If infra changes are proposed, include verification commands. - If work implies architectural policy change, tie it to an ADR path. - Start with search, expand only as needed. Don't waste context on full book dumps.