pdf-brain-ingest

# PDF Brain Ingest v2 (ADR-0234) Staged artifact-chain pipeline with durable NAS storage, nomic embeddings, and workload queue orchestration. ## Pipeline v2 Architecture ``` PDF (source, immutable) → Stage 1: CONVERT — opendataloader-pdf → {docId}.md (NAS artifact) → Stage 2: CLASSIFY + SUMMARIZE — taxonomy + LLM summary → {docId}.meta.json (NAS artifact) → Stage 3: CHUNK — markdown-native headings, no overlap → {docId}.chunks.jsonl (NAS artifact) → Stage 4: INDEX — upsert to docs + docs_chunks_v2 (nomic-embed-text-v1.5, 768-dim) ``` **Key properties:** - **Durable**: artifacts on NAS RAID5, survive reboots/crashes - **Resumable**: each stage checks for existing artifacts, skips if present - **Recoverable**: re-run any stage from existing artifacts without re-extracting - **Observable**: OTEL event per stage per book **Artifacts dir**: `${DOCS_ARTIFACTS_DIR}/{docId}/`, using the machine-local value from `~/.config/system-bus.env` - `{docId}.md` — structured markdown extraction - `{docId}.meta.json` — taxonomy, summary, metadata - `{docId}.chunks.jsonl` — chunk records, one per line ## Core Workflow ### 1) Preflight ```bash joelclaw status joelclaw docs status ``` Status now shows both v1 and v2 collection stats plus artifact availability. ### 2) Single File Ingest (v1 pipeline) ```bash joelclaw docs add "/absolute/path/to/file.pdf" joelclaw docs add "/absolute/path/to/file.pdf" --title "Title" --tags "tag1,tag2" --category programming ``` ### 3) Single File v2 Reindex (artifact pipeline) ```bash joelclaw docs reindex-v2 "/absolute/path/to/file.pdf" joelclaw docs reindex-v2 "/absolute/path/to/file.pdf" --title "Title" --skip-existing ``` Fires `docs/reindex-v2.requested` → 4-stage artifact pipeline → NAS artifacts + docs_chunks_v2. ### 4) Batch Reindex (full library) ```bash # Reindex all PDFs from the configured NAS book root joelclaw docs batch-reindex --skip-existing # Reindex from existing Typesense docs collection joelclaw docs batch-reindex --from-collection --skip-existing ``` Fires `docs/reindex-batch.requested` → scans NAS/collection → dispatches individual reindex-v2 events in batches of 10, concurrency 3. `--skip-existing` skips books that already have all 3 artifacts on NAS (default true). ### 5) Monitor Progress ```bash # Artifact count on NAS (load machine-local config without printing it) set -a; . ~/.config/system-bus.env; set +a find "$DOCS_ARTIFACTS_DIR" -mindepth 1 -maxdepth 1 -type d | wc -l # v2 collection chunk count joelclaw docs status # OTEL events from pipeline joelclaw otel search "docs.reindex" --hours 4 joelclaw o11y session system-bus --hours 4 # Individual run trace joelclaw runs --count 10 joelclaw run <run-id> ``` ### 6) Inspect Artifacts ```bash # Read the extracted markdown joelclaw docs markdown <doc-id> # Read the summary + taxonomy metadata joelclaw docs summary <doc-id> # Or directly on NAS after loading machine-local config set -a; . ~/.config/system-bus.env; set +a cat "$DOCS_ARTIFACTS_DIR/<docId>/<docId>.md" cat "$DOCS_ARTIFACTS_DIR/<docId>/<docId>.meta.json" | jq wc -l "$DOCS_ARTIFACTS_DIR/<docId>/<docId>.chunks.jsonl" ``` ### 7) Retrieval from v2 ```bash # Search uses docs_chunks_v2 (nomic 768-dim) by default joelclaw docs search "distributed consensus" --limit 8 # Context expansion joelclaw docs context <chunk-id> --mode snippet-window joelclaw docs context <chunk-id> --mode parent-section joelclaw docs context <chunk-id> --mode section-neighborhood ``` ### 8) Coverage Reconcile ```bash joelclaw docs reconcile --sample 20 ``` ### 9) Recovery If the batch stalls or books fail: - Inngest retries each step automatically (default retry policy) - `--skip-existing` means re-firing the batch only processes unfinished books - Check OTEL for errors: `joelclaw otel list --level error --hours 4` - Individual retry: `joelclaw docs reindex-v2 "/path/to/failed.pdf"` ## Extraction Details - **Primary**: opendataloader-pdf v2.0.0 (Java-based, #1 in benchmarks, 0.90 accuracy) - **Fallback**: pypdf (basic text extraction if Java unavailable) - **Requires**: Java 11+ (OpenJDK 25 installed on panda, PATH configured in worker start.sh) - **Speed**: ~2.5s per book on M4 Pro ## Embedding Model - **v2**: nomic-embed-text via ollama (GPU-accelerated on M4 Pro, 768-dim, retrieval-tuned). Pre-computed at ingest time, stored as raw float[] vectors. ~150x faster than Typesense CPU auto-embed. - **v1**: `ts/all-MiniLM-L12-v2` — 384-dim, general-purpose, Typesense auto-embed (legacy, still in `docs_chunks`) - Ollama runs on panda at `localhost:11434`. System-bus-worker (host process) embeds at ingest time. ## Chunking Strategy (ADR-0234) - Markdown-native heading detection (`#` markers, not heuristics) - Recursive splitting within sections exceeding target tokens - No overlap (arxiv R100-0 finding: 45% higher precision) - Two-level hierarchy: section chunks + snippet sub-chunks - heading_path derived from actual markdown heading levels - Context inheritance: retrieval_text includes `[DOC: title] [SUMMARY: ...] [PATH: heading > path] [CONCEPTS: ...]` ## Acquisition Pipeline (aa-book → ingest) ```bash joelclaw send pipeline/book.download -d '{ "query": "designing data-intensive applications", "format": "pdf", "reason": "library expansion" }' ``` Downloads via aa-book → NAS backup → fires `docs/ingest.requested` for immediate processing. ## Inngest Events | Event | Function | Purpose | |-------|----------|---------| | `docs/ingest.requested` | docs-ingest | v1 pipeline (single file) | | `docs/reindex-v2.requested` | docs-reindex-v2 | v2 artifact pipeline (single file) | | `docs/reindex-batch.requested` | docs-reindex-batch | Batch orchestrator (all PDFs) | | `docs/backlog.requested` | docs-backlog | Legacy manifest-based backfill | | `docs/enrich.requested` | docs-enrich | Re-enrich metadata for existing doc | | `pipeline/book.download` | book-download | Acquire + ingest new book |