doc-reader

# Universal Document Reader ## Purpose Return extracted text from any supported file in a single unified JSON envelope. The tool dispatches by file extension — you always call the same tool regardless of format. ### Supported formats | Category | Extensions | Notes | |---|---|---| | PDF | `.pdf` | Text pages extracted in ms; scanned/image pages fall back to OCR | | Word | `.docx` | Paragraphs + table cells | | Excel | `.xlsx`, `.xls` | All sheets, first 100 rows per sheet as preview | | PowerPoint | `.pptx` | Slide text content | | Images | `.png/.jpg/.jpeg/.gif/.bmp/.webp/.tiff` | OCR only | | CSV / TSV | `.csv`, `.tsv` | Raw text with encoding fallback | | Plain text | `.txt/.md/.log/.rst` | Encoding fallback | | Config | `.json/.yaml/.yml/.toml/.ini/.cfg/.env` | Raw text | | Markup | `.html/.htm/.xml` | Raw text (no HTML stripping) | | Source code | `.py/.js/.ts/.tsx/.go/.rs/.java/.cpp/.c/.sql/.sh/...` | Raw text | | Unknown extension | anything else | Best-effort read as UTF-8/GBK text | **Blocked** (rejected at `/upload`): executables (`.exe/.dll/.so/...`) and archives (`.zip/.tar/...`). Ask the user to unpack archives locally first. ## Usage **Always call the tool directly — do not run Python from bash.** ``` read_document(file_path="uploads/paper.pdf") read_document(file_path="uploads/annual_report.pdf", pages="1-10") read_document(file_path="uploads/contract.docx") read_document(file_path="uploads/sales.xlsx") read_document(file_path="uploads/deck.pptx") read_document(file_path="uploads/chart.png") # image → OCR read_document(file_path="uploads/config.yaml") read_document(file_path="uploads/notes.md") ``` The `pages` parameter only applies to PDF; other formats ignore it. ## Return envelope All formats share this shape: ```json { "status": "ok", "file": "paper.pdf", "format": "pdf", "char_count": 52000, "truncated": true, "text": "..." } ``` Format-specific extra fields: | Format | Extra keys | |---|---| | `pdf` | `total_pages`, `pages_read`, `ocr_pages`, `ocr_engine`, `ocr_quality`, `skipped_pages` | | `docx` | `paragraphs`, `tables` | | `excel` | `sheets` (array of `{name, rows, cols}`) | | `pptx` | `slides` | | `text` | `encoding`, `size` | Content longer than 15000 chars is truncated; for PDFs use the `pages` parameter to read slices. ## Workflows ### Paper / report summary ``` 1. read_document(file_path="paper.pdf") → full text 2. Extract abstract, methodology, conclusion → summarize ``` ### Contract review ``` 1. read_document(file_path="contract.docx") → paragraphs + tables 2. Flag key clauses (termination, liability, payment, IP) ``` ### Spreadsheet quick-look ``` 1. read_document(file_path="sales.xlsx") → all sheet previews 2. If user wants trade journal analysis specifically, pivot to `analyze_trade_journal` tool instead (see trade-journal skill). ``` ### Chart / screenshot / scanned PDF ``` 1. read_document(file_path="scan.png") → OCR text 2. If OCR returns empty, tell the user; don't fabricate. ``` ## OCR Configuration The `read_document` tool automatically uses OCR for PDF pages with insufficient extractable text. ### OCR Threshold Use `min_text_per_page` to control when OCR is triggered (default: 50 characters): ```python read_document("scanned_report.pdf", min_text_per_page=10) # More aggressive OCR read_document("mixed_pdf.pdf", min_text_per_page=100) # Less aggressive OCR ``` ### OCR Engine Configuration Two OCR engines are built in — no extra packages needed beyond the engine SDK: | Engine | Type | Requires | Install | |--------|------|----------|---------| | `rapid` | Local (offline) | `rapidocr_onnxruntime` | `pip install rapidocr_onnxruntime` | | `llm-vision` | Cloud | A vision-capable LLM model + API key | No extra install — uses your existing LLM provider config | The `llm-vision` engine works with **any OpenAI-compatible vision model** (GPT-4o, Qwen-VL, Gemini, Claude, GLM-4V, etc.). It reuses your existing `LANGCHAIN_PROVIDER` / `LANGCHAIN_MODEL_NAME` / API key configuration — no separate provider mapping needed. If you explicitly set `VIBE_TRADING_OCR_ENGINE=llm-vision`, your model choice is trusted; a real API error from the provider is clearer feedback than a heuristic guess. To override the model used for OCR (without changing your agent's main model): ``` VIBE_TRADING_OCR_LLM_MODEL=qwen3.7-plus ``` Set `VIBE_TRADING_OCR_ENGINE` to select the engine: - `auto` (default): use local engines only, never cloud (privacy: document pages never leave the machine) - `rapid`: force RapidOCR (local, ONNX) - `llm-vision`: force LLM vision OCR (cloud — pages are sent to your configured LLM provider) - `none`: disable OCR entirely ### Response Fields PDF responses include OCR metadata: - `ocr_engine`: name of the OCR engine used (e.g. "rapid", "llm-vision") or `null` - `ocr_pages`: number of pages processed via OCR - `skipped_pages`: number of pages skipped (no OCR engine available) - `ocr_quality`: object with `quality_flag` (`good`/`degraded`/`no_ocr_engine`/`no_ocr_needed`), `ocr_pages`, and `text_density` (chars per page) ## Notes - **Encoding fallback** order for text: utf-8 → utf-8-sig → gbk → gb2312 → big5 → latin-1. - **OCR** uses the configured engine (RapidOCR for local, or LLM vision for cloud). If no engine is available, image/scanned files return empty `text` with a `note` field — tell the user to install `rapidocr-onnxruntime` or set `VIBE_TRADING_OCR_ENGINE=llm-vision` with a vision-capable model. - **Excel previews** are limited to 100 rows per sheet to stay in budget. If the user needs full data (e.g. trade journals), call `analyze_trade_journal` instead. - **Source-code files** are returned raw; do not re-format or re-indent.