xberg

<!-- AI-RULEZ :: GENERATED FILE — DO NOT EDIT Content-Hash: blake3:41a41741e3865333a4b14e83a550aaecdae016a9469db77a921f556b83cd4854 Source-Hash: blake3:8d48733330070809966ed2939557b5ca2c73f676d4c16daa306750bc5fb8d5bc Schema-Version: v1 --> # Xberg Document Extraction Xberg is a high-performance document intelligence library with a Rust core and native bindings for Python, Node.js/TypeScript, Ruby, Go, Java, C#, PHP, and Elixir. It extracts text, tables, metadata, and images from 91+ file formats including PDF, Office documents, images (with OCR), HTML, email, archives, and academic formats. Use this skill when writing code that: - Extracts text or metadata from documents - Performs OCR on scanned documents or images - Batch-processes multiple files - Configures extraction options (output format, chunking, OCR, language detection) - Implements custom plugins (post-processors, validators, OCR backends) > If the `xberg` MCP server is registered in this session, prefer its tools over shelling out to the CLI — they expose the same extraction surface with structured arguments and results. ## Installation ### Python ```bash pip install xberg ``` ### Node.js ```bash npm install @xberg-io/xberg ``` ### Rust ```bash cargo add xberg ``` ```toml # Cargo.toml — the crate is currently 1.0.0-rc.1 (a pre-release, so pin it explicitly) [dependencies] xberg = { version = "1.0.0-rc.1", features = ["full"] } tokio = { version = "1", features = ["full"] } # feature flags: pdf, ocr, chunking, embeddings, language-detection, keywords, api, mcp # (or "formats" / "full" aggregates); tokio-runtime is on by default ``` ### CLI ```bash brew install xberg-io/tap/xberg # or run without a persistent install (the CLI proxy package self-installs the binary): npx @xberg-io/xberg-cli --help uvx --from xberg-cli xberg --help # or download a prebuilt binary from the latest GitHub release: # https://github.com/xberg-io/xberg/releases/latest # or build from source: cargo install xberg-cli ``` ## Quick Start The library entry points are `extract(input, config)` and `extract_batch(inputs, config)`. Both return an `ExtractionResult` **envelope** — the extracted document(s) live in `result.results`, and per-document data (`content`, `tables`, `metadata`, …) is on each `result.results[i]`. Python and Node are async-only. ### Python ```python import asyncio from xberg import ExtractInput, extract, ExtractionConfig async def main() -> None: result = await extract(ExtractInput.from_uri("document.pdf"), ExtractionConfig()) doc = result.results[0] print(doc.content) # extracted text print(doc.metadata) # document metadata print(doc.tables) # extracted tables asyncio.run(main()) ``` ### Node.js ```typescript import { extract } from "@xberg-io/xberg"; const output = await extract({ kind: "uri", uri: "document.pdf" }); const doc = output.results[0]; console.log(doc.content); console.log(doc.metadata); console.log(doc.tables); ``` ### Rust ```rust use xberg::{extract, ExtractInput, ExtractionConfig}; #[tokio::main] async fn main() -> xberg::Result<()> { let output = extract(ExtractInput::from_uri("document.pdf"), &ExtractionConfig::default()).await?; println!("{}", output.results[0].content); Ok(()) } ``` ### CLI ```bash xberg extract document.pdf xberg extract document.pdf --format json xberg extract document.pdf --content-format markdown ``` ## Configuration All languages use the same configuration structure with language-appropriate naming conventions. ### Python (snake_case) ```python from xberg import ( ExtractInput, extract, ExtractionConfig, OcrConfig, TesseractConfig, PdfConfig, ChunkingConfig, ) config = ExtractionConfig( ocr=OcrConfig( backend="tesseract", language="eng", tesseract_config=TesseractConfig(psm=6, enable_table_detection=True), ), pdf_options=PdfConfig(passwords=["secret123"]), chunking=ChunkingConfig(max_characters=1000, overlap=200), output_format="markdown", ) result = await extract(ExtractInput.from_uri("document.pdf"), config) ``` ### Node.js (camelCase) ```typescript import { extract, type ExtractionConfig } from "@xberg-io/xberg"; const config: ExtractionConfig = { ocr: { backend: "tesseract", language: "eng" }, pdfOptions: { passwords: ["secret123"] }, chunking: { maxChars: 1000, maxOverlap: 200 }, outputFormat: "markdown", }; const output = await extract({ kind: "uri", uri: "document.pdf" }, config); ``` ### Rust (snake_case) ```rust use xberg::{extract, ExtractInput, ExtractionConfig, OcrConfig, ChunkingConfig, OutputFormat}; let config = ExtractionConfig { ocr: Some(OcrConfig { backend: "tesseract".into(), language: "eng".into(), ..Default::default() }), chunking: Some(ChunkingConfig { max_characters: 1000, overlap: 200, ..Default::default() }), output_format: OutputFormat::Markdown, ..Default::default() }; let output = extract(ExtractInput::from_uri("document.pdf"), &config).await?; ``` ### Config File (TOML) ```toml output_format = "markdown" [ocr] backend = "tesseract" language = "eng" [chunking] max_characters = 1000 overlap = 200 [pdf_options] passwords = ["secret123"] ``` ```bash # CLI: auto-discovers xberg.toml in current/parent directories xberg extract doc.pdf # or explicit: xberg extract doc.pdf --config xberg.toml xberg extract doc.pdf --config-json '{"ocr":{"backend":"tesseract","language":"deu"}}' ``` ## Batch Processing `extract_batch` takes a list of `ExtractInput`s and returns one envelope whose `results` array holds a document per input (in input order); per-input failures are reported in `result.errors`. ### Python ```python from xberg import ExtractInput, extract_batch inputs = [ ExtractInput.from_uri("doc1.pdf"), ExtractInput.from_uri("doc2.docx"), ExtractInput.from_uri("doc3.xlsx"), ] output = await extract_batch(inputs) for doc in output.results: print(f"{len(doc.content)} chars extracted") ``` ### Node.js ```typescript import { extractBatch } from "@xberg-io/xberg"; const output = await extractBatch([ { kind: "uri", uri: "doc1.pdf" }, { kind: "uri", uri: "doc2.docx" }, ]); for (const doc of output.results) { console.log(`${doc.content.length} chars`); } ``` ### Rust ```rust use xberg::{extract_batch, ExtractInput, ExtractionConfig}; let config = ExtractionConfig::default(); let inputs = vec![ExtractInput::from_uri("doc1.pdf"), ExtractInput::from_uri("doc2.docx")]; let output = extract_batch(inputs, &config).await?; ``` ### CLI ```bash xberg batch *.pdf --format json xberg batch docs/*.docx --content-format markdown ``` ## OCR OCR runs automatically for images and scanned PDFs. Tesseract is the default backend (native binding, no external install required). ### Backends Select with `OcrConfig.backend`: - **tesseract** (default): built-in native binding. All Tesseract languages supported. - **paddleocr** (`"paddleocr"` / `"paddle-ocr"`): ONNX-based PaddleOCR. - **vlm**: Vision-Language-Model OCR (configure via `OcrConfig.vlm_config`). Custom backends can be registered in Python/Node via `register_ocr_backend` (see [Advanced Features](references/advanced-features.md)). ### Language Codes ```python config = ExtractionConfig(ocr=OcrConfig(language="eng")) # English config = ExtractionConfig(ocr=OcrConfig(language="eng+deu")) # Multiple (string) config = ExtractionConfig(ocr=OcrConfig(language=["eng", "deu"])) # Multiple (list) ``` ### Force OCR ```python config = ExtractionConfig(force_ocr=True) # OCR even if text is extractable ``` ## Result Envelope and Document Fields `extract` / `extract_batch` return an `ExtractionResult` envelope: `results` (list of documents), `errors` (per-input failures), and `summary` (counts). Per-document fields live on each document in `results` — bind `doc = result.results[0]` (Python/Node) or `&output.results[0]` (Rust) first. | Field | Python (`doc.`) | Node.js (`doc.`) | Rust (`document.`) | Description | | ------------ | ---------------------- | --------------------- | ----------------------- | --------------------------------------------- | | Text content | `content` | `content` | `content` | Extracted text (str/String) | | MIME type | `mime_type` | `mimeType` | `mime_type` | Input document MIME type | | Metadata | `metadata` | `metadata` | `metadata` | Document metadata (flat mapping) | | Tables | `tables` | `tables` | `tables` | Extracted tables with cells + markdown | | Languages | `detected_languages` | `detectedLanguages` | `detected_languages` | Detected languages (if enabled) | | Chunks | `chunks` | `chunks` | `chunks` | Text chunks (if chunking enabled) | | Images | `images` | `images` | `images` | Extracted images (if enabled) | | Elements | `elements` | `elements` | `elements` | Semantic elements (if element_based format) | | Pages | `pages` | `pages` | `pages` | Per-page content (if page extraction enabled) | | Keywords | `extracted_keywords` | `extractedKeywords` | `extracted_keywords` | Extracted keywords (if enabled) | ## Error Handling ### Python Exceptions inherit from `XbergError` (note: the OCR exception is `OcrError`, not `OCRError`). Per-input failures during `extract_batch` are reported non-fatally in `result.errors`. ```python from xberg import ( ExtractInput, extract, ExtractionConfig, XbergError, ParsingError, OcrError, ValidationError, MissingDependencyError, ) try: result = await extract(ExtractInput.from_uri("file.pdf"), ExtractionConfig()) except ParsingError as e: print(f"Failed to parse: {e}") except OcrError as e: print(f"OCR failed: {e}") except ValidationError as e: print(f"Invalid input: {e}") except MissingDependencyError as e: print(f"Missing dependency: {e}") except XbergError as e: print(f"Extraction failed: {e}") ``` ### Node.js The Node binding throws plain `Error` objects (it does not export typed error subclasses). Catch with `instanceof Error`, and inspect `output.errors` for non-fatal per-input failures. ```typescript import { extract } from "@xberg-io/xberg"; try { const output = await extract({ kind: "uri", uri: "file.pdf" }); if (output.errors.length > 0) { console.error("Per-input errors:", output.errors); } } catch (e) { if (e instanceof Error) { console.error(`Extraction failed: ${e.message}`); } } ``` ### Rust ```rust