xberg

<!-- AI-RULEZ :: GENERATED FILE — DO NOT EDIT Content-Hash: blake3:bdb659ec3959943d7e4b8da0cc3b8fbe51dc3bfec07e27dbbd8f84e190a63e1a Source-Hash: blake3:1e46ba9bed26e6452c3c82c60b33a2fde65ced1c447fbee44693532abe509487 Schema-Version: v1 --> # Xberg Document Extraction Xberg is a document intelligence library with a Rust core and bindings for Python, TypeScript/Node.js, Ruby, PHP, Go, Java, C#, Elixir, WebAssembly, Dart, Kotlin Android, Swift, Zig, and C. It extracts text, tables, metadata, and images from 107 formats across 140 unique file extensions and accepts 53 compatibility MIME aliases, including PDF, Office documents, images, HTML, email, archives, and academic formats. Use this skill when writing code that: - Extracts text or metadata from documents - Performs OCR on scanned documents or images - Batch-processes multiple files - Configures extraction options (output format, chunking, OCR, language detection) - Implements custom plugins (post-processors, validators, OCR backends) > If the `xberg` MCP server is registered in this session, prefer its tools over shelling out to the CLI — they expose the same extraction surface with structured arguments and results. ## Installation ### Python ```bash pip install xberg ``` ### Node.js ```bash npm install @xberg-io/xberg ``` ### Rust ```bash cargo add xberg ``` ```toml # Cargo.toml [dependencies] xberg = { version = "1.1.0", features = ["full"] } tokio = { version = "1", features = ["full"] } # feature flags: pdf, ocr, chunking, embeddings, language-detection, keywords, api, mcp # (or "formats" / "full" aggregates); tokio-runtime is on by default ``` ### CLI ```bash brew install xberg-io/tap/xberg # or run without a persistent install (the CLI proxy package self-installs the binary): npx @xberg-io/xberg-cli@1.3.10 --help uvx --from xberg-cli==1.3.8 xberg --help # or download a prebuilt binary from the latest GitHub release: # https://github.com/xberg-io/xberg/releases/latest # or build from source: cargo install xberg-cli ``` ## Quick Start The library entry points are `extract(input, config)` and `extract_batch(inputs, config)`. Both return an `ExtractionResult` **envelope** — the extracted document(s) live in `result.results`, and per-document data (`content`, `tables`, `metadata`, …) is on each `result.results[i]`. Python and Node are async-only. ### Python ```python import asyncio from xberg import ExtractInput, extract, ExtractionConfig async def main() -> None: result = await extract(ExtractInput(uri="document.pdf"), ExtractionConfig()) doc = result.results[0] print(doc.content) # extracted text print(doc.metadata) # document metadata print(doc.tables) # extracted tables asyncio.run(main()) ``` ### Node.js ```typescript import { extract } from "@xberg-io/xberg"; const output = await extract({ kind: "uri", uri: "document.pdf" }); const doc = output.results[0]; console.log(doc.content); console.log(doc.metadata); console.log(doc.tables); ``` ### Rust ```rust use xberg::{extract, ExtractInput, ExtractionConfig}; #[tokio::main] async fn main() -> xberg::Result<()> { let output = extract(ExtractInput::from_uri("document.pdf"), &ExtractionConfig::default()).await?; println!("{}", output.results[0].content); Ok(()) } ``` ### CLI ```bash xberg extract document.pdf xberg extract document.pdf --format json xberg extract document.pdf --content-format markdown ``` ## Configuration All languages use the same configuration structure with language-appropriate naming conventions. ### Python (snake_case) ```python from xberg import ( ExtractInput, extract, ExtractionConfig, OcrConfig, TesseractConfig, PdfConfig, ChunkingConfig, OutputFormat, ) config = ExtractionConfig( ocr=OcrConfig( backend="tesseract", language=["eng"], tesseract_config=TesseractConfig(psm=6, enable_table_detection=True), ), pdf_options=PdfConfig(passwords=["secret123"]), chunking=ChunkingConfig(max_characters=1000, overlap=200), output_format=OutputFormat("markdown"), ) result = await extract(ExtractInput(uri="document.pdf"), config) ``` ### Node.js (camelCase) ```typescript import { extract, type ExtractionConfig } from "@xberg-io/xberg"; const config: ExtractionConfig = { ocr: { backend: "tesseract", language: ["eng"] }, pdfOptions: { passwords: ["secret123"] }, chunking: { maxCharacters: 1000, overlap: 200 }, outputFormat: "markdown", }; const output = await extract({ kind: "uri", uri: "document.pdf" }, config); ``` ### Rust (snake_case) ```rust use xberg::{extract, ExtractInput, ExtractionConfig, OcrConfig, ChunkingConfig, OutputFormat}; let config = ExtractionConfig { ocr: Some(OcrConfig { backend: "tesseract".into(), language: vec!["eng".to_string()], ..Default::default() }), chunking: Some(ChunkingConfig { max_characters: 1000, overlap: 200, ..Default::default() }), output_format: OutputFormat::Markdown, ..Default::default() }; let output = extract(ExtractInput::from_uri("document.pdf"), &config).await?; ``` ### Config File (TOML) ```toml output_format = "markdown" [ocr] backend = "tesseract" language = "eng" [chunking] max_characters = 1000 overlap = 200 [pdf_options] passwords = ["secret123"] ``` ```bash # CLI: auto-discovers xberg.toml/yaml/yml/json in current/parent directories xberg extract doc.pdf # or explicit: xberg extract doc.pdf --config xberg.toml xberg extract doc.pdf --config-json '{"ocr":{"backend":"tesseract","language":"deu"}}' ``` ## Batch Processing `extract_batch` takes a list of `ExtractInput`s and returns one envelope whose `results` array holds documents in discovery order; one input may yield multiple documents. Per-input failures are reported in `result.errors` with an input index. ### Python ```python from xberg import ExtractInput, extract_batch, ExtractionConfig inputs = [ ExtractInput(uri="doc1.pdf"), ExtractInput(uri="doc2.docx"), ExtractInput(uri="doc3.xlsx"), ] output = await extract_batch(inputs, ExtractionConfig()) for doc in output.results: print(f"{len(doc.content)} chars extracted") ``` ### Node.js ```typescript import { extractBatch } from "@xberg-io/xberg"; const output = await extractBatch([ { kind: "uri", uri: "doc1.pdf" }, { kind: "uri", uri: "doc2.docx" }, ]); for (const doc of output.results) { console.log(`${doc.content.length} chars`); } ``` ### Rust ```rust use xberg::{extract_batch, ExtractInput, ExtractionConfig}; let config = ExtractionConfig::default(); let inputs = vec![ExtractInput::from_uri("doc1.pdf"), ExtractInput::from_uri("doc2.docx")]; let output = extract_batch(inputs, &config).await?; ``` ### CLI ```bash xberg batch *.pdf --format json xberg batch docs/*.docx --content-format markdown ``` ## OCR OCR runs automatically for images and scanned PDFs. Tesseract is the default backend (native binding, no external install required). ### Backends Select with `OcrConfig.backend`: - **tesseract** (default): built-in native binding. All Tesseract languages supported. - **paddleocr** (`"paddleocr"` / `"paddle-ocr"`): ONNX-based PaddleOCR. - **vlm**: Vision-Language-Model OCR (configure via `OcrConfig.vlm_config`). Custom backends can be registered in Python/Node via `register_ocr_backend` (see [Advanced Features](references/advanced-features.md)). ### Language Codes ```python config = ExtractionConfig(ocr=OcrConfig(language=["eng"])) # English config = ExtractionConfig(ocr=OcrConfig(language=["eng", "deu"])) # Multiple # The single-string shorthand ("eng+deu") is only accepted in config files / --config-json, # not in the OcrConfig constructor (Python takes a list, Node takes an array). ``` ### Force OCR ```python config = ExtractionConfig(force_ocr=True) # OCR even if text is extractable ``` ## Result Envelope and Document Fields `extract` / `extract_batch` return an `ExtractionResult` envelope: `results` (list of documents), `errors` (per-input failures), and `summary` (counts). Per-document fields live on each document in `results` — bind `doc = result.results[0]` (Python/Node) or `&output.results[0]` (Rust) first. | Field | Python (`doc.`) | Node.js (`doc.`) | Rust (`document.`) | Description | | ------------ | ---------------------- | --------------------- | ----------------------- | --------------------------------------------- | | Text content | `content` | `content` | `content` | Extracted text (str/String) | | MIME type | `mime_type` | `mimeType` | `mime_type` | Input document MIME type | | Metadata | `metadata` | `metadata` | `metadata` | Document metadata (flat mapping) | | Tables | `tables` | `tables` | `tables` | Extracted tables with cells + markdown | | Languages | `detected_languages` | `detectedLanguages` | `detected_languages` | Detected languages (if enabled) | | Chunks | `chunks` | `chunks` | `chunks` | Text chunks (if chunking enabled) | | Images | `images` | `images` | `images` | Extracted images (if enabled) | | Elements | `elements` | `elements` | `elements` | Semantic elements (if element_based format) | | Pages | `pages` | `pages` | `pages` | Per-page content (if page extraction enabled) | | Keywords | `extracted_keywords` | `extractedKeywords` | `extracted_keywords` | Extracted keywords (if enabled) | ## Error Handling ### Python `extract` / `extract_batch` raise an exported `XbergError` subclass, such as `ParsingError`, `SecurityError`, or `ValidationError`, including calls with `on_progress`. Catch `XbergError` for extraction failures. Per-input failures during `extract_batch` are reported non-fatally in `result.errors`. ```python from xberg import ExtractInput, extract, ExtractionConfig, XbergError try: result = await extract(ExtractInput(uri="file.pdf"), ExtractionConfig()) for err in result.errors: print(f"Per-input error: {err}") except XbergError as e: print(f"Extraction failed: {e}") ``` ### Node.js The Node binding throws plain `Error` objects (it does not export typed error subclasses). Catch with `instanceof Error`, and inspect `output.errors` for non-fatal per-input failures. ```typescript import { extract } from "@xberg-io/xberg"; try { const output = await extract({ kind: "uri", uri: "file.pdf" }); if (output.errors.length > 0) { console.error("Per-input errors:", output.errors); } } catch (e) { if (e instanceof Error) { console.error(`Extraction failed: ${e.message}`); } } ``` ### Rust ```rust use xberg::{extract, ExtractInput, ExtractionConfig, XbergError};