lexoid-python

name

lexoid-python

description

Parse and convert documents (PDFs, images, web pages, DOCX/XLSX/PPTX, audio) inside a Python program using the `lexoid` library. Use when the user is writing Python code that needs to extract markdown from documents, run schema-constrained extraction, convert files to LaTeX, get bounding boxes, recursively crawl URLs, or integrate document parsing into a larger pipeline. Triggers include `from lexoid` imports, "use lexoid in Python", "parse PDFs programmatically", "extract structured data with a Pydantic/dataclass schema", or any request to embed parsing into a Python app/notebook.

Lexoid Python API

Lexoid's Python API is the right choice whenever the user is writing Python — notebooks, services, batch pipelines, or anything that needs the parsed result as a Python dict/list. For shell one-offs, use the lexoid-cli skill instead.

When to use this skill

• The user is writing Python and wants to parse PDFs, images, URLs, DOCX/XLSX/PPTX, audio, or text-format files.
• The user needs per-page segments, token usage, parser metadata, or bounding boxes in code.
• The user wants schema-based structured extraction (dict, dataclass, or Pydantic BaseModel).
• The user is integrating parsing into a larger app (Streamlit, FastAPI, RAG pipeline, etc.).

Setup checks

Before writing code, confirm:

1. lexoid is installed (pip install lexoid).
2. Required API key env vars are set for the chosen provider (see below).
3. For Linux DOCX → PDF conversion, LibreOffice (lowriter) is on PATH.
4. For api_provider="ollama": an ollama serve process is running at OLLAMA_BASE_URL (default http://localhost:11434) and the target model has been pulled with ollama pull <model>.
5. For api_provider="local" (SmolDocling/granite-docling, PaddleOCR-VL): no server needed — these models run in-process via transformers / PaddleOCR. The first call downloads weights from Hugging Face, so the host needs network access (or pre-cached weights) and enough disk/RAM/GPU for the chosen model.

API keys by provider:

Provider	Env var
gemini	GOOGLE_API_KEY
openai	OPENAI_API_KEY
anthropic	ANTHROPIC_API_KEY
mistral	MISTRAL_API_KEY
huggingface	HUGGINGFACEHUB_API_TOKEN
together	TOGETHER_API_KEY
openrouter	OPENROUTER_API_KEY
fireworks	FIREWORKS_API_KEY
ollama	none (uses OLLAMA_BASE_URL)
local	none

Loading API keys from .env

Lexoid reads API keys from the process environment; it does not load a .env file on its own. When keys live in a project .env, load them before calling any LLM-based API (LLM_PARSE, parse_with_schema, parse_to_latex, or AUTO when it routes to an LLM). python-dotenv ships as a Lexoid dependency, so it is already available:

from dotenv import load_dotenv

# Loads .env from the current dir (or a parent) into os.environ if the file
# exists; a no-op that returns False when no .env is found. Existing env vars
# are not overwritten unless override=True is passed.
load_dotenv()

from lexoid.api import parse
result = parse("document.pdf", parser_type="LLM_PARSE", model="gpt-4o")

Call load_dotenv() once at program/notebook startup, before the first parse(...) call. If the keys are already exported in the environment, this step is unnecessary (and harmless).

Public API

Four entry points in lexoid.api:

• parse(path, parser_type="AUTO", pages_per_split=4, max_processes=4, **kwargs) — main function. Returns a dict.
• parse_with_schema(path, schema, api=None, model="gpt-4o-mini", **kwargs) — structured JSON extraction. Returns a Python list whose shape depends on the mode and the model's output (see "Schema return shape" below).
• parse_to_latex(path, api=None, model="gpt-4o-mini", **kwargs) — returns a LaTeX string.
• parse_chunk(path, parser_type, **kwargs) — low-level single-chunk parser; users rarely need this.

ParserType enum: LLM_PARSE, STATIC_PARSE, AUTO.

parse() return shape

{
    "raw": str,                  # full markdown
    "segments": [                # one dict per page / section; may be empty
        # bboxes is included only when return_bboxes=True
        {"metadata": {"page": int}, "content": str, "bboxes": [(text, [x0, top, x1, bottom]), ...]},
        ...
    ],
    "title": str,
    "url": str,                  # input URL or "" if input was a local file
    "parent_title": str,         # parent doc title when recursive; "" otherwise
    "recursive_docs": [...],     # empty unless depth > 1
    # --- optional keys below ---
    "token_usage": {"input": int, "output": int, "total": int, "llm_page_count": int},
        # zeros under STATIC_PARSE-only; ABSENT on the HTML/recursive-URL path
        # (URL input that isn't a file-typed URL and as_pdf is not set)
    "parsers_used": [str, ...],  # ABSENT on the HTML/recursive-URL path (same condition)
    "token_cost": {...},         # only when api_cost_mapping is supplied
    "pdf_path": str,             # only when as_pdf=True; file is removed unless save_dir is also set
}

For URL inputs that resolve to HTML (no .pdf/image extension, and as_pdf=False), parse() short-circuits to recursive_read_html(), which returns only raw, segments, title, url, parent_title, and recursive_docs. Code that always reads result["token_usage"] or result["parsers_used"] will KeyError on that path — use .get(...) or guard with if "token_usage" in result.

Common recipes

Basic parsing

from lexoid.api import parse

result = parse("document.pdf")
markdown = result["raw"]
for seg in result["segments"]:
    print(seg["metadata"]["page"], seg["content"][:80])

Choose a parser explicitly

# Native-text PDFs — fastest, no API key
parse("document.pdf", parser_type="STATIC_PARSE", framework="pdfplumber")

# Scanned PDFs / images — local OCR, no API key
parse("scanned.pdf", parser_type="STATIC_PARSE", framework="paddleocr")

# LLM parsing
parse("document.pdf", parser_type="LLM_PARSE", model="gpt-4o")
parse("document.pdf", parser_type="LLM_PARSE", model="gemini-2.5-pro")
parse("document.pdf", parser_type="LLM_PARSE", model="claude-3-5-sonnet-20241022")

AUTO routing with a priority

# Speed (default): static if no images, LLM otherwise
parse("doc.pdf", parser_type="AUTO", router_priority="speed")

# Accuracy: prefers LLM, except PDFs with hidden hyperlinks
parse("doc.pdf", parser_type="AUTO", router_priority="accuracy")

# Cost: tries PaddleOCR first; LLM fallback if extracted text is too short
parse("doc.pdf", parser_type="AUTO", router_priority="cost", character_threshold=100)

# ML-based LLM auto-selection (uses lexoid/core/llm_selector.py)
parse("doc.pdf", parser_type="AUTO", autoselect_llm=True)

Local inference (no API key)

# Ollama — Lexoid forces max_processes=1 for Ollama
parse("doc.pdf", parser_type="LLM_PARSE",
      api_provider="ollama", model="gemma4:latest", max_processes=1)

# SmolDocling / granite-docling
parse("doc.pdf", parser_type="LLM_PARSE",
      api_provider="local", model="ds4sd/SmolDocling-256M-preview")

# PaddleOCR-VL
parse("doc.pdf", parser_type="LLM_PARSE",
      api_provider="local", model="PaddlePaddle/PaddleOCR-VL")

Schema-based structured extraction

Schema return shape

parse_with_schema returns a Python list. The exact shape depends on the mode and on what JSON the model emits:

• Default (per-page) mode — one entry per page. Each entry is the JSON the model returned for that page: a single dict for single-record schemas, or a list of dicts for multi-record schemas (e.g., a page of table rows). Index as result[page_index] for the page's value, and result[page_index][record_index] when each page has multiple records.
• fill_single_schema=True — a single-element list whose element is the JSON for the whole document (typically one dict).

If you need to support both single- and multi-record schemas in the same code path, normalize the per-page entries yourself (e.g., wrap a dict in [dict]).

from lexoid.api import parse_with_schema
from pydantic import BaseModel

class Invoice(BaseModel):
    invoice_number: str
    total: float

# Per-page extraction. `pages` is a list with one entry per page;
# each entry is whatever JSON the model returned for that page.
pages = parse_with_schema("invoice.pdf", schema=Invoice, model="gpt-4o-mini")
# e.g., pages[0] -> {"invoice_number": "...", "total": ...}
#       pages[0][0] -> first record if the model returned a list per page

# Single instance for the whole document — returns a one-element list.
[full] = parse_with_schema("contract.pdf", schema=Invoice,
                           model="gpt-4o", fill_single_schema=True)

# Dict schema with example data + alternate keys (improves match)
pages = parse_with_schema(
    "invoice.pdf",
    schema={"invoice_number": "string", "total": "number"},
    example_schema={"invoice_number": "INV-001", "total": 199.95},
    alternate_keys={"invoice_number": ["Invoice #", "Invoice No."]},
)

# Dataclass schemas also work
from dataclasses import dataclass
@dataclass
class Receipt:
    merchant: str
    amount: float

parse_with_schema("receipt.pdf", schema=Receipt)

LaTeX conversion

from lexoid.api import parse_to_latex
latex_source = parse_to_latex("paper.pdf", model="gpt-4o")

URLs and recursive crawling

# Single page
parse("https://example.com")

# Crawl 2 levels deep
parse("https://example.com", depth=2)

# Render webpage → PDF first, then parse and keep the intermediate PDF
result = parse(
    "https://example.com",
    as_pdf=True,
    save_dir="output/",
    save_filename="example.pdf",
)
intermediate = result["pdf_path"]

Bounding boxes

result = parse("doc.pdf", return_bboxes=True, bbox_framework="auto")
for seg in result["segments"]:
    for text, bbox in seg.get("bboxes", []):
        # bbox = [x0, top, x1, bottom], normalized [0, 1]
        ...

Audio

Audio inputs require a Gemini model (the only provider with audio support).

result = parse("interview.mp3", model="gemini-2.5-flash")
print(result["raw"])

Token cost tracking

result = parse(
    "doc.pdf",
    model="gpt-4o",
    api_cost_mapping="tests/api_cost_mapping.json",
)
print(result["token_cost"])  # {"input": ..., "output": ..., "input-image": ..., "total": ...}

Key kwargs reference

kwarg	Purpose
model	LLM model name (default from DEFAULT_LLM, falls back to gemini-2.5-flash).
api_provider	Override inferred provider.
framework	pdfplumber / pdfminer / paddleocr for STATIC_PARSE.
temperature	LLM sampling temperature (default 0.0).
max_tokens	LLM output token limit (default 1024, 4096 for Ollama).
pages_per_split	Pages per parallel chunk.
max_processes	Parallel workers (forced to 1 when parser_type="LLM_PARSE" and api_provider="ollama").
page_nums	Specific 1-indexed pages to parse (PDFs only).
depth	Recursive URL parsing depth.
as_pdf	Convert input to PDF before parsing.
save_dir	Where to keep the intermediate PDF if as_pdf=True.
return_bboxes	Attach bounding boxes per segment.
bbox_framework	auto / pdfplumber / paddleocr.
router_priority	speed / accuracy / cost for AUTO mode.
character_threshold	Min char count for STATIC accept under cost priority.
autoselect_llm	ML-based LLM choice in AUTO mode.
retry_on_fail	Fall back to alternate parser on error (default True).
max_image_dimension	Max px to which images / page renders are downscaled.
api_cost_mapping	Dict or JSON path with per-model cost — enables token_cost in output.
system_prompt / user_prompt	Override the default LLM prompts.
verbose	Verbose logging during LLM parsing.

Things to verify before reporting success

• The result dict has non-empty raw. Empty raw with an error key means a recoverable failure occurred and Lexoid returned a stub.
• For LLM_PARSE, token_usage["total"] is non-zero — zero suggests the API call silently failed.
• For multi-page PDFs, len(result["segments"]) matches the expected page count (or len(page_nums) if used).
• For parse_with_schema, each per-page entry may be a dict or a list of dicts depending on the schema and the model. Check the type before indexing, and verify that the keys actually match the schema — the LLM can drift; pass example_schema to anchor it.

See also

• API reference: docs/api.rst.
• CLI equivalent: lexoid-cli skill.
• Example notebooks: examples/example_notebook.ipynb, examples/example_notebook_colab.ipynb.