unlimited-ocr-parse-document

name	unlimited-ocr-parse-document
description	Parse an image, PDF, or directory into markdown with LaTeX formulas, HTML/markdown tables, and per-block layout bounding boxes, using baidu/Unlimited-OCR locally — free, offline, no API key, no upload quota. Runs on Apple Silicon via MLX (~2.4 s/image, ~5 GB) or an NVIDIA GPU via transformers. Use when transcribing scanned or image-based documents, extracting equations from a paper, converting a PDF to markdown, OCR-ing screenshots, or replacing a paid vision-API OCR call with a local one. TRIGGERS - unlimited-ocr, ocr this, ocr a pdf, extract text from image, transcribe document, pdf to markdown, document parsing, extract formulas, extract equations, latex from image, local ocr, offline ocr, scanned pdf, image to text, layout parsing, bounding boxes from document.
allowed-tools	Bash, Read, Write, Glob

Unlimited-OCR — parse a document locally

baidu/Unlimited-OCR (MIT, arXiv 2606.23050) is a 3B-total / 500M-activated MoE document parser. It reads a page image and returns markdown with LaTeX mathematics, tables, and a <|det|> bounding box for every block it found.

Self-Evolving Skill: improves through use. Every number below was MEASURED on this hardware, not copied from a README — provenance in ../../references/EMPIRICAL.md. If a measurement drifts, a flag stops working, or a new failure mode appears, fix this file and the references immediately rather than deferring. Re-verify before changing a documented fact.

---

The one rule: Free OCR. is the only prompt mode that works

Of the three prompt modes the upstream project documents, two are broken on MLX and both are refused by this CLI with exit 2 and a printed reason:

Mode	Measured behaviour
free-ocr	works — correct output with layout boxes in 2.4 s. The default.
document-parsing	decodes parsing.parsing.parsing… until max_tokens. This is the prompt the upstream README documents.
multi-page	deterministically hallucinates the word industrydocuments onto a single image.

--allow-withheld-prompt-mode forces one anyway. They are kept nameable rather than deleted so the failures stay reproducible.

This CLI sends one image per forward pass, always. The model's headline multi-page capability is deliberately unused — a five-page single-pass run dropped a page with no error, which is the worst failure mode there is for an archive. Per-page is slower and verifiable.

---

Use it

S=~/eon/cc-skills/plugins/unlimited-ocr/scripts/unlimited_ocr.py

# Which backends does this machine actually have?
uv run --no-project $S doctor

# One image -> markdown on stdout
uv run --no-project $S parse --input page.png

# A PDF -> one .md per page, plus a JSON report, rendered at 300 DPI
uv run --no-project $S parse --input paper.pdf --output ./out

# A directory of images, machine-readable, with math repaired for cross-model comparison
uv run --no-project $S parse --input ./images --format json --collapse-math-spacing --quiet

# Just the prose: strip the layout markers
uv run --no-project $S parse --input page.png --strip-det

# The CLI contract itself, as JSON (the SSoT an agent should introspect)
uv run --no-project $S spec

uv run reads the script's PEP 723 header and materialises its own dependencies. There is nothing to install and no virtualenv to manage.

---

What it returns

<|det|>equation [249, 82, 699, 202]<|/det|>\[
c u r p d f = t r i a n g (l o w, h i g h, v o l)
\]

• Bounding boxes are [x0, y0, x1, y1] normalised to 0–1000 on BOTH axes, independent of the real aspect ratio. Verified: a box of y1=909 on a 211-pixel-tall image. The CLI also emits pixel coordinates in --format json.
• Categories observed so far: equation, title, text, chart, header, page_footnote, page_number, image. The taxonomy is not documented upstream; treat any new value as valid.
• Mathematics comes back character-spaced — c u r p d f, not curpdf. In LaTeX math mode those render identically, so it is not wrong, but it is not byte-identical to any other model's output either. Pass --collapse-math-spacing to rejoin it. This matters enormously if you are comparing transcriptions across models: without it, every formula scores as a disagreement.

---

Choosing a backend

--backend auto (default) prefers MLX, then CUDA.

	Apple Silicon (MLX)	NVIDIA (transformers)
Model	mlx-community/Unlimited-OCR-mxfp8 (3.66 GB)	baidu/Unlimited-OCR (6.67 GB bf16)
Measured on	M3 Max, 36 GB	RTX 4090, 24 GB, Ada SM 8.9
Single image	2.4 s, 5.2 GB peak	3.7 s, 8.3 GB peak VRAM
Multi-page	per page, one call each	1.5 s/page synthetic, 11 s/page dense academic
Determinism	greedy, T=0	byte-identical on re-run

Do not follow the upstream SGLang instructions on an Ada GPU. They specify --attention-backend fa3; FlashAttention-3 requires Hopper (SM 9.0). The transformers path needs no backend flag and is what these numbers come from. doctor reports compute capability and says so explicitly.

---

What it will NOT do

It does not describe charts. It localises them and transcribes nothing inside. On a nine-panel matplotlib figure it emitted nine perfectly-placed chart boxes and zero characters — not even the panel titles, which were plainly legible. That is correct behaviour for a layout parser and a nasty surprise if you expected an image captioner.

If you need the content of a chart, use unlimited-ocr-segment-figure to crop each panel and hand the crops to a model that does describe images.

The model writes tables as HTML <table>, never as pipe-markdown. Measured over 103 real tables: 88 came back as <table> markup and 15 as prose; none as pipe-markdown. The CLI converts them for you — --table-format defaults to pipe; pass --table-format html to keep the model's own serialization. Pipe-markdown is suitable for concatenating into markdown documents and for comparing against other readers (M3, GLM-4.6v) which emit pipe-markdown natively.

Other limits, all with evidence, in ../../references/PITFALLS.md: degenerate repetition, multi-page page-dropping, and why "Unlimited" is not unlimited.

---

Exit codes

Code	Meaning
0	parsed; no degenerate repetition detected
1	parsed, but repetition suspected in at least one output — inspect before trusting it
2	usage or input error, including the refused prompt mode

Exit 1 is not a crash and not a success. The CLI detects loops structurally (a short chunk repeating in the tail), so a NEW loop is caught, not just the known one.

---

Post-Execution Reflection

After this skill completes, check before closing:

1. Did it exit 1? — repetition was detected. Record which prompt mode and image type triggered it in references/PITFALLS.md § 3; the measured base rate is ~4 % on tables.
2. Did a prompt mode behave differently than documented? — the two withheld modes are withheld on measured evidence. If one now works, update the withholding reason and its test, not just the doc.
3. Did the output need reformatting downstream? — tables come back as HTML. If you wrote a converter, it belongs in this plugin rather than in the calling project.