pdf-to-markdown-multisource

name

pdf-to-markdown-multisource

description

Convert scientific, academic, scanned, two-column, and math-heavy PDFs into provenance-rich Markdown with CUDA Docling, PyMuPDF4LLM, native geometry, GROBID, MinerU, span-preserving PP-StructureV3 region rescue, and benchmark-gated VLM evidence. Use when extraction quality matters, especially for interleaved columns, glued words, headers/footers, watermark fragments, broken section order, empty grids, missing sections, or corrupt equations, tables, figures, and captions. Treat every automated output as a quality-gated draft that requires direct native-PDF inspection and manual Markdown postprocessing before final use.

Multisource PDF to Markdown

Contract

Produce faithful Markdown, not a summary. Preserve the paper's wording, page order, sections, equations, tables, figures, captions, references, and appendices. Repair only extraction errors that are supported by page-local geometry or an independent extractor.

The automated output is never the final reviewed transcription, including when the process exits 0. An agent must open the original PDF with its native PDF reading/rendering capabilities, compare it directly with the Markdown, and manually postprocess the Markdown before delivering it or replacing a previously reviewed extraction. Automated gates are triage aids, not proof of semantic or visual fidelity.

The skill is code-self-contained. All durable code, environment manifests, configuration templates, tests, fixtures, benchmark decisions, and documentation live below this directory. It must not import a sibling skill or repository module. Its only runtime dependencies outside the directory are the micromamba executable and environment prefixes declared in config/runtime.local.json (or the file named by PDF_MD_RUNTIME_CONFIG), plus the caller's input and output paths. Model caches are forced below their owning environment prefixes.

Use the configured core prefix as the orchestrator and CUDA environment. Do not collapse the stack into one environment: mutually incompatible Torch, Transformers, Paddle, Java, and model stacks remain in the isolated prefixes documented in references/RUNTIMES.md. If runtime configuration is absent, run scripts/bootstrap_environments.py from this skill before conversion.

Default conversion

Run the best profile unless the user explicitly requests a cheaper diagnostic pass:

SKILL_ROOT=/absolute/path/to/pdf-to-markdown-multisource

python3 "$SKILL_ROOT/scripts/run_converter.py" \
  input.pdf \
  --output output.md \
  --profile best \
  --keep-sources

Do not pass --overwrite unless replacing the named output is intended. Do not pass --allow-quality-warnings for an acceptance run; that flag is only for retaining exploratory output when a material gate is known to fail.

CUDA is the default and required workstation path. The converter preflights core PyTorch, records the RTX/CUDA device and peak allocation, and fails rather than silently sending Docling to CPU. Use --docling-device cpu only when the user explicitly requests CPU execution.

Required workflow

1. On first use, after an environment change, or when CUDA/OCR resolution is in doubt, run the environment verifier. If it fails, repair only the failed configured runtime with the bootstrap command in references/RUNTIMES.md, then rerun verification.
2. Run the default conversion without relaxing quality gates. Keep source sidecars for difficult or important papers.
3. Read the JSON report, not only the process exit code. Require blocking_gate_count == 0, no syntax errors, no material warnings, and successful core sources. Inspect escalation, page_decisions, material_element_actions, and the source/output censuses.
4. Open the original PDF with native PDF reading/rendering capabilities and compare it page by page with the Markdown. For every page, verify reading order, section placement, paragraph continuity, and the absence of furniture, watermark, reviewer-mark, and duplicate fragments. Inspect every equation, table, figure, caption, section boundary, repaired region, and local image target against the rendered source page. Source greps, extractor sidecars, and censuses may guide this review but cannot replace direct PDF inspection.
5. Manually postprocess the Markdown using only evidence visible in the original PDF or independently corroborated page-local evidence. Fix ordering, word boundaries, headings, duplicate blocks, table structure, equations, captions, figure placement, and residual pollution. Preserve page markers and the paper's wording. Never infer or rewrite scientific content merely to make it look plausible.
6. Re-read each edited region beside the source PDF and validate the completed Markdown for broken links, missing assets, malformed HTML/LaTeX, empty grids, duplicate material, and section-order regressions. Record which pages or material elements were manually changed and any unresolved uncertainty in the task handoff.
7. If a gate or manually discovered defect remains, retain the output and diagnostics, identify the page/element, and use the narrowest relevant rescue. If faithful repair is not possible, preserve an exact captioned crop where appropriate and report the limitation. Never invent a missing section, cell, equation, figure, or caption.

Exit status 0 means the requested automated policy completed and all hard gates passed; it does not waive native-PDF review or certify that the Markdown is final. Status 1 means input, runtime, configuration, or conversion failure. Status 2 means diagnostics and output were written but one or more material quality gates failed.

Known limitations requiring manual review

Assume that any of the following can remain even when all automated gates pass:

• Global element counts can miss a page-local substitution, swapped caption, duplicate element, wrong page order, or missing item offset by an extra item elsewhere. A gate can also be a false positive when a valid heading or caption uses unexpected punctuation.
• Two-column order, full-width transitions, blank-page gaps, appendices, and blocks inserted from a secondary engine can still be shifted, interleaved, or duplicated.
• Complex tables can preserve the visible numbers while flattening or corrupting hierarchical headers, row groups, merged cells, units, superscripts, or metric-to-value alignment. A plausible-looking table is not necessarily a correct table.
• Formula enrichment or OCR can lose subscripts, superscripts, accents, operators, delimiters, matrices, equation numbers, or neighboring prose. An equation crop preserves appearance but is not a complete searchable transcription.
• Figure export can omit panels, change a montage's arrangement, pair a caption with the wrong asset, or misclassify reviewer ink, watermarks, decorative marks, and table regions as figures.
• OCR and reconciliation can leave glued or split words, incorrect dehyphenation, Unicode substitutions, repeated headers/footers, watermark fragments, and subtle prose omissions that lexical checks do not recognize.
• Heading consensus can miss a core section or accept a similarly named but incorrect block. References, acknowledgements, limitations, supplements, and appendices require explicit source-PDF confirmation.
• MinerU, PP-StructureV3, and VLM evidence can be incomplete or internally inconsistent. Whole-document VLM runs can be slow, buffer output without a usable result contract, exhaust memory, or time out; prefer page- or region-scoped rescue and inspect its result before adoption.
• Docling remains canonical, but reconciliation and enrichment mean a new run is not guaranteed to reproduce a previously correct extraction byte-for-byte or semantically. Compare against the reviewed Markdown before replacing it and retain the better version element by element.

Only call an extraction final after the automated gates pass, the original PDF has been inspected directly, the Markdown has been manually corrected, and every material uncertainty has either been resolved or explicitly reported.

Target pipeline

The best profile uses this ordered pipeline. Docling remains the canonical structure and content source throughout; a specialist may repair one failed material region, but it never becomes the document skeleton.

1. Validate the input PDF, output paths, core environment identity, Tesseract data, and CUDA access.
2. Extract native page geometry, glyph direction/color/size, text, images, vector regions, and candidate tables. Detect repeated page-edge furniture and defensible large/light/rotated watermarks.
3. Run CUDA Docling as the structured primary source with table structure, real figure export, torch RapidOCR, page provenance, and formula enrichment. Reject enriched formulas that lose native scripts, leak prose, repeat implausibly, or contain structurally bad LaTeX; retain native formula text or a crop fallback.
4. Run layout-enabled PyMuPDF4LLM with selective Tesseract OCR and page chunks. Run Poppler pdftotext -layout and pdfplumber as independent spacing, order, geometry, and table evidence.
5. Run the local GROBID service as a TEI section/reference oracle. GROBID may confirm missing academic structure, but it is never the page-layout authority.
6. Reconcile page by page and block by block. Preserve explicit engine page numbers, including empty-page gaps. Use geometry bands for two-column order, but treat native geometry only as evidence rather than insertable material. Align equivalent blocks, filter confirmed furniture and structured uncaptioned non-figures, repair boundaries only with independent evidence, insert only material blocks actually found by another semantic source, and suppress duplicate formula/image representations.
7. Recover tables deterministically from valid grids and reject empty decorative grids. Collect unresolved confirmed table and corrupt-formula crops, then run one cached isolated PP-StructureV3 batch on gpu:0. Preserve its raw table HTML, cell boxes, OCR tokens, formula candidates, and JSON. Adopt a table only when an independent source confirms its complete numeric/lexical content and independent or OCR geometry confirms its row/cell structure; reject missing values and ambiguous multi-metric cells. Preserve rowspan/colspan HTML verbatim. Adopt LaTeX only when its coefficients and symbol structure agree with page-local evidence and formula sanity passes. Otherwise retain the exact captioned crop.
8. Build the initial source/output element census and hard quality gates.
9. Run isolated CUDA MinerU in best mode and normalize its content-list sidecar into page-aligned evidence. Drop its header/footer/aside blocks, corrupt equations, and empty grids before reconciliation.
10. If hard gates remain, compare the benchmark-selected VLM. auto currently selects olmOCR; it may supply page-local evidence but cannot overwrite a clean born-digital block merely because its prose looks nicer. Lexical suspicions remain reportable warnings and do not justify a whole-document VLM run by themselves.
11. Run Mathpix or Mistral only when the caller explicitly selected a provider, explicitly authorized upload, supplied credentials, and local gates still need rescue.

Failure-mode policy

Failure	Required behavior
Two-column interleaving	Order positioned blocks within full-width vertical bands: left column top-to-bottom, then right; retain explicit page markers.
Glued words	Insert a boundary only when another page-local source shows it. WordNinja/Wordfreq can flag a warning but cannot authorize a repair or a whole-document VLM run by itself.
Headers and footers	Remove repeated normalized blocks only when they occur at page edges or another source labels them furniture.
Watermark fragments	Remove large, light, rotated native text or independently confirmed furniture; never delete a body block merely because it contains words such as “draft.”
Broken/missing sections	Compare the union of Docling, GROBID, and layout headings; geometry-reorder or insert a matching source block, otherwise fail the section gate.
Empty/corrupt table grids	Reject them as tables. Keep a nearby caption as text, then use a valid native/Docling matrix, consensus-gated PP-StructureV3 HTML, or an exact crop fallback. Never flatten accepted cell spans into a pipe table.
Corrupt equation	Reject bad enrichment; use clean native formula evidence, consensus-gated PP-StructureV3 LaTeX, or an equation crop plus native text. Never synthesize LaTeX from context.
Missing/corrupt figure	Export the real structured asset or crop confirmed geometry; keep a searchable caption and require a resolvable image target.
Duplicate equation/figure	Align across representation types and geometry, prefer the selected structured element, and prune unreferenced generated assets.
OCR-only page	Use CUDA torch RapidOCR/selective Tesseract first, MinerU evidence next, and the benchmark-approved VLM only when gates require it.

See references/QUALITY_MODEL.md for exact algorithms, gates, and deliberate limits.

Profiles and escalation controls

• --profile best is the quality-first default. It enables GROBID, MinerU, PP-StructureV3 material-region rescue, benchmark-gated VLM comparison, and optionally authorized cloud rescue.
• --profile native-only runs the local deterministic/native stack and disables isolated enrichers. Use it for fast diagnosis, not the final version of a difficult paper.
• --vlm auto uses benchmarks/selection.json only if the four-engine benchmark is complete and the selected engine passed quality, license, and observed-GPU gates.
• --vlm never disables VLM comparison. Explicit marker or chandra additionally requires --allow-restricted-models.
• --paddleocr auto retains its backward-compatible CLI name but invokes one PP-StructureV3 batch only for unresolved confirmed table/formula crops. It never parses whole-page prose, and identical crops reuse the recorded result instead of reloading the models.
• --no-grobid, --no-mineru, --no-docling, --no-pymupdf4llm, and --no-formula-enrichment are diagnostic switches. Do not use them to hide a broken runtime in a final conversion.
• --max-pages N is a bounded smoke-test option; the report warns that the document was truncated.

VLM benchmark policy

The bundled adversarial fixture contains two columns, deliberately separated words, repeated furniture, a rotated watermark, a populated table, an empty decorative grid, a formula, a figure/caption, ordered core sections, and an OCR-only page.

The current RTX 3090 benchmark attempted Marker, Chandra OCR 2, olmOCR, and Infinity-Parser2. Only olmOCR scored 1.00, used CUDA, retained order/math/table/caption evidence, and emitted neither forbidden glue nor DRAFT COPY; it is the automatic selection. Marker, Chandra, and Infinity each scored 0.94 but retained the watermark. Marker and Chandra are also explicit-only because of code/model license terms. Never change automatic selection by reputation or a single anecdotal PDF; rerun the complete fixture benchmark.

SKILL_ROOT=/absolute/path/to/pdf-to-markdown-multisource

python3 "$SKILL_ROOT/scripts/run_in_core.py" \
  "$SKILL_ROOT/scripts/benchmark_vlm_engines.py" \
  --output-dir "$SKILL_ROOT/benchmarks/latest" \
  --fixture "$SKILL_ROOT/benchmarks/latest/adversarial-academic.pdf" \
  --resume \
  --acknowledge-restricted-models \
  --write-selection

Use --resume only when the fixture path matches and successful engine records remain valid. The selection writer still requires all four engines.

Cloud providers are strict opt-in

Never upload by default. Cloud execution needs --profile best, a named provider, --allow-cloud-upload, unresolved local quality, and credentials:

# Mathpix: MATHPIX_APP_ID and MATHPIX_APP_KEY
python3 "$SKILL_ROOT/scripts/run_converter.py" input.pdf \
  --output output.md --cloud mathpix --allow-cloud-upload

# Mistral: MISTRAL_API_KEY
python3 "$SKILL_ROOT/scripts/run_converter.py" input.pdf \
  --output output.md --cloud mistral --allow-cloud-upload

Mathpix requests improve_mathpix=false, page-aligned line geometry, and explicit job deletion. Mistral disables base64 images, separates headers/footers, and explicitly deletes the uploaded file. Both adapters redact credentials, signed URLs, and base64 material from diagnostics. Deletion is attempted in finally and recorded; it is not a guarantee about provider backups or legal retention. Check provider policy before uploading confidential, personal, unpublished, or licensed material.

Output contract

For output.md, the converter writes:

• output.md: faithful Markdown with <!-- page: N --> provenance markers;
• output.md.report.json unless --report overrides it: runtime/GPU state, source status, hashes, element censuses, gates, escalation, decisions, warnings, and material actions;
• output.md.provenance.json unless overridden: every selected block with page, source, box, evidence, and artifact metadata;
• output_assets/ unless overridden: only local assets referenced by final Markdown;
• output.md.sources/ when --keep-sources is set, or the explicit --debug-dir: extractor outputs and engine contracts.

PP-StructureV3 evidence lives below the source/debug tree as ppstructure/ppstructure-manifest.json, ppstructure/engine-result.json, ppstructure/ppstructure-regions.json, and one raw JSON file per crop. Accepted HTML/LaTeX and rejected candidates both retain their consensus record in block provenance and material_element_actions.

Automated hard acceptance requires page coverage, confirmed core-section coverage/order, at least 80% confirmed body-token retention, structured or image-preserved confirmed tables, resolvable confirmed figures, searchable captions, present and sane formulas, nonempty tables, valid image targets, and clean text integrity. lexical-glue-suspicion remains a review warning because scientific compounds cannot be exhaustively enumerated. These checks authorize the mandatory manual review stage; they do not replace it or establish final acceptance by themselves.

Runtime maintenance and research rationale

Use references/RUNTIMES.md for installation, verification, cache ownership, and GROBID lifecycle. Use references/ENGINE_SURVEY.md for the official-source package survey, selection rationale, license boundaries, and alternatives intentionally not placed in the automatic path. Use references/PPSTRUCTUREV3_INTEGRATION.md for the region contract, adoption gates, and local challenge evidence.