paper-extraction

name	paper-extraction
description	Extract content from academic and scientific papers (PDF). Use when processing research papers for equations, algorithms, figures, references, or structured sections. Combine with pdf-extraction skill.

Paper Extraction

Extract structured content from academic and scientific papers — equations, algorithms, section text, references, and figures — for software implementation or analysis.

When to Use

• Extracting equations, algorithms, or pseudocode from research papers
• Pulling parameter values, experimental results, or model specifications
• Parsing references/bibliography sections
• Converting papers to structured markdown for implementation work

Prerequisite: Load the pdf-extraction skill for the underlying PDF tools and scripts.

Tool Selection

Content	Recommended Tool	Why
General text	pymupdf4llm	Handles multi-column, fastest
Math equations	Mistral OCR API	94% accuracy on LaTeX math
Complex layout	marker-pdf or MinerU	Built for academic papers
Tables/figures	pdfplumber	Best table detection

Decision Flow

1. Heavy math / equations? → Mistral OCR or marker-pdf
2. Multi-column layout? → pymupdf4llm (handles it well) or marker-pdf
3. Need tables from paper? → pdfplumber
4. Just need full text? → pymupdf4llm

Quick Start

import pymupdf4llm

# pymupdf4llm handles multi-column academic layouts well
md = pymupdf4llm.to_markdown("paper.pdf")

Common Patterns

Extract Sections

import re

def extract_sections(markdown_text):
    """Split paper markdown into named sections."""
    # Match common academic headings (numbered or not)
    pattern = r'^(?:#{1,3}\s+)?(?:\d+\.?\s+)?([A-Z][^\n]+)'
    parts = re.split(pattern, markdown_text, flags=re.MULTILINE)

    sections = {}
    for i in range(1, len(parts), 2):
        name = parts[i].strip()
        body = parts[i + 1].strip() if i + 1 < len(parts) else ""
        sections[name] = body
    return sections

Extract Equations (Mistral OCR)

Best option for LaTeX math extraction from papers:

from mistralai import Mistral
import base64

client = Mistral(api_key="your-key")

with open("paper.pdf", "rb") as f:
    b64 = base64.standard_b64encode(f.read()).decode()

response = client.ocr.process(
    model="mistral-ocr-latest",
    document={"type": "base64", "base64": b64}
)

# Equations appear as LaTeX in the markdown output
for page in response.pages:
    # Find inline ($...$) and display ($$...$$) math
    equations = re.findall(r'\$\$(.+?)\$\$', page.markdown, re.DOTALL)
    equations += re.findall(r'(?<!\$)\$(.+?)\$(?!\$)', page.markdown)
    for eq in equations:
        print(eq.strip())

Extract Algorithm / Pseudocode

def extract_algorithms(markdown_text):
    """Extract algorithm blocks from paper text."""
    import re

    algorithms = []
    # Common patterns: "Algorithm N:", code blocks, or indented pseudocode
    patterns = [
        r'(?:Algorithm\s+\d+[:.]\s*[^\n]*\n)((?:[ \t]+.+\n)+)',
        r'```(?:pseudo|algorithm)?\n(.+?)```',
    ]
    for pattern in patterns:
        for match in re.finditer(pattern, markdown_text, re.DOTALL | re.IGNORECASE):
            algorithms.append(match.group(1).strip())
    return algorithms

Parse References

def extract_references(markdown_text):
    """Extract bibliography entries from paper."""
    import re

    # Find references section
    ref_match = re.search(
        r'(?:^#{1,3}\s*)?(?:References|Bibliography)\s*\n(.*)',
        markdown_text, re.DOTALL | re.IGNORECASE | re.MULTILINE
    )
    if not ref_match:
        return []

    ref_text = ref_match.group(1)
    # Split by numbered references [1], [2], etc.
    entries = re.split(r'\n\s*\[(\d+)\]\s*', ref_text)

    references = []
    for i in range(1, len(entries), 2):
        num = entries[i]
        text = entries[i + 1].strip() if i + 1 < len(entries) else ""
        references.append({"id": int(num), "text": text})
    return references

Extract Tables from Paper

import pdfplumber
import pandas as pd

def extract_paper_tables(pdf_path):
    """Extract all tables from a paper with page context."""
    results = []
    with pdfplumber.open(pdf_path) as pdf:
        for i, page in enumerate(pdf.pages):
            tables = page.extract_tables()
            for j, table in enumerate(tables):
                if table and len(table) > 1:
                    df = pd.DataFrame(table[1:], columns=table[0])
                    # Get surrounding text for table caption
                    text = page.extract_text() or ""
                    results.append({
                        "page": i + 1,
                        "table_num": j + 1,
                        "data": df,
                        "context": text[:500],
                    })
    return results

Specialized Tools

marker-pdf

End-to-end PDF→Markdown conversion built on Surya OCR. Excellent for academic papers.

uv pip install marker-pdf
marker_single paper.pdf --output_dir ./output --output_format markdown

MinerU

Open source parser specialized in academic papers with complex multi-column layouts.

uv pip install magic-pdf

alphaxiv.org (arXiv papers)

For arXiv papers, get pre-processed LLM-friendly text instead of parsing the PDF:

https://alphaxiv.org/abs/{PAPER_ID}     # structured page
https://alphaxiv.org/html/{PAPER_ID}    # full text

Extract the paper ID from URLs like arxiv.org/abs/2401.12345 or arxiv.org/pdf/2401.12345.

Tips

• Multi-column layouts: pymupdf4llm handles these well out of the box
• Equations: Mistral OCR has 94% accuracy on LaTeX math extraction
• References: Extract text, then parse with regex for citation data
• For heavy math, consider marker-pdf or MinerU which specialize in academic layout
• arXiv papers: Try alphaxiv.org first — avoids PDF parsing entirely
• Figure captions: Usually extractable as text below/above image regions
• Supplementary material: Often in separate PDFs — process each independently