pdf

> **Important:** All `scripts/` paths are relative to this skill directory. > Use `run_skill_script` tool to execute scripts, or run with: `cd {this_skill_dir} && python scripts/...` # PDF Processing Guide ## Prerequisites - **pypdf**: core PDF reading and writing - **pdfplumber**: text and table extraction - **reportlab**: PDF creation - **pdftotext** (poppler-utils): command-line text extraction - **pdftoppm** (poppler-utils): PDF-to-image conversion - **qpdf**: PDF manipulation (merge, split, rotate, decrypt) ## Tool Selection Decision Table Choose the right approach before starting: | Input | Condition | Recommended Tool | |-------|-----------|-----------------| | URL | PDF accessible via URL | `web_extract(url)` — fastest, no download needed | | Local file | Text-native PDF (generated by software) | `pymupdf` — ~25 MB install, instant extraction | | Local file | Scanned/image-only PDF (no selectable text) | `marker-pdf` — OCR with layout preservation (~5 GB, needs GPU or CPU) | | Local file | Form filling or page manipulation | `pypdf` / `pdfplumber` + form scripts | | Local file | NLP editing or semantic search | `nano-pdf` — sentence-level operations | **URL-first rule**: If the user provides a URL, always try URL extraction first before downloading. ## Overview This guide covers essential PDF processing operations using Python libraries and command-line tools. ## URL-First Extraction If the user provides a URL pointing to a PDF, extract it without downloading: ``` web_extract(url="https://example.com/report.pdf") ``` Fall back to download + local processing only if `web_extract` returns empty or errors. --- ## Fast Extraction: pymupdf (fitz) **Best for**: Text-native PDFs (digital, not scanned). Install: `pip install pymupdf` (~25 MB). ```python import fitz # pymupdf doc = fitz.open("document.pdf") print(f"Pages: {doc.page_count}") # Extract all text (fast) full_text = "\n".join(page.get_text() for page in doc) # Extract with layout blocks (tables, columns) for page in doc: blocks = page.get_text("blocks") # (x0,y0,x1,y1,text,block_no,block_type) for block in blocks: print(block[4]) # text content # Extract images for page in doc: for img in page.get_images(): xref = img[0] base = doc.extract_image(xref) with open(f"img_{xref}.{base['ext']}", "wb") as f: f.write(base["image"]) ``` pymupdf is 5-10× faster than pypdf for text extraction and preserves layout better. --- ## OCR Extraction: marker-pdf **Best for**: Scanned PDFs, image-only PDFs, or documents where `pymupdf` returns garbled text. Install: `pip install marker-pdf` (~5 GB with models). ```bash # Single file marker_single document.pdf output_dir/ --batch_multiplier 2 # Batch marker input_dir/ output_dir/ --workers 4 ``` Outputs Markdown with preserved headings, tables, and code blocks. **Decision signal**: Run `pymupdf` first. If extracted text has <50% printable characters or looks like garbage, switch to `marker-pdf`. --- ## NLP Editing: nano-pdf **Best for**: Semantic search, sentence-level edits, keyword replacement in text-native PDFs. Install: `pip install nano-pdf`. ```python from nano_pdf import NanoPDF doc = NanoPDF("document.pdf") # Search sentences results = doc.search("termination clause", top_k=5) for r in results: print(r.page, r.text, r.score) # Replace text (produces new PDF) doc.replace("old phrase", "new phrase", output="modified.pdf") ``` --- ## Python Libraries ### pypdf - Basic Operations #### Merge PDFs ```python from pypdf import PdfWriter, PdfReader writer = PdfWriter() for pdf_file in ["doc1.pdf", "doc2.pdf", "doc3.pdf"]: reader = PdfReader(pdf_file) for page in reader.pages: writer.add_page(page) with open("merged.pdf", "wb") as output: writer.write(output) ``` #### Split PDF ```python reader = PdfReader("input.pdf") for i, page in enumerate(reader.pages): writer = PdfWriter() writer.add_page(page) with open(f"page_{i+1}.pdf", "wb") as output: writer.write(output) ``` #### Extract Metadata ```python reader = PdfReader("document.pdf") meta = reader.metadata print(f"Title: {meta.title}") print(f"Author: {meta.author}") ``` #### Rotate Pages ```python reader = PdfReader("input.pdf") writer = PdfWriter() page = reader.pages[0] page.rotate(90) # Rotate 90 degrees clockwise writer.add_page(page) with open("rotated.pdf", "wb") as output: writer.write(output) ``` ### pdfplumber - Text and Table Extraction #### Extract Text with Layout ```python import pdfplumber with pdfplumber.open("document.pdf") as pdf: for page in pdf.pages: text = page.extract_text() print(text) ``` #### Extract Tables ```python with pdfplumber.open("document.pdf") as pdf: for i, page in enumerate(pdf.pages): tables = page.extract_tables() for j, table in enumerate(tables): print(f"Table {j+1} on page {i+1}:") for row in table: print(row) ``` #### Advanced Table Extraction ```python import pandas as pd with pdfplumber.open("document.pdf") as pdf: all_tables = [] for page in pdf.pages: tables = page.extract_tables() for table in tables: if table: df = pd.DataFrame(table[1:], columns=table[0]) all_tables.append(df) if all_tables: combined_df = pd.concat(all_tables, ignore_index=True) combined_df.to_excel("extracted_tables.xlsx", index=False) ``` ### reportlab - Create PDFs #### Basic PDF Creation ```python from reportlab.lib.pagesizes import letter from reportlab.pdfgen import canvas c = canvas.Canvas("hello.pdf", pagesize=letter) width, height = letter c.drawString(100, height - 100, "Hello World!") c.line(100, height - 140, 400, height - 140) c.save() ``` #### Subscripts and Superscripts **IMPORTANT**: Never use Unicode subscript/superscript characters in ReportLab PDFs. The built-in fonts do not include these glyphs, causing them to render as solid black boxes. Use ReportLab's XML markup tags instead: ```python from reportlab.platypus import Paragraph from reportlab.lib.styles import getSampleStyleSheet styles = getSampleStyleSheet() chemical = Paragraph("H<sub>2</sub>O", styles['Normal']) squared = Paragraph("x<super>2</super> + y<super>2</super>", styles['Normal']) ``` ## PDF Form Processing ### Check if PDF has fillable fields ```bash python scripts/check_fillable_fields.py document.pdf ``` ### Extract form field info ```bash python scripts/extract_form_field_info.py document.pdf ``` ### Extract form structure (non-fillable PDFs) ```bash python scripts/extract_form_structure.py document.pdf ``` ### Fill form fields ```bash python scripts/fill_fillable_fields.py document.pdf output.pdf --fields '{"field_name": "value"}' ``` ### Fill with annotations (non-fillable PDFs) ```bash python scripts/fill_pdf_form_with_annotations.py document.pdf output.pdf --data '{"x,y": "text"}' ``` ### Validate bounding boxes ```bash python scripts/check_bounding_boxes.py document.pdf ``` ### Convert PDF to images ```bash python scripts/convert_pdf_to_images.py document.pdf output_dir/ --dpi 150 ``` ### Create validation image with overlays ```bash python scripts/create_validation_image.py document.pdf output.png ``` ## Command-Line Tools ### pdftotext (poppler-utils) ```bash pdftotext input.pdf output.txt # Extract text pdftotext -layout input.pdf output.txt # Preserve layout pdftotext -f 1 -l 5 input.pdf output.txt # Pages 1-5 ``` ### qpdf ```bash qpdf --empty --pages file1.pdf file2.pdf -- merged.pdf # Merge qpdf input.pdf --pages . 1-5 -- pages1-5.pdf # Split qpdf input.pdf output.pdf --rotate=+90:1 # Rotate qpdf --password=mypassword --decrypt encrypted.pdf out.pdf # Decrypt ``` ## Common Tasks ### Extract Text from Scanned PDFs (OCR) ```python import pytesseract from pdf2image import convert_from_path images = convert_from_path('scanned.pdf') text = "" for i, image in enumerate(images): text += f"Page {i+1}:\n" text += pytesseract.image_to_string(image) text += "\n\n" ``` ### Add Watermark ```python from pypdf import PdfReader, PdfWriter watermark = PdfReader("watermark.pdf").pages[0] reader = PdfReader("document.pdf") writer = PdfWriter() for page in reader.pages: page.merge_page(watermark) writer.add_page(page) with open("watermarked.pdf", "wb") as output: writer.write(output) ``` ### Password Protection ```python from pypdf import PdfReader, PdfWriter reader = PdfReader("input.pdf") writer = PdfWriter() for page in reader.pages: writer.add_page(page) writer.encrypt("userpassword", "ownerpassword") with open("encrypted.pdf", "wb") as output: writer.write(output) ``` ## Quick Reference | Task | Best Tool | Command/Code | |------|-----------|--------------| | URL → text | web_extract | `web_extract(url=...)` | | Fast text extraction | pymupdf | `fitz.open(...).get_text()` | | Scanned / OCR | marker-pdf | `marker_single doc.pdf out/` | | Semantic search/edit | nano-pdf | `NanoPDF(...).search(...)` | | Merge PDFs | pypdf | `writer.add_page(page)` | | Split PDFs | pypdf | One page per file | | Extract text (layout) | pdfplumber | `page.extract_text()` | | Extract tables | pdfplumber | `page.extract_tables()` | | Create PDFs | reportlab | Canvas or Platypus | | Fill forms | scripts | `fill_fillable_fields.py` |