pdf-text-extraction

name	pdf-text-extraction
description	Extract text from PDFs on macOS — tools, failure modes, and fallbacks for image-based, linearized, and xref-corrupted PDFs. Includes verification-first triage for ingestion decisions when extraction fails.
version	1.1
author	hermes
tags	["pdf","extraction","macos","text","xref-corruption","verification-first"]
category	devops

PDF Text Extraction on macOS

Trigger

Extract text from PDF files. Common scenarios:

• User attaches a PDF in chat (Feishu/Telegram/etc.) — gateway auto-lands it in /tmp/
• User downloads a PDF manually to ~/Downloads/
• Cron job pulls a PDF from a URL

Chat-attached PDF discovery (2026-06-10 verified)

Hermes gateways auto-land chat attachments to /tmp/ with original-ish filenames. When user says "我把文件发给你了" / "PDF 已发" / [Attachment: filename.pdf], the file is almost certainly in /tmp/.

Discovery order (fast → slow):

# 1. PRIMARY — /tmp/ (where Feishu/Telegram gateways write)
ls -la /tmp/*.pdf -t 2>/dev/null | head -10
ls -la /tmp/*.PDF -t 2>/dev/null | head -10
ls -la /tmp/*.{pdf,PDF} -t 2>/dev/null | head -10   # glob brace

# 2. ~/Downloads — only if user explicitly said "下载到本地了"
find ~/Downloads -maxdepth 2 -name "*.pdf" -mtime -1 2>/dev/null

# 3. DO NOT search ~/Library/Group Containers/*xinWeChat/
#    — find on this path TIMEOUTS (>30s) repeatedly. WeChat/Feishu container
#    caches are gated by TCC permissions and Hermes can't enumerate them
#    within a useful budget. The gateway already pulled the file to /tmp/,
#    so container-cache search adds zero value.

# 4. ~/Library/Caches/com.bytedance.feishu/ — often empty even when Feishu
#    cache is populated. Skip.

Why "DO NOT search container caches": The user explicitly attached the file, so the gateway ALREADY extracted it to a known path. Searching container caches is 30s+ wasted timeout.

Filename matching heuristics: gateways preserve original names but may:

• Lowercase / replace spaces with underscores (清华大学：基于 AI 的自进化组织研究报告.pdf → tsinghua_report.pdf)
• Add (1) suffix on collision
• Use random hash (a3f2b1c4.pdf) in some configs

Match by mtime -t (most recent) + fuzzy keyword, not exact name.

Tools Available

Tool	Command	Notes
textutil	textutil -convert txt -stdout file.pdf	Built-in, fastest for standard PDFs
pdftotext	pdftotext file.pdf -	Requires poppler (not on macOS by default)
PyMuPDF (pymupdf)	python3 -c "import pymupdf as fitz"	Best for complex/layout PDFs. NOT preinstalled on macOS /usr/bin/python3 (Apple CommandLineTools) — must pip3 install pymupdf first (2026-06-04 verified)
pdfminer	python3 -c "import pdfminer"	Good for structured text. Same install issue — pip3 install pdfminer.six
pypdf	python3 -c "import pypdf"	Lightweight, no native deps. Same install issue — pip3 install pypdf

The pip3 install cost is ~10 seconds on Apple CommandLineTools Python (Successfully installed pymupdf-1.26.5 in a recent session). Worth doing on demand for any non-trivial PDF — but for one-off extraction, textutil first, then pymupdf if it fails, is the right order.

Common Failure Modes

1. Linearized PDF (web-served academic papers)

Symptom: textutil exits with no error but produces empty output. textutil -info reports "Text encoding Unicode (UTF-8) isn't applicable."

Detection: head -c 100 file.pdf | xxd | grep -i linear If you see Linearized in the first 100 bytes, it's a web-optimized PDF.

Fallback: Extract text from the source webpage instead of the PDF binary. Academic paper pages (arXiv, Semantic Scholar, project pages like picrew.github.io) often have the full text in HTML which is trivially scrapable.

2. Image-based / scanned PDF

Symptom: textutil produces empty output. head -c 100 shows no text streams, only image streams. Or: PyMuPDF floods stderr with MuPDF error: format error: cannot find object in xref (N 0 R) errors and most pages return empty get_text().

Detection signals (any combination is a strong indicator):

• File size / page count ratio is very high (e.g. 14MB / 82 pages ≈ 170KB per page — typical for scanned reports)
• PyMuPDF doc.page_count reports a high number but doc[i].get_text() returns empty for almost all pages
• Only 1-2 pages near the start (cover / index) have any extractable text
• Multiple "cannot find object in xref" warnings even after pip install pymupdf
• pdfinfo reports Creator: WPS 演示 and Page size: 960 x 540 pts — strong fingerprint for WPS-PPT-exported image-only PDFs (common with Chinese research reports 2025-2026). All pages are pure images; pdftotext returns 0 chars. Confirmed with multiple 清华 reports in this vault.

Fallback options (in order of preference):

1. Find the source URL — if the PDF was downloaded from a webpage (arXiv, official report landing page, 微信公众号 article), the HTML is trivially scrapable and almost always has the full text. This is the best fallback — do this first if you have any URL context.
2. vision_analyze on SAMPLED page images — render pages 1, 4-12, 15-25, 30-45, last 5 with pdftoppm -r 150, then vision_analyze ~12-15 pages total. For 80+ page reports, sampling 15-20% of pages typically captures: cover (1), early conceptual chapters (4-12), sub-detail/case studies (15-25), design + appendix (30-45), start-up checklist + conclusion (last 5). Verified on 89-page Chinese PDF (清华循环工程, 2026-06-12): 14 sampled pages yielded complete v×c scoring + 10-concept synthesis. Do NOT sample <5 pages — risk of fabricating based on cover material.
3. PDF-to-image + OCR pipeline — pdftoppm + Tesseract or similar. High error rate, high time cost, only use when no source URL exists.
4. Report extraction failure honestly — do NOT hallucinate an entity based on 1-2 pages of cover material. Log the failure with the file path, size, page count, and exact error symptoms; ask the user for an alternative source or text-extractable version.

3. PDF with xref corruption (2026-06-04 verified — Tsinghua OpenAI FDE 14MB/82-page case)

Symptom: textutil reports "Text encoding Unicode (UTF-8) isn't applicable" (looks like a linearized/scan problem). But on pymupdf, you get spammed with MuPDF error: format error: cannot find object in xref (N 0 R) errors across many pages, and only 1-2 of N pages return any text.

Distinguishing from #1 and #2: No Linearized flag in the first 100 bytes. The xref errors are a structural PDF defect (cross-reference table is broken), not a text-layer problem. Some pages will extract text successfully (often the early ones) while most return empty.

Detection (run in this order):

import pymupdf as fitz
doc = fitz.open('file.pdf')
print('Pages:', doc.page_count)
for i in range(min(5, doc.page_count)):
    text = doc[i].get_text()
    print(f'Page {i+1}: {len(text)} chars')
# If you see the xref errors + <100 chars on most of the first 5 pages → xref corruption

Fallback (in order of preference):

1. Re-fetch the source if the PDF came from a URL (download was likely glitchy)
2. Re-upload by user if the PDF was dropped locally (file may be corrupt on disk)
3. Try qpdf --decrypt file.pdf to repair (sometimes fixes minor xref damage)
4. OCR as last resort — but for 80+ page PDFs, OCR is impractical; the right answer is to ask the user, not to OCR

DO NOT:

• Do NOT assume the PDF is "image-based" just because most pages return empty — the xref error message is the discriminator
• Do NOT proceed with hallucinated content / fabricated v×c scores — the entity becomes source of truth and will be referenced months later
• Do NOT report "extraction failed" without first running the diagnostic above (it distinguishes this from failure modes #1 and #2)

Honest reply template (verified 2026-06-04, well-received by user):

⏸️ 阻塞 — 无法处理该 PDF
- file: <filename>, <N> pages
- 障碍: <具体原因> (e.g., xref 错误 + 扫描版)
- 检测: textutil "<具体报错>" + pymupdf "<具体报错>"
- 处理: 已记入 log.md (commit blocked-<slug>)，等用户决策
请告诉我怎么继续：1. 重新上传 2. 提供原网页链接 3. 跳过 4. 坚持 OCR

4. Password-protected PDF

Symptom: textutil exits with error.

Fallback: Not applicable without password. Report extraction failure.

Quick Diagnostic

# Check PDF type
head -c 200 file.pdf | strings | grep -E "Linearized|%PDF|text" | head -5

# Try textutil (silent failure)
textutil -convert txt -stdout file.pdf > /dev/null 2>&1 && echo "OK" || echo "FAILED"

# Check for image content
grep -o "/Type /Page" file.pdf | wc -l  # page count
grep -c "Image" file.pdf  # image objects (high count = scanned)

# Size-to-page ratio smell test (scanned reports: ~100-300KB per page)
SIZE=$(stat -f%z file.pdf)
PAGES=$(python3 -c "import pymupdf; print(pymupdf.open('file.pdf').page_count)" 2>/dev/null)
[ -n "$PAGES" ] && [ "$PAGES" -gt 0 ] && echo "avg KB/page: $((SIZE / 1024 / PAGES))"

# Try PyMuPDF (catches xref errors that textutil hides)
python3 -c "import pymupdf; d=pymupdf.open('file.pdf'); print('pages:', d.page_count); [print(f'p{i}:', len(d[i].get_text())) for i in range(min(5, d.page_count))]" 2>&1 | tail -10

Honest Extraction Failure Pattern

If you've determined a PDF can't be text-extracted cleanly, do NOT:

• Build an entity based on 1-2 cover pages and call it v×c scored
• Run OCR on 80+ pages and present the result as if it were authoritative
• Commit a raw/articles/...pdf-ingest.md with fabricated body content

DO:

• Log the failure to log.md with: file path, size, page count, exact error, and the detection signal that triggered the decision
• Commit the log entry (so the user can see the attempt and decision)
• Ask the user via clarify for an alternative (text-extractable version, source URL, or skip)
• Skip the ingestion entirely until the blocker is resolved

Best Practice

1. Find the source URL first if you have one — if the PDF came from a webpage, the HTML is almost always easier to extract than the PDF binary, and is usually more complete. This is the cheapest path for any "user attached a PDF" scenario where the article originated online.
2. Try textutil next — fastest, no deps for standard PDFs.
3. On textutil failure, check linearity — if Linearized, the binary is web-optimized; try the source page.
4. If PyMuPDF floods with xref errors and most pages are empty — it's almost certainly a scanned PDF. Don't burn time on OCR for 80+ page reports; ask the user for a text-extractable version or source URL.
5. For batch PDF ingestion, prefer PyMuPDF which handles both standard and complex layouts consistently.
6. When extraction fails, STOP and report — do NOT proceed to ingest. Hallucinated entities based on title-only or first-page-only content become durable false data. Cost of asking the user: 30 seconds. Cost of fabricating: hours of cleanup later.
7. The 5-question user prompt is the standard reply when extraction fails (re-upload / original URL / skip / OCR / different file).

Sampled Vision Recipe (2026-06-12 verified)

When Mode 2 fallback #2 applies and you have no source URL + a WPS-演示 or pure-image PDF (e.g. user dropped a 清华 report PDF into ~/wiki/assets/), the sampled-vision path is the right play. Full page-by-page vision is too slow; giving up wastes the work.

Page-pick heuristic for Chinese research reports (typical structure: cover → exec summary → conceptual chapters → case studies → design/architecture → appendix → start-up checklist → conclusion):

# Render target pages at 150 DPI (good balance of OCR fidelity vs file size)
mkdir -p /tmp/<slug>_pages
pdftoppm -r 150 -f 1 -l 1 file.pdf /tmp/<slug>_pages/cover -png
pdftoppm -r 150 -f 4 -l 12 file.pdf /tmp/<slug>_pages/p -png   # early conceptual
pdftoppm -r 150 -f 15 -l 25 file.pdf /tmp/<slug>_pages/p -png  # sub-detail / cases
pdftoppm -r 150 -f 30 -l 45 file.pdf /tmp/<slug>_pages/p -png  # design + appendix
pdftoppm -r 150 -f <last5> file.pdf /tmp/<slug>_pages/p -png   # conclusion
# Total: ~25-30 PNGs, ~1-2MB each, 14-16 sampled pages

Then vision_analyze each PNG with a "请完整抄录" prompt. For 89-page reports this gives 14 sampled pages in ~3-4 min.

Quality bar:

• ≥ 8 of 14 pages return dense, structured content → safe to synthesize at v×c ≥ 49
• 5-7 of 14 pages return content → synthesize with v×c cap 35-40 (lower confidence)
• < 5 pages return content → give up, fall back to honest failure

Cost: ~14 vision calls × ~1-2s each = ~30s of LLM. Don't batch — each page has unique layout.

DO NOT sample:

• Only cover + 1-2 pages (fabrication risk)
• All from the same chapter (miss structural variety)
• Skipping the conclusion (misses author's stated takeaways)

See references/sampled-vision-page-pick.md for the full page-pick rubric + worked example.