pdf-corpus-investigator

name	pdf-corpus-investigator
description	Make sense of large PDF dumps — one massive PDF or many — by building a cached, searchable workspace with OCR, per-page index, section splits, and a standalone summary. Use when a user hands over hundreds or thousands of pages and wants to know what's in there, search across it, or hand a digest to someone else.
when_to_use	Large PDF dumps (100+ pages total) — single long PDF, a folder of many PDFs, or a ZIP archive. Works for discovery productions, FOIA releases, research archives, board packets, meeting minute archives, medical records, financial statement runs, exported archives. Anything where a human needs a map.
allowed-tools	Bash(pdfinfo *) Bash(pdftotext *) Bash(pdftoppm *) Bash(pdfseparate *) Bash(pdfunite *) Bash(tesseract *) Bash(python3 *) Bash(brew *) Bash(xargs *) Bash(find *) Bash(mkdir *) Bash(cp *) Bash(ls *) Bash(wc *) Bash(du *) Bash(open *) Bash(chmod *)
license	MIT

PDF corpus investigator

Turn a pile of PDFs (or one gigantic PDF) into a navigable, searchable, cacheable workspace. Optimized so reruns and refinements cost nothing: every heavy step writes per-artifact outputs and skips if they already exist.

The workflow deliberately includes a second investigative pass after initial classification — that's where the real value is, and most one-shot pipelines miss it.

When to use

Invoke this skill when the user describes a large document set they need to understand. Examples of triggering requests:

• "Help me make sense of this 400-page PDF a vendor sent me"
• "I have a folder of 60 board-packet PDFs from the last three years"
• "Here's a FOIA release — what's in it?"
• "Client dumped a zip of discovery on me"
• "Summarize these research papers I downloaded"

If the request is for one short PDF (under ~30 pages), skip this skill — use plain pdftotext or Read directly.

Core philosophy

1. Cache every expensive step per-artifact. OCR writes one .txt per page and skips if the file exists. Reruns cost nothing.
2. Separate expensive from cheap. Rasterization and OCR are expensive; regex classification and indexing are cheap. Iterate freely on the cheap stuff.
3. Always do a second pass. First-pass heuristic classification always misses something. Grep for domain-specific patterns (exhibit markers, Bates stamps, "REDACTED", section numbers, document titles) after initial classification and reconcile.
4. Emit a standalone summary. The final deliverable should work with only the original PDF(s) — no dependency on the generated workspace.

Workspace layout it produces

<workspace>/
├── README.md                   human-readable overview of this specific dump
├── SUMMARY.md                  standalone companion to the original PDF(s)
├── source/                     copies of the original PDF(s)
├── pages/p-NNN.png             rasterized pages at 200 DPI (grayscale)
├── text/p-NNN.txt              OCR output, one file per page
├── sections/*.pdf              the PDF split into logical chunks (built from index)
├── page_pdfs/p-N.pdf           each page as its own PDF (for pulling one out)
└── index/
    ├── pages.jsonl             one JSON record per page (classification, dates, stats)
    ├── pages.csv               same data as CSV
    ├── documents.md            block-level map (consecutive pages of same type)
    └── patterns.jsonl          second-pass finds (exhibit letters, bates, headers)

Pages are numbered in the corpus's own order. For a multi-PDF corpus, pages are numbered globally and index/pages.jsonl records which source PDF each page came from.

The workflow

Work through these steps in order. Tell the user what you're doing at each major step — they can't see your tool calls.

0 — Confirm the input and pick a workspace location

Ask (or infer) where the PDFs are and pick a workspace directory. Default convention:

<parent>/<category>-YYYY-MM-DD-<slug>/

Copy (don't move) the source PDF(s) into <workspace>/source/.

1 — Install tools if missing

Need poppler (gives pdfinfo, pdftotext, pdftoppm, pdfseparate, pdfunite) and tesseract. On macOS: brew install poppler tesseract. On Debian/Ubuntu: apt install poppler-utils tesseract-ocr.

2 — Inventory

For each source PDF, run pdfinfo and a small pdftotext probe on the first several pages:

pdftotext -f 1 -l 10 source/<file>.pdf - | wc -c

If that returns <100 bytes for 10 pages, the PDF is scanned with no text layer and you'll need OCR. If it returns a healthy amount of text, skip OCR and use pdftotext per page instead.

3 — Rasterize if scanned

${CLAUDE_SKILL_DIR}/scripts/rasterize.sh <workspace> [dpi]

Default DPI is 200 (good OCR/size balance). Grayscale PNGs are written to pages/. This is the most time-consuming step for large PDFs — expect 1–3 pages per second depending on DPI and page complexity.

4 — OCR (parallel, cached)

${CLAUDE_SKILL_DIR}/scripts/ocr.sh <workspace> [parallelism]

Default parallelism is 8. One .txt per page, skipped if already present. Safe to interrupt and resume — next run picks up where it stopped.

5 — First-pass index

python3 ${CLAUDE_SKILL_DIR}/scripts/build_index.py <workspace>

Classifies every page by heuristic: legal-filing, financial-statement, email, text-message-screenshot, calendar-screenshot, document-text, image-heavy, image-or-blank. Extracts dates, dollar amounts, times, potential Bates numbers. Writes:

• index/pages.jsonl — one record per page
• index/pages.csv — same, for spreadsheet use
• index/documents.md — block-level summary grouped by consecutive same-type runs

Reruns are ~1 second. Tweak scripts/build_index.py (copy it into the workspace if you want to customize without editing the skill) and rerun freely.

6 — Second pass: targeted pattern discovery ⚠ don't skip

Heuristic classification always misses specific structural markers. After reading the block map in index/documents.md and a few sample pages, figure out what structural markers are actually in this corpus and grep for them.

Common markers across document types:

Marker kind	Regex to try
Legal exhibits	^\s*Exhibit\s+([A-Z])(?:\.([a-z]))?\.\s*(.+)$
Bates stamps	\b[A-Z]{2,}[-_]?\d{4,7}\b
Section numbers	^\s*§\s*\d+(\.\d+)*
Meeting minutes	`^\s*(Motion
Board packet tabs	^\s*Tab\s+\d+\b
Email headers	`^(From
Research paper section	`^\s*(Abstract
Bank statement period	`(Statement Period
Medical record headers	`\bMRN:\s*\d+
Redactions	`[REDACTED]

Use:

${CLAUDE_SKILL_DIR}/scripts/find_pattern.sh <workspace> '<regex>' > index/patterns-<name>.txt

Or grep the text directly. For each pattern, produce a small table (marker → page number → context line) and persist it in index/patterns.jsonl or a named index/patterns-<name>.md.

Why this matters: the Wiley 015 case that inspired this skill had 22 lettered exhibits (A–V). First-pass classification caught 12 of them. The last exhibit (V — a critical one tying to a hospital admission) was only found after a targeted Exhibit [A-Z] grep across the OCR text. Budget time for this pass every time.

After the grep pass, reconcile with index/documents.md and identify:

• Marker sequence gaps (you found A, B, D, E — where's C?)
• Markers where the OCR captured only part of the title (wrap-around)
• Pages the classifier put in the wrong bucket

For partial OCR captures, re-OCR that specific page with a different tesseract --psm mode (try 3, 4, 6, 11 in sequence) or open the page image (pages/p-NNN.png) and read it directly.

7 — Split into section PDFs

Once you have a block-level understanding (from step 5 + 6), split the source into navigable section PDFs. Edit scripts/split_sections.sh to list your desired ranges, then:

${CLAUDE_SKILL_DIR}/scripts/split_sections.sh <workspace>

This writes sections/01-<name>.pdf, sections/02-<name>.pdf, etc. Section PDFs make the corpus shareable — someone can open just the chunk they need instead of the whole dump.

8 — Write the standalone summary

Produce <workspace>/SUMMARY.md (and, if appropriate, a separate README.md).

The SUMMARY must satisfy the standalone rule: it can only reference page numbers in the original PDF(s). No paths to sections/, text/, index/. Someone reading the SUMMARY with only the original PDF should be fully oriented.

The README can freely reference the workspace layout — it's for people working inside the workspace.

Recommended SUMMARY structure:

1. What the corpus is (source, delivery date, total pages, brief characterization)
2. Composition table (page range → contents), high level
3. Per-section narrative (one short section of prose per major block)
4. Key people/entities/identifiers referenced
5. Notable dates extracted
6. Caveats about OCR quality and heuristic classifications

9 — Present the workspace to the user

Open the workspace in Finder/Explorer, show the SUMMARY path, and state one sentence about what the corpus actually is. Don't leave the user to figure out where to look.

Handling many PDFs instead of one

If the corpus is a folder of PDFs rather than one long file, the scripts still work — they iterate over source/*.pdf. Two differences:

1. Global page numbering: the index assigns a global page number across all PDFs, and records each page's source_file and source_page in pages.jsonl. This makes it possible to search across the whole corpus but still point back to the specific file.
2. Section splitting: for a multi-file corpus, "sections" usually means grouping whole PDFs by topic/type rather than splitting individual files. Edit split_sections.sh accordingly or skip section splitting and use the original filenames.

Handling giant corpora (10k+ pages)

• Rasterize and OCR in batches using xargs with a lower parallelism to avoid swapping (-P 4 instead of -P 8).
• Consider lower DPI (150) for a first pass; re-do specific pages at 300 DPI only if OCR quality is poor.
• Commit the index directory to git inside the workspace so large corpora stay version-controlled even as you iterate on the classifier.
• For truly huge corpora, switch OCR from tesseract to a cloud OCR API — the rest of the pipeline (classification, indexing, summary) doesn't change.

Non-obvious rules

• Never delete raw artifacts until the user has seen the SUMMARY. The rasterized pages and per-page text are how you verify anything the OCR or classifier got wrong.
• The SUMMARY is the deliverable, not the workspace. The workspace is scaffolding; the SUMMARY is what the user hands to someone else.
• Flag OCR weak spots in the SUMMARY. Specifically call out: handwriting, non-Latin scripts, tiny UI text in chat screenshots, faint stamps, photo captions. Anyone quoting from the OCR should verify against the page image.
• Don't invent page ranges. If you're unsure where a section ends, say so and cite the page where the OCR last clearly identified the section.

Reference material

• reference.md — deeper notes on OCR tuning, classifier design, multi-PDF handling
• examples.md — worked example of the multi-pass discovery flow with actual regex patterns

Philosophy

The measure of this skill is whether the user can come back in six months, rerun the pipeline, tweak the classifier, and pay zero marginal cost on the heavy steps. If the cache stops working, the skill has failed.