book-structurer

name

book-structurer

description

Process raw-extracted JSON files from PDF book extractions into clean, well-structured markdown. Use this skill whenever a user uploads a raw-extracted.json (or similar JSON with metadata and chunks arrays) from a PDF text extraction pipeline, or when they mention cleaning up, formatting, structuring, or organizing extracted book text. Also trigger when the user mentions OCR output, pypdf extraction, book chapters, or converting messy PDF extractions into readable markdown. This skill handles books with varying extraction quality — from clean text to garbled OCR — and adapts its approach accordingly.

Book Structurer

Transform raw PDF-extracted JSON into clean, structured markdown books.

Overview

This skill processes JSON files produced by PDF text extraction pipelines (pypdf, OCR tools like glm-ocr, etc.) and converts them into well-organized markdown. The extraction quality varies wildly — from pristine text to completely empty chunks to garbled OCR — so the skill adapts its approach based on what it finds.

Input Format

The expected input is a JSON file with this structure:

{
  "metadata": {
    "title": "Book title...",
    "original_file": "source.pdf",
    "total_chunks": 57,
    "processed_at": "2026-02-25T18:52:37.129Z"
  },
  "chunks": [
    "chunk of extracted text with -- N of M -- page markers...",
    "another chunk..."
  ]
}

A companion metadata.json may also be present but the raw JSON is the primary source.

Workflow

Follow these stages in order. Each stage produces output that feeds the next.

Stage 1: Triage — Assess Extraction Quality

Before doing anything else, run the triage script to understand what you're working with.

python3 /path/to/skill/scripts/triage.py /path/to/raw-extracted.json

The script outputs a quality report. Based on the results, take one of three paths:

GOOD (>80% chunks have content, avg content length >500 chars): Proceed to Stage 2. The text is usable and structure detection should work well.

PARTIAL (30-80% chunks have content, or avg content length 100-500 chars): Proceed to Stage 2 but warn the user that the extraction has gaps. Some chapters or sections may be incomplete. Flag which page ranges are missing.

UNUSABLE (<30% chunks have content, or avg content length <100 chars): Stop and tell the user the extraction failed — the PDF likely needs re-processing with a better OCR tool. Suggest they re-extract with a vision-capable model or dedicated OCR software. Do not attempt to structure empty text.

Stage 2: Clean and Concatenate

Run the cleaning script to strip extraction artifacts and produce a single clean text file:

python /path/to/skill/scripts/clean.py /path/to/raw-extracted.json ./cleaned.txt

(Write cleaned.txt to a writable working directory — /home/claude/ on claude.ai containers, the project or temp directory when running locally.)

This script:

• Joins all chunks into continuous text
• Removes page markers (-- N of M --)
• Removes page headers/footers (e.g., RELAXATION | 7, 264 | 112 ACTING GAMES)
• Normalizes whitespace (collapses excessive newlines, fixes tab-broken words)
• Preserves paragraph breaks

After running, review the first ~100 lines of cleaned.txt to confirm it looks reasonable.

Stage 3: Detect Structure

This is where LLM analysis is needed. The structure of books varies enormously, so rather than assuming a pattern, detect what pattern this specific book uses.

Extract a sample for analysis. Take the first ~3000 words and last ~1000 words of the cleaned text. This typically covers the front matter, first chapter or two, and back matter — enough to identify the structural pattern.

Analyze the sample and identify:

1. Front matter boundary — Where does the actual content begin? (After title page, copyright, dedication, table of contents, introduction, etc.)
2. Chapter pattern — How are chapters marked? Common patterns:
   • CHAPTER N followed by a title on the next line
   • Chapter N: Title on one line
   • Just a title in ALL CAPS with no number
   • Numbered sections like 1. SECTION TITLE
   • Part/Chapter hierarchy
3. Sub-section pattern — Are there numbered items, named sections, or other divisions within chapters?
4. Back matter boundary — Where does the content end? (Before appendices, bibliography, index, about the author, etc.)
5. Recurring artifacts — Page headers/footers that survived cleaning, OCR junk patterns, etc.

Output the detected structure as JSON:

{
  "front_matter_end": "text snippet or line number where content begins",
  "chapter_pattern": {
    "regex": "pattern to match chapter boundaries",
    "description": "human-readable description of the pattern",
    "examples": ["CHAPTER 1", "CHAPTER 2"]
  },
  "subsection_pattern": {
    "regex": "pattern if applicable, or null",
    "description": "description or null",
    "examples": ["1. ON THE BEACH", "2. BREATHING"]
  },
  "back_matter_start": "text snippet or line number where back matter begins",
  "artifacts_to_remove": ["list of recurring patterns to strip"],
  "chapters_found": [
    {"number": 1, "title": "Relaxation", "marker_text": "CHAPTER 1\nRELAXATION"},
    {"number": 2, "title": "Body Awareness", "marker_text": "CHAPTER 2\nBODY AWARENESS"}
  ]
}

If the text is clean enough, you can also run the structure detection script as an aid:

python /path/to/skill/scripts/detect_structure.py ./cleaned.txt

This script does regex-based detection of common chapter patterns and outputs candidates. Use it as a starting point and verify/adjust with your own analysis.

Stage 4: Split and Format

Using the detected structure, split the cleaned text into chapters and format as markdown.

Create the output markdown file with this structure:

# [Book Title]

*[Author Name]*

---

## Front Matter

[Introduction, preface, etc. — only if substantive content exists]

---

## Chapter 1: [Title]

[Chapter content with proper paragraph breaks]

### [Subsection Title] (if applicable)

[Subsection content]

---

## Chapter 2: [Title]

...

---

## Back Matter

### Resources / Bibliography
### About the Author
### Index (if present and useful)

Formatting rules:

• Use ## (h2) for chapter titles
• Use ### (h3) for subsections within chapters
• Maintain paragraph breaks from the original
• Remove orphaned page numbers
• Fix obvious OCR artifacts: broken words (stu dents → students), missing spaces, stray characters
• Preserve dialogue and quoted text with proper quotation marks
• Keep footnotes as inline parenthetical references or as a notes section at the end of each chapter
• Strip any remaining page headers/footers
• Do NOT rewrite or editorialize content — preserve the author's words

Stage 5: Review and Present

After generating the structured markdown:

1. Check the chapter count matches what was detected in Stage 3
2. Spot-check a few chapters for completeness — are there obvious gaps where content was lost?
3. Note any sections that seem incomplete or garbled
4. Save the final output as [book-title-slug].md to the conversation's output location (/mnt/user-data/outputs/ on claude.ai; the project or working directory when running locally)
5. Present the file to the user with a brief summary:
   • Number of chapters found
   • Any quality issues or gaps detected
   • Sections that may need manual review

Edge Cases

No clear chapter structure: Some books don't have chapters. In this case, look for any structural divisions (parts, sections, numbered items, thematic breaks). If truly no structure exists, output the cleaned text as a single flowing document with just front/back matter separated.

Mixed extraction quality: Some chunks may be clean while others are garbled. Flag the garbled sections with <!-- [EXTRACTION ERROR: Pages X-Y may be incomplete or garbled] --> HTML comments so the user can identify what needs manual attention.

Very large books: For books over 200K characters, process in stages and be mindful of context limits. The scripts handle the mechanical work; LLM analysis should focus on representative samples rather than trying to read the entire text.

Multiple books in one upload: If the user provides several JSON files, process each independently. Offer to batch-process them all with the same workflow.

Scripts Reference

All scripts are in the scripts/ directory of this skill:

Script	Purpose
triage.py	Assess extraction quality, output quality report
clean.py	Strip artifacts, concatenate chunks, normalize text
detect_structure.py	Regex-based chapter pattern detection

Read the scripts before running them to understand their behavior. They are aids, not replacements for judgment — especially for messy extractions where the LLM's pattern recognition is essential.