Use this Skill to transcribe historical documents: Tesseract 5 OCR with preprocessing, Kraken for historical fonts, confidence filtering, and post-OCR correction with symspellpy.
Standardmäßig ist der Prompt ausgewählt, der zuerst die Quelle prüft. Sie können zu einem direkten Befehl wechseln oder eine lokale Kopie herunterladen.
Quelldateien prüfen
Lesen Sie SKILL.md und alle von SkillsMP angezeigten Begleitdateien, bevor Sie sich für eine Installation entscheiden.
Mit Codex oder Claude installieren Kopieren Sie diesen Prompt, fügen Sie ihn in Codex, Claude oder einen anderen Assistant ein und lassen Sie die Skill-Seite prüfen und installieren.
Ein direkter Befehl überspringt den Prüf-Prompt. Prüfen Sie die Quelle, bevor Sie ihn ausführen.
Use this Skill to transcribe historical documents: Tesseract 5 OCR with preprocessing, Kraken for historical fonts, confidence filtering, and post-OCR correction with symspellpy.
TL;DR — Transcribe printed and handwritten historical documents using Tesseract 5 LSTM
with OpenCV preprocessing, Kraken for historical typefaces, confidence filtering, and
symspellpy post-correction to reduce word error rate on archaic vocabulary.
When to Use
Use this Skill when you need to:
Digitize printed historical documents (early modern newspapers, books, pamphlets)
Transcribe manuscripts with non-standard letterforms (long-s, ligatures, blackletter)
Process large batches of archival page scans and generate quality reports
Apply post-OCR correction to reduce errors from unusual historical spelling
Extract word-level bounding boxes and confidence scores for downstream NLP
Do not use this Skill for:
Modern documents with clean typography (use a simple pytesseract.image_to_string call)
Real-time camera OCR on mobile (latency budget too low)
HTML with word bounding boxes, baselines, and per-word confidence
Language packs
eng, lat, deu, fra, spa; install with apt install tesseract-ocr-[lang]
Long-s / ligatures
Characters unique to early modern printing; require historical traineddata
Confidence score
Per-word integer 0–100 from image_to_data(); filter below 60 as uncertain
Kraken is an alternative OCR engine trained explicitly on historical typefaces and
handwriting. It uses a segmentation-first workflow: kraken.pageseg.segment() detects
text lines, then a trained model reads each line.
Post-OCR correction with symspellpy applies edit-distance lookup against a historical
vocabulary dictionary. Words with confidence below threshold are replaced by the closest
dictionary entry, reducing character error rate by 15–30% on 17th-century German texts.
Raw archival scans typically require noise removal, binarization, and deskew before OCR.
The pipeline below uses OpenCV and converts to a PIL Image for pytesseract.
import cv2
import numpy as np
from PIL import Image
import pytesseract
defpreprocess_historical_image(
image_path: str,
deskew: bool = True,
method: str = "otsu",
) -> tuple[np.ndarray, Image.Image]:
"""
Preprocess a historical document scan for Tesseract OCR.
Steps:
1. Load in colour, convert to grayscale.
2. Denoise with a Gaussian blur (light despeckling).
3. Binarize with Otsu or adaptive thresholding.
4. Optionally deskew using the Hough line transform.
Args:
image_path: Absolute path to the input image (JPG, PNG, TIFF).
deskew: Whether to correct rotational skew.
method: Binarization method: "otsu" | "adaptive".
Returns:
Tuple of (binary numpy array, PIL Image ready for pytesseract).
"""
img_bgr = cv2.imread(image_path)
if img_bgr isNone:
raise FileNotFoundError(f"Cannot open image: {image_path}")
gray = cv2.cvtColor(img_bgr, cv2.COLOR_BGR2GRAY)
# Despeckling: Gaussian blur removes salt-and-pepper noise
denoised = cv2.GaussianBlur(gray, (3, 3), 0)
# Binarizationif method == "otsu":
_, binary = cv2.threshold(denoised, 0, 255,
cv2.THRESH_BINARY + cv2.THRESH_OTSU)
elif method == "adaptive":
binary = cv2.adaptiveThreshold(
denoised, 255,
cv2.ADAPTIVE_THRESH_GAUSSIAN_C,
cv2.THRESH_BINARY,
blockSize=31,
C=10,
)
else:
raise ValueError(f"Unknown method '{method}'. Use 'otsu' or 'adaptive'.")
# Morphological closing to reconnect broken character strokes
kernel = cv2.getStructuringElement(cv2.MORPH_RECT, (2, 2))
binary = cv2.morphologyEx(binary, cv2.MORPH_CLOSE, kernel)
# Deskewif deskew:
binary = _deskew(binary)
pil_image = Image.fromarray(binary)
return binary, pil_image
def_deskew(binary: np.ndarray) -> np.ndarray:
"""
Correct rotational skew using minAreaRect on the foreground text pixels.
Args:
binary: Binarized image as numpy array (white text on black OR inverted).
Returns:
Deskewed binary image.
"""# Invert so text pixels are white (foreground)
inverted = cv2.bitwise_not(binary)
coords = np.column_stack(np.where(inverted > 0))
iflen(coords) < 10:
return binary # not enough foreground pixels
angle = cv2.minAreaRect(coords)[-1]
if angle < -45:
angle = 90 + angle
(h, w) = binary.shape
center = (w // 2, h // 2)
M = cv2.getRotationMatrix2D(center, angle, 1.0)
deskewed = cv2.warpAffine(
binary, M, (w, h),
flags=cv2.INTER_CUBIC,
borderMode=cv2.BORDER_REPLICATE,
)
return deskewed
defrun_tesseract_with_confidence(
pil_image: Image.Image,
lang: str = "eng",
confidence_threshold: int = 60,
psm: int = 6,
) -> dict:
"""
Run Tesseract OCR and return full text plus confidence-filtered word list.
Args:
pil_image: Preprocessed PIL Image.
lang: Tesseract language code(s), e.g. "eng+lat".
confidence_threshold: Discard words with confidence below this value.
psm: Page segmentation mode (6 = uniform text block).
Returns:
Dict with keys: full_text (str), words (list of dicts with
keys text/conf/left/top/width/height), low_conf_words (list).
"""
config = f"--oem 1 --psm {psm}"
full_text = pytesseract.image_to_string(pil_image, lang=lang, config=config)
data = pytesseract.image_to_data(
pil_image, lang=lang, config=config,
output_type=pytesseract.Output.DICT,
)
words = []
low_conf_words = []
for i, word_text inenumerate(data["text"]):
word_text = word_text.strip()
ifnot word_text:
continue
conf = int(data["conf"][i])
entry = {
"text": word_text,
"conf": conf,
"left": data["left"][i],
"top": data["top"][i],
"width": data["width"][i],
"height": data["height"][i],
}
words.append(entry)
if conf < confidence_threshold:
low_conf_words.append(entry)
return {
"full_text": full_text,
"words": words,
"low_conf_words": low_conf_words,
"mean_confidence": float(np.mean([w["conf"] for w in words])) if words else0.0,
}
Step 2 — Batch Folder OCR with Quality Report
import os
import pandas as pd
from pathlib import Path
defbatch_ocr_folder(
input_dir: str,
output_dir: str,
lang: str = "deu",
extensions: tuple = (".jpg", ".jpeg", ".tif", ".tiff", ".png"),
confidence_threshold: int = 60,
) -> pd.DataFrame:
"""
Run OCR on every image in a folder and produce a quality report CSV.
Args:
input_dir: Directory containing page scans.
output_dir: Directory to write .txt transcript files.
lang: Tesseract language code(s).
extensions: Accepted image file extensions.
confidence_threshold: Confidence threshold for quality flagging.
Returns:
DataFrame with one row per image: filename, mean_conf, low_conf_pct,
word_count, flagged (bool).
"""
os.makedirs(output_dir, exist_ok=True)
image_paths = sorted(
p for p in Path(input_dir).iterdir()
if p.suffix.lower() in extensions
)
ifnot image_paths:
raise ValueError(f"No images found in {input_dir}")
records = []
for img_path in image_paths:
try:
_, pil_img = preprocess_historical_image(str(img_path))
result = run_tesseract_with_confidence(
pil_img, lang=lang,
confidence_threshold=confidence_threshold,
)
except Exception as exc:
print(f"[WARN] Failed on {img_path.name}: {exc}")
records.append({
"filename": img_path.name,
"mean_conf": 0.0,
"low_conf_pct": 100.0,
"word_count": 0,
"flagged": True,
"error": str(exc),
})
continue# Write transcript
out_path = Path(output_dir) / (img_path.stem + ".txt")
out_path.write_text(result["full_text"], encoding="utf-8")
word_count = len(result["words"])
low_conf_pct = (
100.0 * len(result["low_conf_words"]) / word_count
if word_count > 0else100.0
)
records.append({
"filename": img_path.name,
"mean_conf": round(result["mean_confidence"], 1),
"low_conf_pct": round(low_conf_pct, 1),
"word_count": word_count,
"flagged": result["mean_confidence"] < confidence_threshold,
"error": "",
})
print(f" {img_path.name}: mean_conf={result['mean_confidence']:.1f} "f"words={word_count} low_conf={low_conf_pct:.1f}%")
report_df = pd.DataFrame(records)
report_path = Path(output_dir) / "quality_report.csv"
report_df.to_csv(report_path, index=False)
print(f"\nQuality report saved to {report_path}")
print(report_df.describe())
return report_df
Step 3 — symspellpy Post-OCR Correction
from symspellpy import SymSpell, Verbosity
defbuild_symspell(
dictionary_path: str,
max_edit_distance: int = 2,
) -> SymSpell:
"""
Load a frequency dictionary into SymSpell for post-OCR correction.
Args:
dictionary_path: Path to a SymSpell-format frequency dictionary
(word SPACE frequency, one per line).
max_edit_distance: Maximum edit distance for lookup (2 recommended).
Returns:
Loaded SymSpell instance.
"""
sym_spell = SymSpell(max_dictionary_edit_distance=max_edit_distance)
loaded = sym_spell.load_dictionary(
dictionary_path, term_index=0, count_index=1, encoding="utf-8"
)
ifnot loaded:
raise ValueError(f"Failed to load dictionary from {dictionary_path}")
return sym_spell
defcorrect_ocr_text(
words: list[dict],
sym_spell: SymSpell,
confidence_threshold: int = 60,
max_edit_distance: int = 2,
) -> str:
"""
Apply symspellpy correction to low-confidence OCR words.
Only words below confidence_threshold are looked up; high-confidence
words are kept as-is to avoid over-correction.
Args:
words: List of word dicts from run_tesseract_with_confidence().
sym_spell: Loaded SymSpell instance.
confidence_threshold: Words below this confidence are candidates for correction.
max_edit_distance: Maximum edit distance for the lookup.
Returns:
Corrected full text string.
"""
corrected_tokens = []
for w in words:
token = w["text"]
if w["conf"] < confidence_threshold and token.isalpha():
suggestions = sym_spell.lookup(
token.lower(),
Verbosity.CLOSEST,
max_edit_distance=max_edit_distance,
)
if suggestions:
candidate = suggestions[0].term
# Preserve capitalisation heuristicif token[0].isupper():
candidate = candidate.capitalize()
corrected_tokens.append(candidate)
else:
corrected_tokens.append(token)
else:
corrected_tokens.append(token)
return" ".join(corrected_tokens)
defcalculate_wer(reference: str, hypothesis: str) -> float:
"""
Calculate Word Error Rate (WER) = (S + D + I) / N.
Args:
reference: Ground-truth transcription string.
hypothesis: OCR output string.
Returns:
WER as a float between 0.0 and 1.0.
"""
ref_words = reference.lower().split()
hyp_words = hypothesis.lower().split()
n = len(ref_words)
if n == 0:
return0.0# Dynamic programming edit distance on word sequences
dp = list(range(len(hyp_words) + 1))
for i, rw inenumerate(ref_words):
new_dp = [i + 1]
for j, hw inenumerate(hyp_words):
if rw == hw:
new_dp.append(dp[j])
else:
new_dp.append(1 + min(dp[j], dp[j + 1], new_dp[-1]))
dp = new_dp
return dp[len(hyp_words)] / n
Advanced Usage
hOCR Output with Coordinates
defextract_hocr(
pil_image: Image.Image,
lang: str = "eng",
output_path: str = None,
) -> str:
"""
Generate hOCR (HTML with bounding boxes) from a preprocessed image.
The hOCR format encodes word coordinates as:
title="bbox left top right bottom; x_wconf NN"
which is useful for downstream alignment with page facsimiles.
Args:
pil_image: Preprocessed PIL Image.
lang: Tesseract language code.
output_path: If given, write the hOCR HTML to this file.
Returns:
hOCR HTML string.
"""
config = "--oem 1 --psm 6"
hocr = pytesseract.image_to_pdf_or_hocr(
pil_image, lang=lang, config=config, extension="hocr"
)
hocr_str = hocr.decode("utf-8")
if output_path:
withopen(output_path, "w", encoding="utf-8") as fh:
fh.write(hocr_str)
print(f"hOCR saved to {output_path}")
return hocr_str
Kraken Segmentation for Historical Fonts
# Install Kraken and download a historical Latin model
pip install kraken
kraken get 10.5281/zenodo.6657808
# Binarize input image
kraken -i page_001.jpg page_001_bin.png binarize
# Segment text lines
kraken -i page_001_bin.png page_001_seg.json segment
# Transcribe with a historical model
kraken -i page_001_bin.png page_001.txt ocr -m historical_latin.mlmodel
Troubleshooting
Problem
Cause
Fix
TesseractNotFoundError
Tesseract binary not on PATH
Set pytesseract.pytesseract.tesseract_cmd = '/usr/bin/tesseract'
Empty output on good image
Wrong PSM mode
Try --psm 3 (auto) or --psm 11 (sparse text)
Very low confidence (<30) across page
Binarization threshold too aggressive
Switch from Otsu to adaptive (method="adaptive")
Long-s (ſ) misread as f
No historical traineddata
Download enm (Middle English) or custom traineddata
SymSpell: dictionary not loaded
Wrong path or encoding
Ensure UTF-8; check term/count column indices
Deskew makes image worse
Very curved binding warps
Set deskew=False; use manual GCP-based correction
cv2.error on TIFF
Unsupported TIFF compression
Open with Pillow first: pil_img = Image.open(path).convert("RGB"); convert via numpy
