macos-vision-ocr

name	macos-vision-ocr
description	Vision OCR on macOS for images, screenshots, figure panels.
version	1.0
author	hermes
tags	["ocr","vision","macos","swift","chinese","screenshots","transcription"]
category	devops

macOS Vision Framework Image OCR

Trigger

Transcribe all text from an image (not a PDF): screenshots, 小红书/微信 carousel posts (轮播图), figure panels, memes, scanned crops — especially Chinese + English mixed content. The macOS Vision framework (VNRecognizeTextRequest) is the best native Chinese OCR on macOS (far better than Tesseract); this is the user's preferred fallback when vision_analyze is unavailable or the result needs verification.

For PDFs, see the (user-owned) pdf-text-extraction skill — adjacent territory.

Quick recipe (verified 2026-08-12 on 4 小红书 carousel images)

1. Convert WebP → PNG (macOS 26 sips handles webp):

   sips -s format png img.webp --out img.png
   

2. Run the OCR script:

   swift scripts/ocr.swift img.png          # normal
   swift scripts/ocr.swift img.png nocorr   # garbled/vertical text: disable language correction
   

   Output is sorted top-to-bottom with normalized bounding boxes. nocorr + topCandidates reveal raw glyphs when language correction mangles single characters.

3. Upscale small text — Vision accuracy collapses below ~20px glyphs. Re-OCR at 2x, then 3x:

   sips -z <2xH> <2xW> img.png --out img_2x.png
   

   Text that OCRs identically across 1x/2x/3x passes is trustworthy; text that stays garbled across scales is probably stylized/rotated.

4. Crop-then-upscale for tiny details (model IDs, signatures, superscripts, footnotes): PIL-crop the region, resize ×8–10 with LANCZOS, OCR the crop:

   from PIL import Image
   im = Image.open('img.png')
   c = im.crop((x0, y0, x1, y1)).resize(((x1-x0)*10, (y1-y0)*10), Image.LANCZOS)
   c.save('crop.png')
   

5. Vertical / rotated text: crop the strip, save 90° and 270° rotations, OCR each with nocorr. Vertical Chinese is read as Latin garbage by default ("nal nunbor / suitingn / hindivid") — rotating + disabling correction is the fix.

THE pitfall — coordinate systems (cost ~5 wasted rounds)

Vision boundingBox y is measured from the BOTTOM-left (Quartz convention: 0 = bottom, 1 = top). PIL Image.crop((left, top, right, bottom)) and sips crops are top-origin. Convert with:

y_from_top = 1 - y_vision

Getting this backwards silently crops the WRONG HALF of the image → "crop found nothing / found the wrong text" loops. ALWAYS convert before mapping OCR results onto crop coordinates. Also: sips -c H W --cropOffset y x offset semantics are confusing — prefer PIL Image.crop().

Verification practice

• For key details (model names, signatures, numbers), OCR the tight region at ×8–10 and cross-check against 2–3 full-image passes. Single-pass results on small text are provisional.
• Report genuinely unreadable bits honestly as [?] — do not invent a reading. OCR noise on small glyphs is expected (e.g. "Soqpraty 6onpany" → AI Security Company).
• Figure panels that are webpage screenshots may bleed adjacent page text into the edges — identify and separate it from figure content.

Files

• scripts/ocr.swift — ready-to-run Vision OCR script (zh-Hans + en-US, sorted output, optional nocorr mode).
• references/vision-image-ocr.md — worked example (CoT paper carousel), noise patterns, coordinate-trap detail.