vision-framework

name	vision-framework
description	Analyze images locally using Apple Vision Framework — OCR text recognition, people detection, animal detection, face detection, image classification, barcode/QR reading, body pose estimation, contour tracing, and image similarity. Runs entirely on-device with no network calls. macOS only. Compatible with any agent framework (Claude Code, OpenClaw, Hermes, Codex, Antigravity, etc.).
license	MIT
metadata	{"platform":"macOS","requires":{"bins":["swift"]},"minOS":"13.0","version":"1.0.0"}

Vision Framework

On-device image analysis using Apple's Vision Framework. Supports OCR, people/animal/face detection, image classification, barcode reading, body pose estimation, contour tracing, and image similarity — all running locally with no network calls.

Requires: macOS 13+ and Swift toolchain (Xcode or Command Line Tools)

Setup (first time only)

Build the Swift CLI tool:

cd ${CLAUDE_SKILL_DIR}/swift && swift build -c release

The binary will be at ${CLAUDE_SKILL_DIR}/swift/.build/release/VisionCLI. If the binary already exists and no source files have changed, skip the build.

Usage

${CLAUDE_SKILL_DIR}/swift/.build/release/VisionCLI <image-path> [image-path2 ...] --features <feature-list> [options]

Features

Feature	Description
ocr	Text recognition with confidence scores and bounding boxes
classify	Image classification into 1000+ categories
faces	Face detection with bounding boxes
humans	Human body detection with bounding boxes
animals	Cat/dog detection with species labels
barcodes	QR code and barcode reading (any symbology)
contours	Edge/contour detection
featureprint	Image feature vector for similarity comparison
pose	Human body pose with joint landmarks
all	Run all of the above

Options

Option	Default	Description
--language <tag>	en-US	OCR language hint (BCP 47, e.g. ja-JP, zh-Hans)
--recognition-level <fast|accurate>	accurate	OCR speed vs quality
--confidence <0.0-1.0>	0.0	Minimum confidence threshold
--top <n>	unlimited	Max results per feature

Examples

# Extract all text from a screenshot
VisionCLI screenshot.png --features ocr

# Detect faces in a photo
VisionCLI photo.jpg --features faces --confidence 0.8

# Read a QR code
VisionCLI qr.png --features barcodes

# Classify an image and detect animals
VisionCLI pet.jpg --features classify,animals

# OCR with Japanese text
VisionCLI document.png --features ocr --language ja-JP

# Get body pose landmarks
VisionCLI dance.jpg --features pose --confidence 0.5

# Full analysis of multiple images
VisionCLI img1.jpg img2.jpg --features all

# Image similarity (compare feature vectors)
VisionCLI a.jpg --features featureprint

Output Format

JSON to stdout. Single image returns a flat object; multiple images return an array.

{
  "imagePath": "/path/to/image.jpg",
  "imageSize": {"width": 1920, "height": 1080},
  "features": {
    "ocr": [{"text": "Hello", "confidence": 0.98, "boundingBox": {"x": 0.1, "y": 0.2, "width": 0.3, "height": 0.05}}],
    "faces": [{"boundingBox": {"x": 0.4, "y": 0.3, "width": 0.1, "height": 0.15}, "confidence": 1.0}],
    "animals": [{"animal": "cat", "confidence": 0.95, "boundingBox": {...}}],
    "classify": [{"label": "tabby cat", "confidence": 0.92}],
    "barcodes": [{"type": "QR", "payload": "https://example.com", "boundingBox": {...}}],
    "pose": [{"landmarks": {"nose": {"x": 0.5, "y": 0.3, "confidence": 0.99}, ...}}],
    "contours": {"contourCount": 42},
    "featureprint": {"featureprint": [0.123, -0.456, ...]}
  },
  "errors": []
}

Bounding boxes use normalized coordinates (0.0-1.0) relative to image dimensions. Origin is top-left.

Error Handling

• macOS only: This skill uses Apple Vision Framework, unavailable on Linux/Windows
• Swift not found: Install Xcode Command Line Tools: xcode-select --install
• macOS too old: Requires macOS 13+ for modern async Vision API
• Build failure: Check swift --version and ensure Xcode/CLT is properly installed
• Per-feature errors: If one feature fails, others still run. Check the errors array in JSON output

Notes

• All processing is on-device, no data leaves the machine
• Vision Framework leverages Apple Neural Engine on Apple Silicon for fast inference
• OCR supports 100+ languages via the --language flag
• The featureprint feature generates vectors that can be compared for image similarity (see references)
• For video frame analysis, extract frames first with ffmpeg then analyze with vision-cli