smart-screenshot

name	smart-screenshot
description	Intelligent screenshot and screen capture with OCR, markdown conversion, and annotation. Triggered by PrtSc key, captures screen regions, extracts text with OCR, converts to markdown using MarkItDown, saves with auto-formatting. Similar to Windows Snipping Tool with AI enhancements for text extraction and document processing.

Smart Screenshot

Intelligent screen capture with OCR, markdown conversion, and smart formatting. Capture screen regions, extract text, convert images/PDFs to markdown, and save with automated formatting.

Quick Start

Trigger methods:

1. Keyboard shortcut: Press PrtSc (customizable)
2. Command line: python scripts/capture.py
3. Claude Code: Ask Claude to "take a screenshot"

Workflow:

1. Press PrtSc → Capture mode activates
2. Choose: Image or Text
3. Select region/window
4. If Image: Save with annotation options
5. If Text: OCR → MarkItDown → Save markdown

Prerequisites

System Requirements

• Windows 10/11, macOS 10.14+, or Linux
• Python 3.8+
• Screen with display access

Install Dependencies

Core (required):

# Screenshot and OCR
pip install pillow pyautogui mss pytesseract pyscreenshot --break-system-packages

# MarkItDown (Microsoft's converter)
pip install markitdown --break-system-packages

# Keyboard hooks
pip install keyboard pynput --break-system-packages

# GUI for dialogs
pip install tkinter --break-system-packages  # May be pre-installed

OCR engine (Tesseract):

Windows:

# Download installer from:
# https://github.com/UB-Mannheim/tesseract/wiki
# Install to: C:\Program Files\Tesseract-OCR\
# Add to PATH

macOS:

brew install tesseract

Linux:

sudo apt-get install tesseract-ocr
# or
sudo dnf install tesseract

Optional enhancements:

# Better OCR (EasyOCR - slower but more accurate)
pip install easyocr --break-system-packages

# PDF handling
pip install pdf2image pypdf2 --break-system-packages

# Image enhancement
pip install opencv-python --break-system-packages

# Clipboard integration
pip install pyperclip --break-system-packages

See reference/setup-guide.md for detailed installation.

Features

Capture Modes

1. Region Selection

• Click and drag to select area
• Real-time preview
• Pixel-perfect selection

2. Window Capture

• Automatically detect windows
• Capture specific application
• Includes/excludes borders

3. Full Screen

• Entire display
• Multi-monitor support
• All screens at once

4. Scrolling Capture

• Capture long web pages
• Auto-scroll and stitch
• Perfect for documentation

Text Extraction

OCR Engines:

• Tesseract - Fast, free, 100+ languages
• EasyOCR - Slower, more accurate
• Cloud OCR - Azure/Google (highest accuracy)

Smart text processing:

• Automatic language detection
• Text cleanup and formatting
• Table recognition
• Layout preservation

Markdown Conversion

Using MarkItDown (Microsoft):

• Images → Markdown with alt text
• PDFs → Clean markdown
• Screenshots → Formatted text
• Tables → Markdown tables
• Code blocks → Syntax highlighting

Conversion features:

• Smart heading detection
• List preservation
• Link extraction
• Code formatting
• Table structure recognition

Core Operations

Quick Capture

Keyboard shortcut:

# Run as background service
python scripts/screenshot_service.py

# Now press PrtSc anytime:
# 1. Screen freezes
# 2. Choose "Image" or "Text"
# 3. Select region
# 4. Auto-process and save

Command line:

# Capture with UI
python scripts/capture.py

# Capture full screen immediately
python scripts/capture.py --fullscreen --output screenshot.png

# Capture region with coordinates
python scripts/capture.py --region 100,100,800,600 --output region.png

Text Mode (OCR → Markdown)

Interactive:

# Start capture
python scripts/capture.py --mode text

# Process:
# 1. Select region
# 2. OCR extracts text
# 3. MarkItDown formats
# 4. Save dialog opens
# 5. Save as .md file

Automatic:

# Capture and OCR
python scripts/capture_text.py --output extracted.md

# With specific language
python scripts/capture_text.py --lang eng+fra --output text.md

# With enhancement
python scripts/capture_text.py --enhance --output clean.md

Image Mode

Interactive:

# Start capture
python scripts/capture.py --mode image

# Process:
# 1. Select region
# 2. Annotation tools appear
# 3. Add arrows, boxes, text
# 4. Save dialog opens

With annotations:

# Capture and annotate
python scripts/capture_annotate.py --output annotated.png

# Annotation tools:
# - Arrow
# - Rectangle
# - Circle
# - Text
# - Highlight
# - Blur (redact sensitive info)

PDF to Markdown

Convert PDF to markdown:

# Using MarkItDown
python scripts/pdf_to_markdown.py --input document.pdf --output document.md

# With OCR for scanned PDFs
python scripts/pdf_to_markdown.py --input scanned.pdf --ocr --output text.md

# Batch convert folder
python scripts/batch_pdf_convert.py --input ./pdfs/ --output ./markdown/

Screenshot from Image

Process existing image:

# Extract text to markdown
python scripts/image_to_markdown.py --input screenshot.png --output text.md

# Clean up image first
python scripts/enhance_and_extract.py --input noisy.png --output clean.md

Configuration

Settings file: config.yaml

# Keyboard shortcut
hotkey: "Print"  # or "ctrl+shift+s", "cmd+shift+5", etc.

# Default capture mode
default_mode: "prompt"  # "image", "text", or "prompt"

# OCR settings
ocr:
  engine: "tesseract"  # "tesseract", "easyocr", or "cloud"
  language: "eng"
  enhance: true  # Pre-process image for better OCR

# Output settings
output:
  directory: "~/Screenshots"
  filename_pattern: "Screenshot-{date}-{time}"
  auto_save: false  # true = skip save dialog
  clipboard: true   # Copy to clipboard

# Markdown settings
markdown:
  format_code_blocks: true
  detect_tables: true
  preserve_formatting: true
  
# Annotation defaults
annotation:
  arrow_color: "#FF0000"
  box_color: "#0000FF"
  text_color: "#000000"
  text_size: 12
  line_width: 2

Common Workflows

Workflow 1: Code Documentation

Scenario: Capture code from screen → Markdown documentation

# 1. Run screenshot service
python scripts/screenshot_service.py &

# 2. Press PrtSc on your keyboard

# 3. Select "Text" mode

# 4. Select code region on screen

# 5. OCR extracts code

# 6. MarkItDown formats as code block:
```python
def example_function():
    return "formatted code"

7. Save dialog opens → Save as code-snippet.md

### Workflow 2: Meeting Notes from Slides

**Scenario:** Capture presentation slides → Formatted notes

```bash
# Capture multiple slides
python scripts/capture_sequence.py \
  --count 5 \
  --delay 3 \
  --mode text \
  --output slides.md

# Result: All slides as markdown in one file

Workflow 3: Email/Document Processing

Scenario: Screenshot email → Extract and format text

# Capture email
python scripts/capture.py --mode text --enhance

# Text extracted, formatted, and saved
# Perfect for archiving or processing

Workflow 4: Research Paper Annotation

Scenario: Screenshot paper → Annotate → Save

# Capture and annotate
python scripts/capture_annotate.py --output paper-notes.png

# Add arrows, highlights, notes
# Save annotated version

Workflow 5: Batch PDF Conversion

Scenario: Convert all PDFs to markdown

# Convert folder of PDFs
python scripts/batch_pdf_convert.py \
  --input ~/Documents/PDFs/ \
  --output ~/Documents/Markdown/ \
  --ocr  # Enable OCR for scanned docs
  
# Progress shown for each file
# All PDFs → Clean markdown

MarkItDown Features

Microsoft's MarkItDown converts:

Images:

• Screenshots → Extracted text
• Diagrams → Alt text descriptions
• Charts → Data tables

PDFs:

• Native PDFs → Clean markdown
• Scanned PDFs → OCR + markdown
• Preserve structure and formatting

Documents:

• Word docs → Markdown
• PowerPoint → Slide content
• Excel → Markdown tables

Code:

• Syntax highlighted code blocks
• Language detection
• Proper indentation

Tables:

• Visual tables → Markdown tables
• Preserved alignment
• Header detection

Keyboard Shortcuts

During capture:

• Esc - Cancel capture
• Space - Toggle crosshair/selection
• Enter - Confirm selection
• Ctrl+Z - Undo annotation
• Ctrl+C - Copy to clipboard
• Ctrl+S - Save

Annotation mode:

• A - Arrow tool
• R - Rectangle tool
• C - Circle tool
• T - Text tool
• H - Highlight tool
• B - Blur tool
• Delete - Remove last annotation

OCR Accuracy Tips

Better results:

1. Enhance image first - Increase contrast, denoise
2. Correct language - Specify language(s)
3. Proper DPI - Higher resolution = better OCR
4. Clean background - Remove clutter
5. Good lighting - For camera captures
6. Straight text - Rotate if needed

Pre-processing:

# Enhance before OCR
python scripts/enhance_image.py \
  --input screenshot.png \
  --output enhanced.png \
  --operations "grayscale,contrast,denoise"

# Then OCR
python scripts/image_to_markdown.py --input enhanced.png

Multi-Monitor Support

Capture from specific monitor:

# List monitors
python scripts/list_monitors.py

# Capture from monitor 2
python scripts/capture.py --monitor 2

# Capture all monitors
python scripts/capture.py --all-monitors

Save Dialog Options

When save dialog appears:

• Filename: Auto-generated or custom
• Location: Last used or default directory
• Format: .md, .txt, .png, .jpg
• Options:
  • Copy to clipboard
  • Open in editor
  • Share

Skip dialog (auto-save):

# config.yaml
output:
  auto_save: true
  directory: "~/Screenshots"

Integration with Clipboard

Copy to clipboard automatically:

# Text mode - copies markdown
python scripts/capture.py --mode text --clipboard

# Image mode - copies image
python scripts/capture.py --mode image --clipboard

# Both
python scripts/capture.py --clipboard --save

Paste from clipboard:

# Process clipboard image
python scripts/process_clipboard.py --output result.md

Running as Service

Windows

Install as Windows service:

# Install
python scripts/install_windows_service.py

# Service runs on startup
# PrtSc always available

Or use Task Scheduler:

1. Open Task Scheduler
2. Create Basic Task
3. Trigger: At log on
4. Action: Start program
5. Program: python
6. Arguments: path\to\screenshot_service.py

macOS

LaunchAgent setup:

# Install service
python scripts/install_macos_service.py

# Creates ~/Library/LaunchAgents/com.screenshot.service.plist
# Runs on login

Manual:

# Create LaunchAgent plist
# Load with launchctl
launchctl load ~/Library/LaunchAgents/com.screenshot.service.plist

Linux

Systemd service:

# Install
python scripts/install_linux_service.py

# Creates ~/.config/systemd/user/screenshot.service
# Enable and start:
systemctl --user enable screenshot
systemctl --user start screenshot

Or use autostart:

# Copy desktop entry
cp screenshot.desktop ~/.config/autostart/

Cloud OCR (Optional)

For highest accuracy:

Azure Computer Vision:

export AZURE_CV_KEY="your-key"
export AZURE_CV_ENDPOINT="https://your-region.api.cognitive.microsoft.com/"

python scripts/capture.py --mode text --ocr-engine azure

Google Vision:

export GOOGLE_APPLICATION_CREDENTIALS="path/to/credentials.json"

python scripts/capture.py --mode text --ocr-engine google

Costs:

• Azure: 1000 transactions/month free, then $1/1000
• Google: 1000 units/month free, then $1.50/1000

Scripts Reference

Capture:

• capture.py - Main interactive capture
• capture_text.py - Text mode only
• capture_annotate.py - Image with annotations
• capture_sequence.py - Multiple captures

Service:

• screenshot_service.py - Background service
• install_windows_service.py - Windows installer
• install_macos_service.py - macOS installer
• install_linux_service.py - Linux installer

Conversion:

• image_to_markdown.py - Image → Markdown
• pdf_to_markdown.py - PDF → Markdown
• batch_pdf_convert.py - Batch conversion

Processing:

• enhance_image.py - Image enhancement
• process_clipboard.py - Clipboard processing
• extract_tables.py - Table extraction

Utilities:

• list_monitors.py - List displays
• test_ocr.py - Test OCR accuracy
• configure.py - Interactive config

Best Practices

1. Run as service - Always available with hotkey
2. Configure hotkey - Choose comfortable shortcut
3. Enable clipboard - Quick copy-paste workflow
4. Enhance first - Better OCR results
5. Use appropriate OCR - Tesseract for speed, Cloud for accuracy
6. Organize output - Set default directory
7. Backup settings - Save config.yaml
8. Test thoroughly - Verify OCR accuracy for your use case

Troubleshooting

"Tesseract not found"

# Install Tesseract
# Windows: Download installer
# macOS: brew install tesseract
# Linux: apt install tesseract-ocr

# Check installation
tesseract --version

"Permission denied" (screenshot)

Windows: Run as Administrator
macOS: System Preferences → Security → Privacy → Screen Recording
Linux: Check X11 permissions

"Keyboard hook failed"

# Requires administrator/root privileges
# Windows: Run as Administrator
# macOS: Grant Accessibility permissions
# Linux: Run with sudo or add user to input group

"Poor OCR quality"

# Enhance image first
python scripts/enhance_image.py --input screenshot.png

# Try different OCR engine
python scripts/capture.py --ocr-engine easyocr

# Specify language
python scripts/capture.py --lang eng+fra

"MarkItDown not working"

pip install --upgrade markitdown --break-system-packages

# Check version
python -c "import markitdown; print(markitdown.__version__)"

Platform-Specific Notes

Windows

• PrtSc key native support
• Windows Ink integration available
• OneDrive sync compatible
• Notification system integration

macOS

• cmd+shift+5 alternative
• Quick Look preview
• iCloud Drive sync
• Notification Center integration

Linux

• Wayland/X11 support
• Various hotkey daemons
• Desktop environment integration
• Screenshot directories vary

Integration Examples

See examples/ for complete workflows:

• examples/documentation-workflow.md - Code docs
• examples/research-notes.md - Paper processing
• examples/meeting-capture.md - Meeting slides
• examples/email-archival.md - Email processing

Reference Documentation

• reference/setup-guide.md - Complete setup
• reference/ocr-engines.md - OCR comparison
• reference/markitdown-guide.md - MarkItDown features
• reference/hotkey-config.md - Keyboard shortcuts
• reference/service-install.md - Service setup