Analyze images and extract structured data from visual content using AI. Use when analyzing product photos, extracting text from screenshots, generating alt text for accessibility, visual question answering, categorizing images by content, reading receipts and invoices from photos, OCR with AI, describing images for search indexing, product photo categorization, document image processing, chart and graph extraction, UI screenshot analysis, image-to-structured-data pipelines.
Analyze images and extract structured data from visual content using AI. Use when analyzing product photos, extracting text from screenshots, generating alt text for accessibility, visual question answering, categorizing images by content, reading receipts and invoices from photos, OCR with AI, describing images for search indexing, product photo categorization, document image processing, chart and graph extraction, UI screenshot analysis, image-to-structured-data pipelines.
AI Understanding Images
Use DSPy's dspy.Image type to pass images into signatures alongside text. Vision LLMs return structured data from photos, screenshots, documents, and charts.
Step 1 - Understand the image task
Before writing code, ask:
What images will you process? (URLs, local files, base64, cloud storage?)
What do you need to extract? (text, categories, attributes, descriptions?)
Does the output need to be structured? (typed fields vs. free text?)
Are you processing images in batch or one at a time?
Does the task require reasoning about the image, or just direct extraction?
Step 2 - Build a basic image analyzer
import dspy
lm = dspy.LM("openai/gpt-4o-mini") # or "anthropic/claude-sonnet-4-5-20250929", etc.
dspy.configure(lm=lm)
classAnalyzeImage(dspy.Signature):
"""Analyze the image and answer the question."""
image: dspy.Image = dspy.InputField(desc="The image to analyze")
question: str = dspy.InputField(desc="What to extract or analyze")
answer: str = dspy.OutputField(desc=)
analyzer = dspy.Predict(AnalyzeImage)
result = analyzer(
image=dspy.Image(url=),
question=
)
result = analyzer(
image=dspy.Image(url=),
question=
)
(result.answer)
All models listed here support image inputs. Always verify vision support before deploying.
Step 4 - Combine image with text context
For richer analysis, pass supplemental text alongside the image:
from typing importLiteralfrom pydantic import BaseModel
classProductAttributes(BaseModel):
category: str
color: str
condition: Literal["new", "used", "damaged"]
description: strclassCategorizeProduct(dspy.Signature):
"""Categorize a product from its photo and any provided context."""
image: dspy.Image = dspy.InputField(desc="Product photo")
context: str = dspy.InputField(desc="Additional context such as listing title or seller notes")
attributes: ProductAttributes = dspy.OutputField(desc="Extracted product attributes")
categorizer = dspy.Predict(CategorizeProduct)
result = categorizer(
image=dspy.Image(url="https://example.com/item.jpg"),
context="Listed as: Vintage leather jacket, size M"
)
print(result.attributes.category, result.attributes.condition)
Step 5 - Common patterns
Alt text generation
classGenerateAltText(dspy.Signature):
"""Generate concise, accurate alt text for accessibility."""
image: dspy.Image = dspy.InputField(desc="Image to describe")
context: str = dspy.InputField(desc="Page or article context where the image appears")
alt_text: str = dspy.OutputField(desc="Alt text under 125 characters, describing the image content")
alt_gen = dspy.Predict(GenerateAltText)
result = alt_gen(
image=dspy.Image(url="https://example.com/team-photo.jpg"),
context="About page of a SaaS startup"
)
Receipt and invoice OCR
from typing importListclassLineItem(BaseModel):
description: str
quantity: int
unit_price: float
total: floatclassExtractReceipt(dspy.Signature):
"""Extract all line items and totals from a receipt or invoice photo."""
image: dspy.Image = dspy.InputField(desc="Photo of receipt or invoice")
line_items: List[LineItem] = dspy.OutputField(desc="All line items found")
subtotal: float = dspy.OutputField(desc="Subtotal before tax")
tax: float = dspy.OutputField(desc="Tax amount")
total: float = dspy.OutputField(desc="Total amount due")
extractor = dspy.Predict(ExtractReceipt)
result = extractor(image=dspy.Image(url="receipt.jpg"))
Chart and graph data extraction
classExtractChart(dspy.Signature):
"""Extract the data series and labels from a chart or graph image."""
image: dspy.Image = dspy.InputField(desc="Chart or graph image")
chart_type: str = dspy.OutputField(desc="Type of chart - bar, line, pie, etc.")
title: str = dspy.OutputField(desc="Chart title if present")
data_summary: str = dspy.OutputField(desc="Summary of the data shown, including key values")
chart_reader = dspy.Predict(ExtractChart)
UI screenshot analysis
classAnalyzeUI(dspy.Signature):
"""Analyze a UI screenshot and identify components and issues."""
image: dspy.Image = dspy.InputField(desc="UI screenshot")
focus: str = dspy.InputField(desc="What aspect to analyze - layout, accessibility, bugs, etc.")
findings: str = dspy.OutputField(desc="Detailed findings about the UI")
suggestions: List[str] = dspy.OutputField(desc="Actionable improvement suggestions")
ui_analyzer = dspy.ChainOfThought(AnalyzeUI)
result = ui_analyzer(
image=dspy.Image(url="screenshot.png"),
focus="accessibility issues"
)
Step 6 - OCR vs vision model tradeoff
Scenario
Recommended approach
Clean printed text on white background
Tesseract or cloud OCR (faster, cheaper)
Handwritten text
Vision LLM (GPT-4o, Claude Sonnet)
Mixed layout with images and text
Vision LLM
Receipts with varied formatting
Vision LLM
High-volume document digitization
Dedicated OCR service + vision LLM for exceptions
Extracting structured fields from forms
Vision LLM with typed output
Sub-100ms latency requirement
Dedicated OCR only
Step 7 - Image preprocessing
Vision models have token budgets per image. Large images consume more tokens and slow responses.
from PIL import Image as PILImage
import io, base64
defresize_for_vision(image_path: str, max_side: int = 1024) -> dspy.Image:
"""Resize image so the longest side is at most max_side pixels."""
img = PILImage.open(image_path)
ratio = min(max_side / img.width, max_side / img.height, 1.0)
if ratio < 1.0:
new_size = (int(img.width * ratio), int(img.height * ratio))
img = img.resize(new_size, PILImage.LANCZOS)
buf = io.BytesIO()
img.save(buf, format="JPEG", quality=85)
b64 = base64.b64encode(buf.getvalue()).decode()
return dspy.Image(url=f"data:image/jpeg;base64,{b64}")
image = resize_for_vision("large_photo.jpg")
result = analyzer(image=image, question="What is shown?")
Recommended limits:
Max 1024px on the longest side for most tasks
JPEG quality 80-85 for photos; PNG for screenshots with text
Avoid sending multiple large images in a single call
Step 8 - Evaluate visual tasks
For description quality, use an AI judge:
classImageDescriptionJudge(dspy.Signature):
"""Judge whether an image description is accurate and complete."""
image: dspy.Image = dspy.InputField(desc="The original image")
description: str = dspy.InputField(desc="Description to evaluate")
score: int = dspy.OutputField(desc="Score from 1 to 5, where 5 is fully accurate and complete")
reasoning: str = dspy.OutputField(desc="Explanation of the score")
judge = dspy.Predict(ImageDescriptionJudge)
For structured extraction (OCR, receipts), use exact match or field-level comparison:
defeval_receipt_extraction(prediction, ground_truth):
correct_items = sum(
1for item in prediction.line_items
if item.description in [g.description for g in ground_truth.line_items]
)
recall = correct_items / max(len(ground_truth.line_items), 1)
total_match = abs(prediction.total - ground_truth.total) < 0.01return {"item_recall": recall, "total_correct": total_match}
When NOT to use vision LLMs
Object detection at scale - use YOLO, Detectron2, or a dedicated CV API
Simple OCR on clean printed text - Tesseract or cloud OCR is faster and cheaper
Pixel-level segmentation - use Segment Anything or dedicated segmentation models
Real-time video processing - vision LLMs have too much latency
Sub-100ms latency - vision LLMs typically take 1-5 seconds per image
High-volume identical-format documents - train a specialized model or use template OCR
Key patterns
# Pattern 1 - Direct extraction with typed outputclassExtractFields(dspy.Signature):
"""Extract structured fields from the image."""
image: dspy.Image = dspy.InputField()
fields: MyDataModel = dspy.OutputField()
extractor = dspy.Predict(ExtractFields)
# Pattern 2 - Reasoning about image contentclassReasonAboutImage(dspy.Signature):
"""Reason step by step about what the image shows."""
image: dspy.Image = dspy.InputField()
question: str = dspy.InputField()
answer: str = dspy.OutputField()
reasoner = dspy.ChainOfThought(ReasonAboutImage)
# Pattern 3 - Batch processing
images = [dspy.Image(url=p) for p in image_paths]
results = [extractor(image=img) for img in images]
# Pattern 4 - Iterative refinement when quality is low
refiner = dspy.Refine(dspy.Predict(AnalyzeImage), N=3, reward_fn=my_reward)
Gotchas
Wrap image inputs in dspy.Image - Claude writes raw URL strings as image inputs instead of wrapping them. Always wrap the URL or local path directly: dspy.Image(url="https://...") or dspy.Image(url="local/path.jpg"). Raw strings are treated as text, not images. (The old classmethods from_url(), from_file(), and from_PIL() are deprecated — use dspy.Image(url=...) for all forms.)
Verify the model supports vision - Claude picks a model that does not support image inputs. Not all LLMs handle images. Confirm vision support for your chosen model before deploying (GPT-4o, Claude 3.5+, Gemini 2.x all work).
Use dspy.Refine not dspy.Assert - Claude uses dspy.Assert/dspy.Suggest to validate image outputs. Use dspy.Refine with a reward function for iterative improvement instead.
Resize before sending - Claude sends full-resolution images without resizing. Large images (4K, RAW photos) consume excessive tokens and can hit context limits. Resize to max 1024px on the longest side before processing.
Match module to task - Claude applies dspy.ChainOfThought to all image tasks. Use dspy.Predict for direct extraction (OCR, field parsing). Reserve dspy.ChainOfThought for tasks that genuinely benefit from image reasoning, like diagnosing a bug from a screenshot.
Cross-references
Install any skill: npx skills add lebsral/DSPy-Programming-not-prompting-LMs-skills --skill <name>
/ai-parsing-data - parse structured data from text; complement with image extraction for mixed inputs
/ai-stopping-hallucinations - reduce made-up field values in vision extraction pipelines
/ai-checking-outputs - validate extracted fields after vision model output
/dspy-refine - iterative refinement when initial image analysis quality is low
/dspy-modules - understand dspy.Predict vs dspy.ChainOfThought for image tasks
Install /ai-do if you do not have it — it routes any AI problem to the right skill and is the fastest way to work: npx skills add lebsral/DSPy-Programming-not-prompting-LMs-skills --skill ai-do
Additional resources
For API details (dspy.Image constructor, field parameters, module selection table), see reference.md
