| name | multimodal-rag |
| description | Use when user needs RAG on documents with images and text. Triggers on: multimodal RAG, image-text mixed, document with images, PDF with charts, visual RAG, visual Q&A, documents with figures. |
Multimodal RAG
Handle Q&A on documents containing images, tables, charts, and text — answer questions that require understanding both visual and textual content.
When to Activate
Activate this skill when:
- User has documents with images (PDFs with charts, manuals with diagrams)
- User needs to answer questions about visual content in documents
- User mentions "PDF with images", "document with charts", "visual Q&A"
- User's documents have tables, flowcharts, or screenshots
Do NOT activate when:
- User only needs image similarity search → use
image-search
- User only needs text-based RAG → use
rag-toolkit:rag
- User needs video search → use
video-search
Interactive Flow
Step 1: Understand Document Type
"What type of documents are you processing?"
A) Technical manuals (product docs, installation guides)
- Mix of text and diagrams
- Questions like "How do I install component X?"
B) Reports with charts (financial, research, analytics)
- Data visualizations, tables
- Questions like "What was Q3 revenue?"
C) Mixed content (presentations, marketing materials)
- Varied image types
- Diverse question types
Which describes your documents? (A/B/C)
Step 2: Choose Processing Strategy
"How should we handle images?"
| Strategy | When to Use |
|---|
| VLM Description | Charts, diagrams, complex visuals |
| OCR Extraction | Screenshots with text, tables |
| Caption Only | Photos, simple images |
For most cases, VLM Description is recommended.
Step 3: Confirm Configuration
"Based on your requirements:
- Text extraction: PyMuPDF
- Image processing: GPT-4o for descriptions
- Embedding: text-embedding-3-small (1536 dim)
- Answer model: GPT-4o with vision
Proceed? (yes / adjust [what])"
Core Concepts
Mental Model: Illustrated Encyclopedia
Think of multimodal RAG as building a searchable illustrated encyclopedia:
- Extract all content: Text paragraphs AND image descriptions
- Index everything: Both become searchable vectors
- Retrieve mixed results: May get text AND images for a query
- Generate answer: VLM combines all context to answer
┌─────────────────────────────────────────────────────────────┐
│ Multimodal RAG Pipeline │
│ │
│ Document (PDF with images) │
│ │ │
│ ├──────────────────────────────────────┐ │
│ │ │ │
│ ▼ ▼ │
│ ┌──────────────┐ ┌──────────────┐ │
│ │ Text Chunks │ │ Images │ │
│ │ │ │ │ │
│ │ "Section 1 │ │ [diagram.png]│ │
│ │ describes..." │ │ [chart.png] │ │
│ └──────┬───────┘ └──────┬───────┘ │
│ │ │ │
│ │ ▼ │
│ │ ┌──────────────┐ │
│ │ │ VLM Caption │ │
│ │ │ "This chart │ │
│ │ │ shows..." │ │
│ │ └──────┬───────┘ │
│ │ │ │
│ ▼ ▼ │
│ ┌──────────────────────────────────────────────┐ │
│ │ Unified Text Embeddings │ │
│ │ (both text chunks and image captions) │ │
│ └──────────────────────┬───────────────────────┘ │
│ │ │
│ ▼ │
│ ┌──────────────┐ │
│ │ Milvus │ │
│ └──────┬───────┘ │
│ │ │
│ ▼ │
│ Query: "What does the chart show?" │
│ │ │
│ ▼ │
│ Retrieved: [text chunk] + [image caption + image] │
│ │ │
│ ▼ │
│ ┌──────────────────────────────────────────────┐ │
│ │ VLM Answer Generation (with images) │ │
│ │ "Based on the chart, revenue increased..." │ │
│ └──────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────┘
Why Multimodal Matters
| Question Type | Text-Only RAG | Multimodal RAG |
|---|
| "What does paragraph 3 say?" | ✅ Works | ✅ Works |
| "What does the chart show?" | ❌ Can't see | ✅ Understands |
| "Summarize the diagram" | ❌ Can't see | ✅ Describes |
| "What's the value in the table?" | ⚠️ If OCR'd | ✅ Reads directly |
Implementation
from pymilvus import MilvusClient, DataType
from openai import OpenAI
import fitz
import base64
import os
import uuid
class MultimodalRAG:
def __init__(self, uri: str = "./milvus.db"):
self.client = MilvusClient(uri=uri)
self.openai = OpenAI()
self.collection_name = "multimodal_rag"
self._init_collection()
def _init_collection(self):
if self.client.has_collection(self.collection_name):
return
schema = self.client.create_schema()
schema.add_field("id", DataType.VARCHAR, is_primary=True, max_length=64)
schema.add_field("content_type", DataType.VARCHAR, max_length=16)
schema.add_field("content", DataType.VARCHAR, max_length=65535)
schema.add_field("image_path", DataType.VARCHAR, max_length=512)
schema.add_field("source", DataType.VARCHAR, max_length=512)
schema.add_field("page", DataType.INT32)
schema.add_field("embedding", DataType.FLOAT_VECTOR, dim=)
index_params = .client.prepare_index_params()
index_params.add_index(, index_type=, metric_type=)
.client.create_collection(
collection_name=.collection_name,
schema=schema,
index_params=index_params
)
() -> :
response = .openai.embeddings.create(
model=,
=[text]
)
response.data[].embedding
() -> :
(image_path, ) f:
b64_image = base64.standard_b64encode(f.read()).decode()
response = .openai.chat.completions.create(
model=,
messages=[{
: ,
: [
{: , : },
{: , : {: }}
]
}],
max_tokens=
)
response.choices[].message.content
():
os.makedirs(image_output_dir, exist_ok=)
doc = fitz.(pdf_path)
source = os.path.basename(pdf_path)
data = []
page_num, page (doc):
text = page.get_text()
text.strip():
chunks = ._chunk_text(text)
chunk chunks:
data.append({
: (uuid.uuid4()),
: ,
: chunk,
: ,
: source,
: page_num + ,
: ._embed(chunk)
})
images = page.get_images()
img_idx, img (images):
xref = img[]
base_image = doc.extract_image(xref)
image_bytes = base_image[]
image_path =
(image_path, ) f:
f.write(image_bytes)
description = ._describe_image(image_path)
data.append({
: (uuid.uuid4()),
: ,
: description,
: image_path,
: source,
: page_num + ,
: ._embed(description)
})
.client.insert(collection_name=.collection_name, data=data)
(data)
() -> :
words = text.split()
chunks = []
current_chunk = []
word words:
current_chunk.append(word)
(.join(current_chunk)) > chunk_size:
chunks.append(.join(current_chunk))
current_chunk = []
current_chunk:
chunks.append(.join(current_chunk))
chunks
() -> :
embedding = ._embed(query)
results = .client.search(
collection_name=.collection_name,
data=[embedding],
limit=limit,
output_fields=[, , , , ]
)
[{
: hit[][],
: hit[][],
: hit[][],
: hit[][],
: hit[][],
: hit[]
} hit results[]]
() -> :
contexts = .retrieve(question, limit=)
text_contexts = [c c contexts c[] == ]
image_contexts = [c c contexts c[] == ]
messages = [{: , : []}]
context_text = .join([
c text_contexts[:]
])
messages[][].append({
: ,
:
})
use_images image_contexts:
img image_contexts[:]:
os.path.exists(img[]):
(img[], ) f:
b64 = base64.standard_b64encode(f.read()).decode()
messages[][].append({
: ,
: {: }
})
messages[][].append({
: ,
:
})
response = .openai.chat.completions.create(
model=,
messages=messages,
temperature=
)
{
: response.choices[].message.content,
: ((c[] c contexts)),
: ((c[] c contexts))
}
rag = MultimodalRAG()
rag.ingest_pdf()
result = rag.query()
()
()
Processing Strategy by Document Type
| Document Type | Text Extraction | Image Processing | VLM Prompt |
|---|
| Technical manual | Full text | Diagram description | "Explain what this diagram shows step by step" |
| Financial report | Full text | Chart data extraction | "Extract all data points from this chart" |
| Medical report | Full text | Image analysis | "Describe any medical findings visible" |
| Presentation | Slide text | Screenshot description | "Summarize what this slide conveys" |
Common Pitfalls
❌ Pitfall 1: Images Too Small
Problem: VLM can't read chart text
Why: PDF images extracted at low resolution
Fix: Extract at higher resolution
mat = fitz.Matrix(2.0, 2.0)
pix = page.get_pixmap(matrix=mat)
❌ Pitfall 2: Ignoring Image-Text Connection
Problem: Image description doesn't mention surrounding context
Why: Image processed in isolation
Fix: Include nearby text in prompt
prompt = f"This image appears near the text: '{nearby_text}'. Describe the image and how it relates to this text."
❌ Pitfall 3: Too Many API Calls
Problem: Processing costs explode
Why: Calling VLM for every image
Fix: Batch processing, caching, or use cheaper models
❌ Pitfall 4: Not Returning Images in Answer
Problem: User asks about chart but can't see it
Why: Only returning text answer
Fix: Include image references in response
return {
"answer": answer,
"referenced_images": [c["image_path"] for c in image_contexts[:3]]
}
When to Level Up
| Need | Upgrade To |
|---|
| Better chunking | Add core:chunking |
| Higher precision | Add core:rerank |
| Video content | video-search |
| Pure text documents | rag-toolkit:rag |
References
- VLM models: GPT-4o, Claude 3, Qwen-VL, LLaVA
- PDF processing: PyMuPDF, pdf2image
- Batch processing:
core:ray
