multimodal-rag

name	multimodal-rag
description	Use when user needs RAG on documents with images and text. Triggers on: multimodal RAG, image-text mixed, document with images, PDF with charts, visual RAG, visual Q&A, documents with figures.

Multimodal RAG

Handle Q&A on documents containing images, tables, charts, and text — answer questions that require understanding both visual and textual content.

When to Activate

Activate this skill when:

• User has documents with images (PDFs with charts, manuals with diagrams)
• User needs to answer questions about visual content in documents
• User mentions "PDF with images", "document with charts", "visual Q&A"
• User's documents have tables, flowcharts, or screenshots

Do NOT activate when:

• User only needs image similarity search → use image-search
• User only needs text-based RAG → use rag-toolkit:rag
• User needs video search → use video-search

Interactive Flow

Step 1: Understand Document Type

"What type of documents are you processing?"

A) Technical manuals (product docs, installation guides)

• Mix of text and diagrams
• Questions like "How do I install component X?"

B) Reports with charts (financial, research, analytics)

• Data visualizations, tables
• Questions like "What was Q3 revenue?"

C) Mixed content (presentations, marketing materials)

• Varied image types
• Diverse question types

Which describes your documents? (A/B/C)

Step 2: Choose Processing Strategy

"How should we handle images?"

Strategy	When to Use
VLM Description	Charts, diagrams, complex visuals
OCR Extraction	Screenshots with text, tables
Caption Only	Photos, simple images

For most cases, VLM Description is recommended.

Step 3: Confirm Configuration

"Based on your requirements:

• Text extraction: PyMuPDF
• Image processing: GPT-4o for descriptions
• Embedding: text-embedding-3-small (1536 dim)
• Answer model: GPT-4o with vision

Proceed? (yes / adjust [what])"

Core Concepts

Mental Model: Illustrated Encyclopedia

Think of multimodal RAG as building a searchable illustrated encyclopedia:

1. Extract all content: Text paragraphs AND image descriptions
2. Index everything: Both become searchable vectors
3. Retrieve mixed results: May get text AND images for a query
4. Generate answer: VLM combines all context to answer

┌─────────────────────────────────────────────────────────────┐
│                    Multimodal RAG Pipeline                   │
│                                                              │
│  Document (PDF with images)                                  │
│       │                                                      │
│       ├──────────────────────────────────────┐              │
│       │                                      │              │
│       ▼                                      ▼              │
│  ┌──────────────┐                    ┌──────────────┐      │
│  │ Text Chunks  │                    │   Images     │      │
│  │              │                    │              │      │
│  │ "Section 1   │                    │ [diagram.png]│      │
│  │  describes..." │                  │ [chart.png]  │      │
│  └──────┬───────┘                    └──────┬───────┘      │
│         │                                   │               │
│         │                                   ▼               │
│         │                           ┌──────────────┐       │
│         │                           │  VLM Caption │       │
│         │                           │  "This chart │       │
│         │                           │   shows..."  │       │
│         │                           └──────┬───────┘       │
│         │                                  │               │
│         ▼                                  ▼               │
│  ┌──────────────────────────────────────────────┐        │
│  │            Unified Text Embeddings            │        │
│  │     (both text chunks and image captions)     │        │
│  └──────────────────────┬───────────────────────┘        │
│                         │                                 │
│                         ▼                                 │
│                  ┌──────────────┐                        │
│                  │    Milvus    │                        │
│                  └──────┬───────┘                        │
│                         │                                 │
│                         ▼                                 │
│  Query: "What does the chart show?"                       │
│                         │                                 │
│                         ▼                                 │
│  Retrieved: [text chunk] + [image caption + image]        │
│                         │                                 │
│                         ▼                                 │
│  ┌──────────────────────────────────────────────┐        │
│  │    VLM Answer Generation (with images)        │        │
│  │    "Based on the chart, revenue increased..." │        │
│  └──────────────────────────────────────────────┘        │
└─────────────────────────────────────────────────────────────┘

Why Multimodal Matters

Question Type	Text-Only RAG	Multimodal RAG
"What does paragraph 3 say?"	✅ Works	✅ Works
"What does the chart show?"	❌ Can't see	✅ Understands
"Summarize the diagram"	❌ Can't see	✅ Describes
"What's the value in the table?"	⚠️ If OCR'd	✅ Reads directly

Implementation

from pymilvus import MilvusClient, DataType
from openai import OpenAI
import fitz  # PyMuPDF
import base64
import os
import uuid

class MultimodalRAG:
    def __init__(self, uri: str = "./milvus.db"):
        self.client = MilvusClient(uri=uri)
        self.openai = OpenAI()
        self.collection_name = "multimodal_rag"
        self._init_collection()

    def _init_collection(self):
        if self.client.has_collection(self.collection_name):
            return

        schema = self.client.create_schema()
        schema.add_field("id", DataType.VARCHAR, is_primary=True, max_length=64)
        schema.add_field("content_type", DataType.VARCHAR, max_length=16)  # text/image
        schema.add_field("content", DataType.VARCHAR, max_length=65535)
        schema.add_field("image_path", DataType.VARCHAR, max_length=512)
        schema.add_field("source", DataType.VARCHAR, max_length=512)
        schema.add_field("page", DataType.INT32)
        schema.add_field("embedding", DataType.FLOAT_VECTOR, dim=)

        index_params = .client.prepare_index_params()
        index_params.add_index(, index_type=, metric_type=)

        .client.create_collection(
            collection_name=.collection_name,
            schema=schema,
            index_params=index_params
        )

     () -> :
        
        response = .openai.embeddings.create(
            model=,
            =[text]
        )
         response.data[].embedding

     () -> :
        
         (image_path, )  f:
            b64_image = base64.standard_b64encode(f.read()).decode()

        response = .openai.chat.completions.create(
            model=,
            messages=[{
                : ,
                : [
                    {: , : },
                    {: , : {: }}
                ]
            }],
            max_tokens=
        )
         response.choices[].message.content

     ():
        
        os.makedirs(image_output_dir, exist_ok=)
        doc = fitz.(pdf_path)
        source = os.path.basename(pdf_path)
        data = []

         page_num, page  (doc):
            
            text = page.get_text()
             text.strip():
                chunks = ._chunk_text(text)
                 chunk  chunks:
                    data.append({
                        : (uuid.uuid4()),
                        : ,
                        : chunk,
                        : ,
                        : source,
                        : page_num + ,
                        : ._embed(chunk)
                    })

            
            images = page.get_images()
             img_idx, img  (images):
                xref = img[]
                base_image = doc.extract_image(xref)
                image_bytes = base_image[]

                image_path = 
                 (image_path, )  f:
                    f.write(image_bytes)

                
                description = ._describe_image(image_path)

                data.append({
                    : (uuid.uuid4()),
                    : ,
                    : description,
                    : image_path,
                    : source,
                    : page_num + ,
                    : ._embed(description)
                })

        .client.insert(collection_name=.collection_name, data=data)
         (data)

     () -> :
        
        words = text.split()
        chunks = []
        current_chunk = []

         word  words:
            current_chunk.append(word)
             (.join(current_chunk)) > chunk_size:
                chunks.append(.join(current_chunk))
                current_chunk = []

         current_chunk:
            chunks.append(.join(current_chunk))

         chunks

     () -> :
        
        embedding = ._embed(query)

        results = .client.search(
            collection_name=.collection_name,
            data=[embedding],
            limit=limit,
            output_fields=[, , , , ]
        )

         [{
            : hit[][],
            : hit[][],
            : hit[][],
            : hit[][],
            : hit[][],
            : hit[]
        }  hit  results[]]

     () -> :
        
        contexts = .retrieve(question, limit=)

        text_contexts = [c  c  contexts  c[] == ]
        image_contexts = [c  c  contexts  c[] == ]

        
        messages = [{: , : []}]

        
        context_text = .join([
            
             c  text_contexts[:]
        ])
        messages[][].append({
            : ,
            : 
        })

        
         use_images  image_contexts:
             img  image_contexts[:]:
                 os.path.exists(img[]):
                     (img[], )  f:
                        b64 = base64.standard_b64encode(f.read()).decode()
                    messages[][].append({
                        : ,
                        : {: }
                    })

        
        messages[][].append({
            : ,
            : 
        })

        response = .openai.chat.completions.create(
            model=,
            messages=messages,
            temperature=
        )

         {
            : response.choices[].message.content,
            : ((c[]  c  contexts)),
            : ((c[]  c  contexts))
        }


rag = MultimodalRAG()


rag.ingest_pdf()


result = rag.query()
()
()

Processing Strategy by Document Type

Document Type	Text Extraction	Image Processing	VLM Prompt
Technical manual	Full text	Diagram description	"Explain what this diagram shows step by step"
Financial report	Full text	Chart data extraction	"Extract all data points from this chart"
Medical report	Full text	Image analysis	"Describe any medical findings visible"
Presentation	Slide text	Screenshot description	"Summarize what this slide conveys"

Common Pitfalls

❌ Pitfall 1: Images Too Small

Problem: VLM can't read chart text

Why: PDF images extracted at low resolution

Fix: Extract at higher resolution

mat = fitz.Matrix(2.0, 2.0)  # 2x zoom
pix = page.get_pixmap(matrix=mat)

❌ Pitfall 2: Ignoring Image-Text Connection

Problem: Image description doesn't mention surrounding context

Why: Image processed in isolation

Fix: Include nearby text in prompt

prompt = f"This image appears near the text: '{nearby_text}'. Describe the image and how it relates to this text."

❌ Pitfall 3: Too Many API Calls

Problem: Processing costs explode

Why: Calling VLM for every image

Fix: Batch processing, caching, or use cheaper models

# Use gpt-4o-mini for initial pass
# Only use gpt-4o for complex charts

❌ Pitfall 4: Not Returning Images in Answer

Problem: User asks about chart but can't see it

Why: Only returning text answer

Fix: Include image references in response

return {
    "answer": answer,
    "referenced_images": [c["image_path"] for c in image_contexts[:3]]
}

When to Level Up

Need	Upgrade To
Better chunking	Add core:chunking
Higher precision	Add core:rerank
Video content	video-search
Pure text documents	rag-toolkit:rag

References

• VLM models: GPT-4o, Claude 3, Qwen-VL, LLaVA
• PDF processing: PyMuPDF, pdf2image
• Batch processing: core:ray