text-to-image-search

name	text-to-image-search
description	Use when user needs to search images using natural language descriptions. Triggers on: text to image, describe and find, natural language image search, image caption search, find image by description, describe to find.

Text-to-Image Search

Search images using natural language descriptions — find visuals by describing what you're looking for.

When to Activate

Activate this skill when:

• User wants to find images by describing them in natural language
• User mentions "find image of", "search for pictures of", "describe and find"
• User has complex visual queries ("red car turning right at an intersection")
• User's queries are descriptive sentences, not just keywords

Do NOT activate when:

• User has an image to find similar ones → use image-search
• User wants Q&A on documents with images → use multimodal-rag
• User needs video search → use video-search

Interactive Flow

Step 1: Assess Query Complexity

"What type of text queries will users make?"

A) Simple queries ("cat", "sunset beach", "red car")

• CLIP direct encoding works well
• Fast, no extra API costs

B) Complex queries ("a red car turning right at an intersection at night")

• May need VLM-generated captions
• Better semantic understanding

C) Domain-specific ("tumor in left lung lobe", "fault line in seismic data")

• May need specialized models or VLM descriptions
• Domain vocabulary matters

Which describes your queries? (A/B/C)

Step 2: Choose Architecture

"Based on query complexity, here are your options:"

Option	Approach	Pros	Cons
A: CLIP Direct	Text → CLIP → Search	Fast, free	Weak on complex queries
B: VLM Captions	Image → VLM → Caption → Text embedding	Better semantics	Slow indexing, API cost

Step 3: Confirm Configuration

"Based on your requirements:

• Architecture: [CLIP Direct / VLM Captions]
• Model: [clip-ViT-B-32 / BGE + GPT-4o]
• Index: AUTOINDEX with COSINE

Proceed? (yes / adjust [what])"

Core Concepts

Mental Model: Two Approaches

Think of text-to-image search as two different libraries:

Option A: CLIP (Shared Language)

• Both images and text are translated to the same "language" (vector space)
• Works because CLIP was trained on image-text pairs
• Like having a bilingual librarian who speaks both "image" and "text"

Option B: VLM Captions (Description Matching)

• Each image gets a detailed text description
• Search matches query to descriptions
• Like having someone describe every image in words first

┌─────────────────────────────────────────────────────────────┐
│                    Option A: CLIP Direct                     │
│                                                              │
│  Indexing:                      Search:                      │
│  Image → CLIP → Vector          Text → CLIP → Vector         │
│                                           │                  │
│           Same vector space! ────────────→│                  │
│                                           ▼                  │
│                                      Find similar            │
│                                         vectors              │
└─────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────┐
│                Option B: VLM + Text Embedding                │
│                                                              │
│  Indexing:                                                   │
│  Image → VLM → "A red car..." → BGE → Vector                │
│                                                              │
│  Search:                                                     │
│  Query: "red vehicle" → BGE → Vector → Find similar         │
│                                                              │
│  Matching happens in text embedding space                    │
└─────────────────────────────────────────────────────────────┘

When to Use Each

Scenario	Best Option	Why
Simple queries, high volume	CLIP Direct	Fast, no API cost
Complex descriptions	VLM Captions	Better understanding
Domain-specific (medical, legal)	VLM Captions	Can prompt for domain terms
Budget constrained	CLIP Direct	Free
Quality critical	VLM Captions	More accurate

Implementation

Option A: CLIP Direct

from pymilvus import MilvusClient, DataType
from sentence_transformers import SentenceTransformer
from PIL import Image

class CLIPTextToImageSearch:
    def __init__(self, uri: str = "./milvus.db"):
        self.client = MilvusClient(uri=uri)
        self.model = SentenceTransformer('clip-ViT-B-32')
        self.dim = 512
        self.collection_name = "clip_image_search"
        self._init_collection()

    def _init_collection(self):
        if self.client.has_collection(self.collection_name):
            return

        schema = self.client.create_schema()
        schema.add_field("id", DataType.INT64, is_primary=True, auto_id=True)
        schema.add_field("image_path", DataType.VARCHAR, max_length=512)
        schema.add_field("embedding", DataType.FLOAT_VECTOR, dim=self.dim)

        index_params = self.client.prepare_index_params()
        index_params.add_index("embedding", index_type="AUTOINDEX", metric_type="COSINE")

        self.client.create_collection(
            collection_name=self.collection_name,
            schema=schema,
            index_params=index_params
        )

     ():
        
        images = [Image.(p).convert()  p  image_paths]
        embeddings = .model.encode(images).tolist()

        data = [{: path, : emb}
                 path, emb  (image_paths, embeddings)]

        .client.insert(collection_name=.collection_name, data=data)

     ():
        
        
        embedding = .model.encode(text_query).tolist()

        results = .client.search(
            collection_name=.collection_name,
            data=[embedding],
            limit=limit,
            output_fields=[]
        )

         [{: hit[][], : hit[]}
                 hit  results[]]


search = CLIPTextToImageSearch()
search.add_images([, , ])
results = search.search()

Option B: VLM Captions

from pymilvus import MilvusClient, DataType
from sentence_transformers import SentenceTransformer
from openai import OpenAI
import base64

class VLMTextToImageSearch:
    def __init__(self, uri: str = "./milvus.db"):
        self.client = MilvusClient(uri=uri)
        self.text_model = SentenceTransformer('BAAI/bge-large-en-v1.5')
        self.openai = OpenAI()
        self.dim = 1024
        self.collection_name = "vlm_image_search"
        self._init_collection()

    def _init_collection(self):
        if self.client.has_collection(self.collection_name):
            return

        schema = self.client.create_schema()
        schema.add_field("id", DataType.INT64, is_primary=True, auto_id=True)
        schema.add_field("image_path", DataType.VARCHAR, max_length=512)
        schema.add_field("caption", DataType.VARCHAR, max_length=4096)
        schema.add_field("embedding", DataType.FLOAT_VECTOR, dim=self.dim)

        index_params = self.client.prepare_index_params()
        index_params.add_index("embedding", index_type=, metric_type=)

        .client.create_collection(
            collection_name=.collection_name,
            schema=schema,
            index_params=index_params
        )

     () -> :
        
         (image_path, )  f:
            b64_image = base64.standard_b64encode(f.read()).decode()

        response = .openai.chat.completions.create(
            model=,
            messages=[{
                : ,
                : [
                    {: , : },
                    {: , : {: }}
                ]
            }],
            max_tokens=
        )
         response.choices[].message.content

     ():
        
        data = []
         path  image_paths:
            caption = ._generate_caption(path)
            embedding = .text_model.encode(caption).tolist()
            data.append({
                : path,
                : caption,
                : embedding
            })

        .client.insert(collection_name=.collection_name, data=data)

     ():
        
        embedding = .text_model.encode(text_query).tolist()

        results = .client.search(
            collection_name=.collection_name,
            data=[embedding],
            limit=limit,
            output_fields=[, ]
        )

         [{
            : hit[][],
            : hit[][],
            : hit[]
        }  hit  results[]]


search = VLMTextToImageSearch()
search.add_images([])
results = search.search()

Comparison Table

Aspect	CLIP Direct	VLM Captions
Indexing speed	Fast (ms per image)	Slow (seconds per image)
Query speed	Fast	Fast
API cost	Free	~$0.01 per image
Simple queries	★★★★	★★★★★
Complex queries	★★★	★★★★★
Domain-specific	★★	★★★★
Storage	512d vector only	1024d vector + text

Common Pitfalls

❌ Pitfall 1: Expecting CLIP to Understand Complex Queries

Problem: "red car turning right at night" returns random cars

Why: CLIP wasn't trained on such specific scene descriptions

Fix: Use VLM captions for complex queries

❌ Pitfall 2: VLM Captions Too Generic

Problem: All captions say "This is an image of..."

Why: Default prompts generate generic descriptions

Fix: Use specific prompts

prompt = """Describe this image with:
1. Main objects and their colors
2. Actions or movements
3. Setting/environment
4. Time of day if visible
5. Any text in the image"""

❌ Pitfall 3: Mixing Vector Spaces

Problem: Search returns nothing

Why: Used BGE to embed query but CLIP for images

Fix: Use same model for query as for indexing

# CLIP indexing → CLIP query
# BGE caption indexing → BGE query
# Never mix!

❌ Pitfall 4: High VLM Costs

Problem: $100+ API bill for 10K images

Why: Using GPT-4o for everything

Fix: Use cheaper models or local VLMs

# Cheaper options:
# - gpt-4o-mini (~$0.003 per image)
# - Local LLaVA (free, requires GPU)
# - Qwen-VL API (cheaper for Chinese)

When to Level Up

Need	Upgrade To
Also search by image	Add image-search capability
Q&A on image content	multimodal-rag
Filter by metadata	Add filtered-search pattern
Video content	video-search

References

• VLM model comparison: See OpenAI, Anthropic, Google docs
• CLIP variants: image-search/references/image-embeddings.md
• Batch processing: core:ray