multimodal-embedding-generator

license	Apache-2.0
name	multimodal-embedding-generator
description	Generate cross-modal embeddings with CLIP, SigLIP, and ImageBind for text-image-audio search. Activate on: multimodal search, text-to-image search, cross-modal embeddings, CLIP embeddings, visual search. NOT for: text-only embeddings (ai-engineer), image classification (computer-vision-pipeline).
allowed-tools	Read,Write,Edit,Bash(python:*,pip:*,npm:*,npx:*)
category	AI & Machine Learning
tags	["multimodal","embeddings","clip","cross-modal-search","siglip"]
pairs-with	[{"skill":"clip-aware-embeddings","reason":"Shared CLIP foundation for visual-semantic alignment"},{"skill":"rag-document-ingestion-pipeline","reason":"Multimodal embeddings feed into vector DB ingestion"},{"skill":"computer-vision-pipeline","reason":"Image preprocessing before embedding extraction"}]

Multimodal Embedding Generator

Generate unified embeddings across text, images, and audio using CLIP, SigLIP, and ImageBind for cross-modal retrieval and search.

Activation Triggers

Activate on: "multimodal search", "text-to-image search", "image-to-text retrieval", "cross-modal embeddings", "CLIP embeddings", "visual search engine", "SigLIP", "ImageBind", "find similar images by description"

NOT for: Text-only embedding and RAG (ai-engineer), image classification or object detection (computer-vision-pipeline), or image generation from text (image-generation-workflow-engine)

Quick Start

1. Define modalities — Which cross-modal searches do you need? Text-to-image, image-to-text, audio-to-text, or all combinations.
2. Select model — SigLIP for text-image (best accuracy/speed), CLIP for broad compatibility, ImageBind for 6-modality coverage.
3. Preprocess inputs — Resize images to model input size, tokenize text, resample audio to 16kHz.
4. Generate embeddings — Batch encode through the chosen model, normalize to unit vectors.
5. Index and search — Store in a vector DB with modality metadata, query with any modality.

Core Capabilities

Domain	Technologies	Notes
Text-Image	SigLIP, CLIP (ViT-L/14, ViT-bigG), OpenCLIP	SigLIP preferred for 2026: better zero-shot accuracy
6-Modality	ImageBind (Meta)	Text, image, audio, depth, thermal, IMU
Local Inference	transformers, open_clip, torch	GPU or MPS (Apple Silicon)
API-Based	Voyage AI multimodal, Cohere embed-v4	Managed, no GPU needed
Indexing	Pinecone, Qdrant, Weaviate, pgvector	Same vector DB for all modalities

Architecture Patterns

Pattern 1: Unified Multimodal Index

Text ──→ [SigLIP Text Encoder] ──┐
                                  ├──→ [Normalize] ──→ [Vector DB]
Image ──→ [SigLIP Vision Encoder]─┘        │              │
                                       L2 normalize    single index,
                                       to unit sphere  modality in metadata

Query (any modality) ──→ [Encode] ──→ [Vector DB Search] ──→ Results (any modality)