| license | Apache-2.0 |
| name | multimodal-embedding-generator |
| description | Generate cross-modal embeddings with CLIP, SigLIP, and ImageBind for text-image-audio search. Activate on: multimodal search, text-to-image search, cross-modal embeddings, CLIP embeddings, visual search. NOT for: text-only embeddings (ai-engineer), image classification (computer-vision-pipeline). |
| allowed-tools | Read,Write,Edit,Bash(python:*,pip:*,npm:*,npx:*) |
| category | AI & Machine Learning |
| tags | ["multimodal","embeddings","clip","cross-modal-search","siglip"] |
| pairs-with | [{"skill":"clip-aware-embeddings","reason":"Shared CLIP foundation for visual-semantic alignment"},{"skill":"rag-document-ingestion-pipeline","reason":"Multimodal embeddings feed into vector DB ingestion"},{"skill":"computer-vision-pipeline","reason":"Image preprocessing before embedding extraction"}] |
Multimodal Embedding Generator
Generate unified embeddings across text, images, and audio using CLIP, SigLIP, and ImageBind for cross-modal retrieval and search.
Activation Triggers
Activate on: "multimodal search", "text-to-image search", "image-to-text retrieval", "cross-modal embeddings", "CLIP embeddings", "visual search engine", "SigLIP", "ImageBind", "find similar images by description"
NOT for: Text-only embedding and RAG (ai-engineer), image classification or object detection (computer-vision-pipeline), or image generation from text (image-generation-workflow-engine)
Quick Start
- Define modalities — Which cross-modal searches do you need? Text-to-image, image-to-text, audio-to-text, or all combinations.
- Select model — SigLIP for text-image (best accuracy/speed), CLIP for broad compatibility, ImageBind for 6-modality coverage.
- Preprocess inputs — Resize images to model input size, tokenize text, resample audio to 16kHz.
- Generate embeddings — Batch encode through the chosen model, normalize to unit vectors.
- Index and search — Store in a vector DB with modality metadata, query with any modality.
Core Capabilities
| Domain | Technologies | Notes |
|---|
| Text-Image | SigLIP, CLIP (ViT-L/14, ViT-bigG), OpenCLIP | SigLIP preferred for 2026: better zero-shot accuracy |
| 6-Modality | ImageBind (Meta) | Text, image, audio, depth, thermal, IMU |
| Local Inference | transformers, open_clip, torch | GPU or MPS (Apple Silicon) |
| API-Based | Voyage AI multimodal, Cohere embed-v4 | Managed, no GPU needed |
| Indexing | Pinecone, Qdrant, Weaviate, pgvector | Same vector DB for all modalities |
Architecture Patterns
Pattern 1: Unified Multimodal Index
Text ──→ [SigLIP Text Encoder] ──┐
├──→ [Normalize] ──→ [Vector DB]
Image ──→ [SigLIP Vision Encoder]─┘ │ │
L2 normalize single index,
to unit sphere modality in metadata
Query (any modality) ──→ [Encode] ──→ [Vector DB Search] ──→ Results (any modality)
