clip

name	clip
description	Runs OpenAI CLIP for zero-shot image classification, image-text cosine matching, semantic search, and cross-modal retrieval in a shared embedding space. Use when labeling images without training, ranking images by a text query, or scoring vision-language similarity. Not for captioning (BLIP-2), conversational VLM chat (LLaVA), or pixel-level segmentation (SAM); never skip L2-normalizing embeddings before cosine scores.
version	1.0.1
author	Orchestra Research
license	MIT
dependencies	["transformers","torch","pillow","ftfy","regex","tqdm"]
platforms	["windows","linux","macos"]
metadata	{"hermes":{"tags":["Multimodal","CLIP","Vision-Language","Zero-Shot","Image Classification","OpenAI","Image Search","Cross-Modal Retrieval","Content Moderation"]}}

CLIP — Contrastive Language-Image Pre-Training

OpenAI's CLIP connects vision and language in a shared embedding space, enabling zero-shot image classification, image-text similarity, semantic image search, and cross-modal retrieval without fine-tuning. Trained on 400M image-text pairs; matches ResNet-50 zero-shot on ImageNet.

When to Use

Use CLIP when you need:

• Zero-shot image classification (no training data required)
• Image-text similarity or matching scores
• Semantic image search (text query → ranked images)
• Content moderation (detect NSFW, violence, graphic content)
• Cross-modal retrieval (image→text, text→image)
• Visual question answering (broad category level)

Use alternatives instead:

• BLIP-2 — better image captioning
• LLaVA — vision-language conversational chat
• Segment Anything (SAM) — pixel-level image segmentation

Prerequisites

1. Python 3.8+ installed on the host.
2. PyTorch with CUDA support if a GPU is available (10–50× faster). CPU works but is slower.
3. On Windows (PowerShell), ensure python and pip are on PATH:

   python --version
   pip --version
   

4. Install the OpenAI CLIP package and dependencies:

   pip install git+https://github.com/openai/CLIP.git
   pip install torch torchvision ftfy regex tqdm
   

5. Verify import:

   python -c "import clip; print('CLIP OK')"
   

Procedure

1. Load a model and preprocess function

import torch
import clip
from PIL import Image

device = "cuda" if torch.cuda.is_available() else "cpu"
model, preprocess = clip.load("ViT-B/32", device=device)

Available models (sorted by size):

Model	Parameters	Speed