CLIP — Contrastive Language-Image Pre-Training
OpenAI's CLIP connects vision and language in a shared embedding space, enabling zero-shot image classification, image-text similarity, semantic image search, and cross-modal retrieval without fine-tuning. Trained on 400M image-text pairs; matches ResNet-50 zero-shot on ImageNet.
When to Use
Use CLIP when you need:
- Zero-shot image classification (no training data required)
- Image-text similarity or matching scores
- Semantic image search (text query → ranked images)
- Content moderation (detect NSFW, violence, graphic content)
- Cross-modal retrieval (image→text, text→image)
- Visual question answering (broad category level)
Use alternatives instead:
- BLIP-2 — better image captioning
- LLaVA — vision-language conversational chat
- Segment Anything (SAM) — pixel-level image segmentation
Prerequisites
- Python 3.8+ installed on the host.
- PyTorch with CUDA support if a GPU is available (10–50× faster). CPU works but is slower.
- On Windows (PowerShell), ensure
python and pip are on PATH:python --version
pip --version
- Install the OpenAI CLIP package and dependencies:
pip install git+https://github.com/openai/CLIP.git
pip install torch torchvision ftfy regex tqdm
- Verify import:
python -c "import clip; print('CLIP OK')"
Procedure
1. Load a model and preprocess function
import torch
import clip
from PIL import Image
device = "cuda" if torch.cuda.is_available() else "cpu"
model, preprocess = clip.load("ViT-B/32", device=device)
Available models (sorted by size):
| Model |
Parameters |
Speed |
Quality |
| RN50 |
102M |
Fast |
Good |
| RN101 |
— |
Medium |
Good |
| ViT-B/32 |
151M |
Medium |
Better (recommended) |
| ViT-B/16 |
— |
Slower |
Better |
| ViT-L/14 |
428M |
Slow |
Best |
# List all available models
print(clip.available_models())
2. Zero-shot image classification
import torch
import clip
from PIL import Image
device = "cuda" if torch.cuda.is_available() else "cpu"
model, preprocess = clip.load("ViT-B/32", device=device)
# Load and preprocess image
image = preprocess(Image.open("photo.jpg")).unsqueeze(0).to(device)
# Define candidate labels (descriptive phrases work best)
labels = ["a dog", "a cat", "a bird", "a car"]
text = clip.tokenize(labels).to(device)
# Compute similarity and probabilities
with torch.no_grad():
logits_per_image, _ = model(image, text)
probs = logits_per_image.softmax(dim=-1).cpu().numpy()
for label, prob in zip(labels, probs[0]):
print(f"{label}: {prob:.2%}")
3. Image-text similarity (cosine)
# Encode and normalize
with torch.no_grad():
image_features = model.encode_image(image)
text_features = model.encode_text(text)
image_features /= image_features.norm(dim=-1, keepdim=True)
text_features /= text_features.norm(dim=-1, keepdim=True)
similarity = (image_features @ text_features.T).item()
print(f"Similarity: {similarity:.4f}")
HARD RULE: Always normalize embeddings before computing cosine similarity. Unnormalized dot products are not bounded to [-1, 1] and will produce misleading scores.
4. Semantic image search (text → images)
image_paths = ["img1.jpg", "img2.jpg", "img3.jpg"]
image_embeddings = []
for img_path in image_paths:
image = preprocess(Image.open(img_path)).unsqueeze(0).to(device)
with torch.no_grad():
embedding = model.encode_image(image)
embedding /= embedding.norm(dim=-1, keepdim=True)
image_embeddings.append(embedding)
image_embeddings = torch.cat(image_embeddings)
# Encode text query
query = "a sunset over the ocean"
text_input = clip.tokenize([query]).to(device)
with torch.no_grad():
text_embedding = model.encode_text(text_input)
text_embedding /= text_embedding.norm(dim=-1, keepdim=True)
# Rank images
similarities = (text_embedding @ image_embeddings.T).squeeze(0)
top_k = similarities.topk(3)
for idx, score in zip(top_k.indices, top_k.values):
print(f"{image_paths[idx]}: {score:.3f}")
5. Content moderation
categories = [
"safe for work",
"not safe for work",
"violent content",
"graphic content"
]
text = clip.tokenize(categories).to(device)
with torch.no_grad():
logits_per_image, _ = model(image, text)
probs = logits_per_image.softmax(dim=-1)
max_idx = probs.argmax().item()
max_prob = probs[0, max_idx].item()
print(f"Category: {categories[max_idx]} ({max_prob:.2%})")
6. Batch processing
# Batch of images
images = [preprocess(Image.open(f"img{i}.jpg")) for i in range(10)]
images = torch.stack(images).to(device)
with torch.no_grad():
image_features = model.encode_image(images)
image_features /= image_features.norm(dim=-1, keepdim=True)
# Batch of texts
texts = ["a dog", "a cat", "a bird"]
text_tokens = clip.tokenize(texts).to(device)
with torch.no_grad():
text_features = model.encode_text(text_tokens)
text_features /= text_features.norm(dim=-1, keepdim=True)
# Similarity matrix: (10 images × 3 texts)
similarities = image_features @ text_features.T
print(similarities.shape) # torch.Size([10, 3])
7. Integration with a vector database (Chroma)
import chromadb
client = chromadb.Client()
collection = client.create_collection("image_embeddings")
# Add image embeddings
for img_path, embedding in zip(image_paths, image_embeddings):
collection.add(
embeddings=[embedding.cpu().numpy().tolist()],
metadatas=[{"path": img_path}],
ids=[img_path]
)
# Query with text
query = "a sunset"
text_embedding = model.encode_text(clip.tokenize([query]).to(device))
text_embedding /= text_embedding.norm(dim=-1, keepdim=True)
results = collection.query(
query_embeddings=[text_embedding.cpu().numpy().tolist()],
n_results=5
)
print(results)
Best practices
- Use ViT-B/32 for most cases — best speed/quality balance.
- Always normalize embeddings — required for cosine similarity.
- Batch processing — encode multiple images/texts in one forward pass for efficiency.
- Cache embeddings — re-encoding is expensive; store normalized vectors.
- Use descriptive labels — "a photo of a golden retriever" beats "dog".
- GPU recommended — 10–50× faster than CPU.
- Always use the provided
preprocess function — handles resize, center crop, normalization. Do not feed raw PIL images.
Pitfalls
- Not for fine-grained tasks — CLIP excels at broad categories; it struggles with sub-class distinctions (e.g., 200 bird species).
- Vague labels perform poorly — "animal" is weak; "a photo of a sleeping cat on a couch" is strong.
- Dataset biases — trained on web image-text pairs; may reflect societal and dataset biases. Do not use as sole arbiter for sensitive moderation decisions.
- No bounding boxes — CLIP operates on whole images only. Use a detection model (YOLO, DETR) for localization.
- Limited spatial understanding — weak at counting, positional reasoning, and reading text in images.
- Forgetting to normalize — the most common bug. Without L2 normalization, dot products are not cosine similarities.
- Token length limit —
clip.tokenize truncates to 77 context tokens. Long prompts are silently cut; keep labels concise.
- CPU-only latency — image encoding ~200 ms on CPU vs ~20 ms on GPU. For production search, use GPU or pre-compute and cache.
Verification
Verify installation:
python -c "import clip; print(clip.available_models())"
Expected output includes: ['RN50', 'RN101', 'RN50x4', 'RN50x16', 'RN50x64', 'ViT-B/32', 'ViT-B/16', 'ViT-L/14', 'ViT-L/14@336px']
Verify model loads and produces embeddings:
import torch, clip
device = "cuda" if torch.cuda.is_available() else "cpu"
model, preprocess = clip.load("ViT-B/32", device=device)
text = clip.tokenize(["a test label"]).to(device)
with torch.no_grad():
feats = model.encode_text(text)
print(feats.shape) # torch.Size([1, 512])
Verify zero-shot classification output format:
probs = logits_per_image.softmax(dim=-1).cpu().numpy()
assert probs.shape == (1, len(labels))
assert abs(probs.sum() - 1.0) < 1e-5 # probabilities sum to 1
Verify cosine similarity is bounded:
image_features /= image_features.norm(dim=-1, keepdim=True)
text_features /= text_features.norm(dim=-1, keepdim=True)
sim = (image_features @ text_features.T).item()
assert -1.0 <= sim <= 1.0
Performance Reference
| Operation |
CPU |
GPU (V100) |
| Image encoding |
~200 ms |
~20 ms |
| Text encoding |
~50 ms |
~5 ms |
| Similarity compute |
<1 ms |
<1 ms |
Resources
1---
2name: clip
3description: "Runs OpenAI CLIP for zero-shot image classification, image-text cosine matching, semantic search, and cross-modal retrieval in a shared embedding space. Use when labeling images without training, ranking images by a text query, or scoring vision-language similarity. Not for captioning (BLIP-2), conversational VLM chat (LLaVA), or pixel-level segmentation (SAM); never skip L2-normalizing embeddings before cosine scores."
4version: 1.0.1
5author: Orchestra Research
6license: MIT
7dependencies: [transformers, torch, pillow, ftfy, regex, tqdm]
8platforms: [windows, linux, macos]
9metadata:
10 source: https://skillmd.com/skills/kayforkind/clip
11 hermes:
12 tags: [Multimodal, CLIP, Vision-Language, Zero-Shot, Image Classification, OpenAI, Image Search, Cross-Modal Retrieval, Content Moderation]
13
14---
15
16# CLIP — Contrastive Language-Image Pre-Training
17
18OpenAI's CLIP connects vision and language in a shared embedding space, enabling zero-shot image classification, image-text similarity, semantic image search, and cross-modal retrieval without fine-tuning. Trained on 400M image-text pairs; matches ResNet-50 zero-shot on ImageNet.
19
20## When to Use
21
22**Use CLIP when you need:**
23- Zero-shot image classification (no training data required)
24- Image-text similarity or matching scores
25- Semantic image search (text query → ranked images)
26- Content moderation (detect NSFW, violence, graphic content)
27- Cross-modal retrieval (image→text, text→image)
28- Visual question answering (broad category level)
29
30**Use alternatives instead:**
31- **BLIP-2** — better image captioning
32- **LLaVA** — vision-language conversational chat
33- **Segment Anything (SAM)** — pixel-level image segmentation
34
35## Prerequisites
36
371. Python 3.8+ installed on the host.
382. PyTorch with CUDA support if a GPU is available (10–50× faster). CPU works but is slower.
393. On Windows (PowerShell), ensure `python` and `pip` are on `PATH`:
40 ```powershell
41 python --version
42 pip --version
43 ```
444. Install the OpenAI CLIP package and dependencies:
45 ```powershell
46 pip install git+https://github.com/openai/CLIP.git
47 pip install torch torchvision ftfy regex tqdm
48 ```
495. Verify import:
50 ```powershell
51 python -c "import clip; print('CLIP OK')"
52 ```
53
54## Procedure
55
56### 1. Load a model and preprocess function
57
58```python
59import torch
60import clip
61from PIL import Image
62
63device = "cuda" if torch.cuda.is_available() else "cpu"
64model, preprocess = clip.load("ViT-B/32", device=device)
65```
66
67**Available models (sorted by size):**
68
69| Model | Parameters | Speed | Quality |
70|----------|------------|-------|---------|
71| RN50 | 102M | Fast | Good |
72| RN101 | — | Medium| Good |
73| ViT-B/32 | 151M | Medium| Better (recommended) |
74| ViT-B/16 | — | Slower| Better |
75| ViT-L/14 | 428M | Slow | Best |
76
77```python
78# List all available models
79print(clip.available_models())
80```
81
82### 2. Zero-shot image classification
83
84```python
85import torch
86import clip
87from PIL import Image
88
89device = "cuda" if torch.cuda.is_available() else "cpu"
90model, preprocess = clip.load("ViT-B/32", device=device)
91
92# Load and preprocess image
93image = preprocess(Image.open("photo.jpg")).unsqueeze(0).to(device)
94
95# Define candidate labels (descriptive phrases work best)
96labels = ["a dog", "a cat", "a bird", "a car"]
97text = clip.tokenize(labels).to(device)
98
99# Compute similarity and probabilities
100with torch.no_grad():
101 logits_per_image, _ = model(image, text)
102 probs = logits_per_image.softmax(dim=-1).cpu().numpy()
103
104for label, prob in zip(labels, probs[0]):
105 print(f"{label}: {prob:.2%}")
106```
107
108### 3. Image-text similarity (cosine)
109
110```python
111# Encode and normalize
112with torch.no_grad():
113 image_features = model.encode_image(image)
114 text_features = model.encode_text(text)
115
116image_features /= image_features.norm(dim=-1, keepdim=True)
117text_features /= text_features.norm(dim=-1, keepdim=True)
118
119similarity = (image_features @ text_features.T).item()
120print(f"Similarity: {similarity:.4f}")
121```
122
123> **HARD RULE:** Always normalize embeddings before computing cosine similarity. Unnormalized dot products are not bounded to [-1, 1] and will produce misleading scores.
124
125### 4. Semantic image search (text → images)
126
127```python
128image_paths = ["img1.jpg", "img2.jpg", "img3.jpg"]
129image_embeddings = []
130
131for img_path in image_paths:
132 image = preprocess(Image.open(img_path)).unsqueeze(0).to(device)
133 with torch.no_grad():
134 embedding = model.encode_image(image)
135 embedding /= embedding.norm(dim=-1, keepdim=True)
136 image_embeddings.append(embedding)
137
138image_embeddings = torch.cat(image_embeddings)
139
140# Encode text query
141query = "a sunset over the ocean"
142text_input = clip.tokenize([query]).to(device)
143with torch.no_grad():
144 text_embedding = model.encode_text(text_input)
145 text_embedding /= text_embedding.norm(dim=-1, keepdim=True)
146
147# Rank images
148similarities = (text_embedding @ image_embeddings.T).squeeze(0)
149top_k = similarities.topk(3)
150
151for idx, score in zip(top_k.indices, top_k.values):
152 print(f"{image_paths[idx]}: {score:.3f}")
153```
154
155### 5. Content moderation
156
157```python
158categories = [
159 "safe for work",
160 "not safe for work",
161 "violent content",
162 "graphic content"
163]
164
165text = clip.tokenize(categories).to(device)
166
167with torch.no_grad():
168 logits_per_image, _ = model(image, text)
169 probs = logits_per_image.softmax(dim=-1)
170
171max_idx = probs.argmax().item()
172max_prob = probs[0, max_idx].item()
173print(f"Category: {categories[max_idx]} ({max_prob:.2%})")
174```
175
176### 6. Batch processing
177
178```python
179# Batch of images
180images = [preprocess(Image.open(f"img{i}.jpg")) for i in range(10)]
181images = torch.stack(images).to(device)
182
183with torch.no_grad():
184 image_features = model.encode_image(images)
185 image_features /= image_features.norm(dim=-1, keepdim=True)
186
187# Batch of texts
188texts = ["a dog", "a cat", "a bird"]
189text_tokens = clip.tokenize(texts).to(device)
190
191with torch.no_grad():
192 text_features = model.encode_text(text_tokens)
193 text_features /= text_features.norm(dim=-1, keepdim=True)
194
195# Similarity matrix: (10 images × 3 texts)
196similarities = image_features @ text_features.T
197print(similarities.shape) # torch.Size([10, 3])
198```
199
200### 7. Integration with a vector database (Chroma)
201
202```python
203import chromadb
204
205client = chromadb.Client()
206collection = client.create_collection("image_embeddings")
207
208# Add image embeddings
209for img_path, embedding in zip(image_paths, image_embeddings):
210 collection.add(
211 embeddings=[embedding.cpu().numpy().tolist()],
212 metadatas=[{"path": img_path}],
213 ids=[img_path]
214 )
215
216# Query with text
217query = "a sunset"
218text_embedding = model.encode_text(clip.tokenize([query]).to(device))
219text_embedding /= text_embedding.norm(dim=-1, keepdim=True)
220
221results = collection.query(
222 query_embeddings=[text_embedding.cpu().numpy().tolist()],
223 n_results=5
224)
225print(results)
226```
227
228### Best practices
229
2301. **Use ViT-B/32 for most cases** — best speed/quality balance.
2312. **Always normalize embeddings** — required for cosine similarity.
2323. **Batch processing** — encode multiple images/texts in one forward pass for efficiency.
2334. **Cache embeddings** — re-encoding is expensive; store normalized vectors.
2345. **Use descriptive labels** — "a photo of a golden retriever" beats "dog".
2356. **GPU recommended** — 10–50× faster than CPU.
2367. **Always use the provided `preprocess` function** — handles resize, center crop, normalization. Do not feed raw PIL images.
237
238## Pitfalls
239
2401. **Not for fine-grained tasks** — CLIP excels at broad categories; it struggles with sub-class distinctions (e.g., 200 bird species).
2412. **Vague labels perform poorly** — "animal" is weak; "a photo of a sleeping cat on a couch" is strong.
2423. **Dataset biases** — trained on web image-text pairs; may reflect societal and dataset biases. Do not use as sole arbiter for sensitive moderation decisions.
2434. **No bounding boxes** — CLIP operates on whole images only. Use a detection model (YOLO, DETR) for localization.
2445. **Limited spatial understanding** — weak at counting, positional reasoning, and reading text in images.
2456. **Forgetting to normalize** — the most common bug. Without L2 normalization, dot products are not cosine similarities.
2467. **Token length limit** — `clip.tokenize` truncates to 77 context tokens. Long prompts are silently cut; keep labels concise.
2478. **CPU-only latency** — image encoding ~200 ms on CPU vs ~20 ms on GPU. For production search, use GPU or pre-compute and cache.
248
249## Verification
250
2511. **Verify installation:**
252 ```powershell
253 python -c "import clip; print(clip.available_models())"
254 ```
255 Expected output includes: `['RN50', 'RN101', 'RN50x4', 'RN50x16', 'RN50x64', 'ViT-B/32', 'ViT-B/16', 'ViT-L/14', 'ViT-L/14@336px']`
256
2572. **Verify model loads and produces embeddings:**
258 ```python
259 import torch, clip
260 device = "cuda" if torch.cuda.is_available() else "cpu"
261 model, preprocess = clip.load("ViT-B/32", device=device)
262 text = clip.tokenize(["a test label"]).to(device)
263 with torch.no_grad():
264 feats = model.encode_text(text)
265 print(feats.shape) # torch.Size([1, 512])
266 ```
267
2683. **Verify zero-shot classification output format:**
269 ```python
270 probs = logits_per_image.softmax(dim=-1).cpu().numpy()
271 assert probs.shape == (1, len(labels))
272 assert abs(probs.sum() - 1.0) < 1e-5 # probabilities sum to 1
273 ```
274
2754. **Verify cosine similarity is bounded:**
276 ```python
277 image_features /= image_features.norm(dim=-1, keepdim=True)
278 text_features /= text_features.norm(dim=-1, keepdim=True)
279 sim = (image_features @ text_features.T).item()
280 assert -1.0 <= sim <= 1.0
281 ```
282
283## Performance Reference
284
285| Operation | CPU | GPU (V100) |
286|---------------------|----------|------------|
287| Image encoding | ~200 ms | ~20 ms |
288| Text encoding | ~50 ms | ~5 ms |
289| Similarity compute | <1 ms | <1 ms |
290
291## Resources
292
293- **GitHub:** https://github.com/openai/CLIP (25,300+ stars)
294- **Paper:** https://arxiv.org/abs/2103.00020
295- **Colab:** https://colab.research.google.com/github/openai/clip/
296- **License:** MIT