Clip

CLIP — Contrastive Language-Image Pre-Training

OpenAI's CLIP connects vision and language in a shared embedding space, enabling zero-shot image classification, image-text similarity, semantic image search, and cross-modal retrieval without fine-tuning. Trained on 400M image-text pairs; matches ResNet-50 zero-shot on ImageNet.

When to Use

Use CLIP when you need:

• Zero-shot image classification (no training data required)
• Image-text similarity or matching scores
• Semantic image search (text query → ranked images)
• Content moderation (detect NSFW, violence, graphic content)
• Cross-modal retrieval (image→text, text→image)
• Visual question answering (broad category level)

Use alternatives instead:

• BLIP-2 — better image captioning
• LLaVA — vision-language conversational chat
• Segment Anything (SAM) — pixel-level image segmentation

Prerequisites

1. Python 3.8+ installed on the host.
2. PyTorch with CUDA support if a GPU is available (10–50× faster). CPU works but is slower.
3. On Windows (PowerShell), ensure python and pip are on PATH:

   python --version
   pip --version
   

4. Install the OpenAI CLIP package and dependencies:

   pip install git+https://github.com/openai/CLIP.git
   pip install torch torchvision ftfy regex tqdm
   

5. Verify import:

   python -c "import clip; print('CLIP OK')"
   

Procedure

1. Load a model and preprocess function

import torch
import clip
from PIL import Image

device = "cuda" if torch.cuda.is_available() else "cpu"
model, preprocess = clip.load("ViT-B/32", device=device)

Available models (sorted by size):

# List all available models
print(clip.available_models())

2. Zero-shot image classification

import torch
import clip
from PIL import Image

device = "cuda" if torch.cuda.is_available() else "cpu"
model, preprocess = clip.load("ViT-B/32", device=device)

# Load and preprocess image
image = preprocess(Image.open("photo.jpg")).unsqueeze(0).to(device)

# Define candidate labels (descriptive phrases work best)
labels = ["a dog", "a cat", "a bird", "a car"]
text = clip.tokenize(labels).to(device)

# Compute similarity and probabilities
with torch.no_grad():
    logits_per_image, _ = model(image, text)
    probs = logits_per_image.softmax(dim=-1).cpu().numpy()

for label, prob in zip(labels, probs[0]):
    print(f"{label}: {prob:.2%}")

3. Image-text similarity (cosine)

# Encode and normalize
with torch.no_grad():
    image_features = model.encode_image(image)
    text_features = model.encode_text(text)

image_features /= image_features.norm(dim=-1, keepdim=True)
text_features /= text_features.norm(dim=-1, keepdim=True)

similarity = (image_features @ text_features.T).item()
print(f"Similarity: {similarity:.4f}")

HARD RULE: Always normalize embeddings before computing cosine similarity. Unnormalized dot products are not bounded to [-1, 1] and will produce misleading scores.

4. Semantic image search (text → images)

image_paths = ["img1.jpg", "img2.jpg", "img3.jpg"]
image_embeddings = []

for img_path in image_paths:
    image = preprocess(Image.open(img_path)).unsqueeze(0).to(device)
    with torch.no_grad():
        embedding = model.encode_image(image)
        embedding /= embedding.norm(dim=-1, keepdim=True)
    image_embeddings.append(embedding)

image_embeddings = torch.cat(image_embeddings)

# Encode text query
query = "a sunset over the ocean"
text_input = clip.tokenize([query]).to(device)
with torch.no_grad():
    text_embedding = model.encode_text(text_input)
    text_embedding /= text_embedding.norm(dim=-1, keepdim=True)

# Rank images
similarities = (text_embedding @ image_embeddings.T).squeeze(0)
top_k = similarities.topk(3)

for idx, score in zip(top_k.indices, top_k.values):
    print(f"{image_paths[idx]}: {score:.3f}")

5. Content moderation

categories = [
    "safe for work",
    "not safe for work",
    "violent content",
    "graphic content"
]

text = clip.tokenize(categories).to(device)

with torch.no_grad():
    logits_per_image, _ = model(image, text)
    probs = logits_per_image.softmax(dim=-1)

max_idx = probs.argmax().item()
max_prob = probs[0, max_idx].item()
print(f"Category: {categories[max_idx]} ({max_prob:.2%})")

6. Batch processing

# Batch of images
images = [preprocess(Image.open(f"img{i}.jpg")) for i in range(10)]
images = torch.stack(images).to(device)

with torch.no_grad():
    image_features = model.encode_image(images)
    image_features /= image_features.norm(dim=-1, keepdim=True)

# Batch of texts
texts = ["a dog", "a cat", "a bird"]
text_tokens = clip.tokenize(texts).to(device)

with torch.no_grad():
    text_features = model.encode_text(text_tokens)
    text_features /= text_features.norm(dim=-1, keepdim=True)

# Similarity matrix: (10 images × 3 texts)
similarities = image_features @ text_features.T
print(similarities.shape)  # torch.Size([10, 3])

7. Integration with a vector database (Chroma)

import chromadb

client = chromadb.Client()
collection = client.create_collection("image_embeddings")

# Add image embeddings
for img_path, embedding in zip(image_paths, image_embeddings):
    collection.add(
        embeddings=[embedding.cpu().numpy().tolist()],
        metadatas=[{"path": img_path}],
        ids=[img_path]
    )

# Query with text
query = "a sunset"
text_embedding = model.encode_text(clip.tokenize([query]).to(device))
text_embedding /= text_embedding.norm(dim=-1, keepdim=True)

results = collection.query(
    query_embeddings=[text_embedding.cpu().numpy().tolist()],
    n_results=5
)
print(results)

Best practices

1. Use ViT-B/32 for most cases — best speed/quality balance.
2. Always normalize embeddings — required for cosine similarity.
3. Batch processing — encode multiple images/texts in one forward pass for efficiency.
4. Cache embeddings — re-encoding is expensive; store normalized vectors.
5. Use descriptive labels — "a photo of a golden retriever" beats "dog".
6. GPU recommended — 10–50× faster than CPU.
7. Always use the provided preprocess function — handles resize, center crop, normalization. Do not feed raw PIL images.

Pitfalls

1. Not for fine-grained tasks — CLIP excels at broad categories; it struggles with sub-class distinctions (e.g., 200 bird species).
2. Vague labels perform poorly — "animal" is weak; "a photo of a sleeping cat on a couch" is strong.
3. Dataset biases — trained on web image-text pairs; may reflect societal and dataset biases. Do not use as sole arbiter for sensitive moderation decisions.
4. No bounding boxes — CLIP operates on whole images only. Use a detection model (YOLO, DETR) for localization.
5. Limited spatial understanding — weak at counting, positional reasoning, and reading text in images.
6. Forgetting to normalize — the most common bug. Without L2 normalization, dot products are not cosine similarities.
7. Token length limit — clip.tokenize truncates to 77 context tokens. Long prompts are silently cut; keep labels concise.
8. CPU-only latency — image encoding ~200 ms on CPU vs ~20 ms on GPU. For production search, use GPU or pre-compute and cache.

Verification

1. Verify installation:

   python -c "import clip; print(clip.available_models())"
   

   Expected output includes: ['RN50', 'RN101', 'RN50x4', 'RN50x16', 'RN50x64', 'ViT-B/32', 'ViT-B/16', 'ViT-L/14', 'ViT-L/14@336px']

2. Verify model loads and produces embeddings:

   import torch, clip
   device = "cuda" if torch.cuda.is_available() else "cpu"
   model, preprocess = clip.load("ViT-B/32", device=device)
   text = clip.tokenize(["a test label"]).to(device)
   with torch.no_grad():
       feats = model.encode_text(text)
   print(feats.shape)  # torch.Size([1, 512])
   

3. Verify zero-shot classification output format:

   probs = logits_per_image.softmax(dim=-1).cpu().numpy()
   assert probs.shape == (1, len(labels))
   assert abs(probs.sum() - 1.0) < 1e-5  # probabilities sum to 1
   

4. Verify cosine similarity is bounded:

   image_features /= image_features.norm(dim=-1, keepdim=True)
   text_features /= text_features.norm(dim=-1, keepdim=True)
   sim = (image_features @ text_features.T).item()
   assert -1.0 <= sim <= 1.0
   

Performance Reference

Resources

• GitHub: https://github.com/openai/CLIP (25,300+ stars)
• Paper: https://arxiv.org/abs/2103.00020
• Colab: https://colab.research.google.com/github/openai/clip/
• License: MIT