Clip Aware Embeddings

CLIP-Aware Image Embeddings

Smart image-text matching that knows when CLIP works and when to use alternatives.

Quick Decision Tree

Your task:
├─ Semantic search ("find beach images") → CLIP ✓
├─ Zero-shot classification (broad categories) → CLIP ✓
├─ Counting objects → DETR, Faster R-CNN ✗
├─ Fine-grained ID (celebrities, car models) → Specialized model ✗
├─ Spatial relations ("cat left of dog") → GQA, SWIG ✗
└─ Compositional ("red car AND blue truck") → DCSMs, PC-CLIP ✗

When to Use This Skill

✅ Use for:

• Semantic image search
• Broad category classification
• Image similarity matching
• Zero-shot tasks on new categories

❌ Do NOT use for:

• Counting objects in images
• Fine-grained classification
• Spatial understanding
• Attribute binding
• Negation handling

Installation

pip install transformers pillow torch sentence-transformers --break-system-packages

Validation: Run python scripts/validate_setup.py

Basic Usage

Image Search

from transformers import CLIPProcessor, CLIPModel
from PIL import Image

model = CLIPModel.from_pretrained("openai/clip-vit-large-patch14")
processor = CLIPProcessor.from_pretrained("openai/clip-vit-large-patch14")

# Embed images
images = [Image.open(f"img{i}.jpg") for i in range(10)]
inputs = processor(images=images, return_tensors="pt")
image_features = model.get_image_features(**inputs)

# Search with text
text_inputs = processor(text=["a beach at sunset"], return_tensors="pt")
text_features = model.get_text_features(**text_inputs)

# Compute similarity
similarity = (image_features @ text_features.T).softmax(dim=0)

Common Anti-Patterns

Anti-Pattern 1: "CLIP for Everything"

❌ Wrong:

# Using CLIP to count cars in an image
prompt = "How many cars are in this image?"
# CLIP cannot count - it will give nonsense results

Why wrong: CLIP's architecture collapses spatial information into a single vector. It literally cannot count.

✓ Right:

from transformers import DetrImageProcessor, DetrForObjectDetection

processor = DetrImageProcessor.from_pretrained("facebook/detr-resnet-50")
model = DetrForObjectDetection.from_pretrained("facebook/detr-resnet-50")

# Detect objects
results = model(**processor(images=image, return_tensors="pt"))
# Filter for cars and count
car_detections = [d for d in results if d['label'] == 'car']
count = len(car_detections)

How to detect: If query contains "how many", "count", or numeric questions → Use object detection

Anti-Pattern 2: Fine-Grained Classification

❌ Wrong:

# Trying to identify specific celebrities with CLIP
prompts = ["Tom Hanks", "Brad Pitt", "Morgan Freeman"]
# CLIP will perform poorly - not trained for fine-grained face ID

Why wrong: CLIP trained on coarse categories. Fine-grained faces, car models, flower species require specialized models.

✓ Right:

# Use a fine-tuned face recognition model
from transformers import AutoFeatureExtractor, AutoModelForImageClassification

model = AutoModelForImageClassification.from_pretrained(
    "microsoft/resnet-50"  # Then fine-tune on celebrity dataset
)
# Or use dedicated face recognition: ArcFace, CosFace

How to detect: If query asks to distinguish between similar items in same category → Use specialized model

Anti-Pattern 3: Spatial Understanding

❌ Wrong:

# CLIP cannot understand spatial relationships
prompts = [
    "cat to the left of dog",
    "cat to the right of dog"
]
# Will give nearly identical scores

Why wrong: CLIP embeddings lose spatial topology. "Left" and "right" are treated as bag-of-words.

✓ Right:

# Use a spatial reasoning model
# Examples: GQA models, Visual Genome models, SWIG
from swig_model import SpatialRelationModel

model = SpatialRelationModel()
result = model.predict_relation(image, "cat", "dog")
# Returns: "left", "right", "above", "below", etc.

How to detect: If query contains directional words (left, right, above, under, next to) → Use spatial model

Anti-Pattern 4: Attribute Binding

❌ Wrong:

prompts = [
    "red car and blue truck",
    "blue car and red truck"
]
# CLIP often gives similar scores for both

Why wrong: CLIP cannot bind attributes to objects. It sees "red, blue, car, truck" as a bag of concepts.

✓ Right - Use PC-CLIP or DCSMs:

# PC-CLIP: Fine-tuned for pairwise comparisons
from pc_clip import PCCLIPModel

model = PCCLIPModel.from_pretrained("pc-clip-vit-l")
# Or use DCSMs (Dense Cosine Similarity Maps)

How to detect: If query has multiple objects with different attributes → Use compositional model

Evolution Timeline

2021: CLIP Released

• Revolutionary: zero-shot, 400M image-text pairs
• Widely adopted for everything
• Limitations not yet understood

2022-2023: Limitations Discovered

• Cannot count objects
• Poor at fine-grained classification
• Fails spatial reasoning
• Can't bind attributes

2024: Alternatives Emerge

• DCSMs: Preserve patch/token topology
• PC-CLIP: Trained on pairwise comparisons
• SpLiCE: Sparse interpretable embeddings

2025: Current Best Practices

• Use CLIP for what it's good at
• Task-specific models for limitations
• Compositional models for complex queries

LLM Mistake: LLMs trained on 2021-2023 data will suggest CLIP for everything because limitations weren't widely known. This skill corrects that.

Validation Script

Before using CLIP, check if it's appropriate:

python scripts/validate_clip_usage.py \
    --query "your query here" \
    --check-all

Returns:

• ✅ CLIP is appropriate
• ❌ Use alternative (with suggestion)

Task-Specific Guidance

Image Search (CLIP ✓)

# Good use of CLIP
queries = ["beach", "mountain", "city skyline"]
# Works well for broad semantic concepts

Zero-Shot Classification (CLIP ✓)

# Good: Broad categories
categories = ["indoor", "outdoor", "nature", "urban"]
# CLIP excels at this

Object Counting (CLIP ✗)

# Use object detection instead
from transformers import DetrImageProcessor, DetrForObjectDetection
# See /references/object_detection.md

Fine-Grained Classification (CLIP ✗)

# Use specialized models
# See /references/fine_grained_models.md

Spatial Reasoning (CLIP ✗)

# Use spatial relation models
# See /references/spatial_models.md

Troubleshooting

Issue: CLIP gives unexpected results

Check:

1. Is this a counting task? → Use object detection
2. Fine-grained classification? → Use specialized model
3. Spatial query? → Use spatial model
4. Multiple objects with attributes? → Use compositional model

Validation:

python scripts/diagnose_clip_issue.py --image path/to/image --query "your query"

Issue: Low similarity scores

Possible causes:

1. Query too specific (CLIP works better with broad concepts)
2. Fine-grained task (not CLIP's strength)
3. Need to adjust threshold

Solution: Try broader query or use alternative model

Model Selection Guide

Performance Notes

CLIP models:

• ViT-B/32: Fast, lower quality
• ViT-L/14: Balanced (recommended)
• ViT-g-14: Highest quality, slower

Inference time (single image, CPU):

• ViT-B/32: ~100ms
• ViT-L/14: ~300ms
• ViT-g-14: ~1000ms

Further Reading

• /references/clip_limitations.md - Detailed analysis of CLIP's failures
• /references/alternatives.md - When to use what model
• /references/compositional_reasoning.md - DCSMs and PC-CLIP deep dive
• /scripts/validate_clip_usage.py - Pre-flight validation tool
• /scripts/diagnose_clip_issue.py - Debug unexpected results

Changelog

v1.2.0 (2025-03-15)

• Added DCSMs and PC-CLIP alternatives
• Updated for 2025 best practices
• Improved validation scripts

v1.1.0 (2024-06-10)

• Added anti-pattern detection
• Expanded troubleshooting

v1.0.0 (2024-01-15)

• Initial release