Splice Eval

splice-eval

Interpreting CLIP with Sparse Linear Concept Embeddings (SpLiCE) — Bhalla et al. (2024) (arXiv:2402.10376, 2024)

What this evaluates

Evaluates the ability of a sparse linear decomposition method (SpLiCE) to reconstruct CLIP image embeddings while preserving semantic interpretability and downstream task performance. It probes how well concept-based representations align with human captions and zero-shot classification benchmarks compared to random or learned baselines.

Datasets

• CIFAR100 — total ?; splits: test (-1)
• MIT States — total ?; splits: test (-1)
• CelebA — total ?; splits: test (-1)
• MSCOCO — total ?; splits: test (-1)
• ImageNetVal — total ?; splits: val (-1)

Metrics

• cosine similarity (primary) — range: [0, 1]
  • Cosine similarity between the reconstructed SpLiCE embedding and the original CLIP embedding.
• zero-shot accuracy — range: [0, 1]
  • Accuracy of a linear probe or classifier trained on SpLiCE concept weights for zero-shot classification using the prompt template 'A photo of a {}'.
• semantic relevance — range: other
  • Hausdorff distance between the set of non-zero SpLiCE concepts and the set of CLIP embeddings of tokens from human-generated captions (after stop-word filtering).
• l0 norm — range: other
  • Number of non-zero concept weights in the sparse decomposition, targeted to be between 5 and 20.

Input / output format

Input: Dense image embeddings from OpenCLIP ViT-B/32, centered and normalized.

Output: Sparse weight vector over a 15,000-concept dictionary (LAION tokens), with non-negative values and l0 norm between 5 and 20.

Scoring recipe

# Cosine Similarity
cos_sim = cosine_similarity(reconstructed_embedding, original_clip_embedding)

# Zero-shot Accuracy
logits = reconstructed_embedding @ class_prompts
pred = argmax(logits)
acc = (pred == ground_truth_labels).mean()

# Semantic Relevance
caption_tokens = tokenize_and_filter_stopwords(human_captions)
caption_embeds = clip_embed(caption_tokens)
splice_concepts = clip_embed(non_zero_concepts_from_decomposition)
hausdorff = max(dissimilarity(set(splice_concepts), set(caption_embeds)),
                dissimilarity(set(caption_embeds), set(splice_concepts)))

Common pitfalls

• Cosine similarity reconstruction is intentionally low because SpLiCE only captures semantics, discarding non-semantic 'noise' present in CLIP embeddings.
• The method is task-agnostic and requires no training or labeled data; performance relies entirely on the quality of the overcomplete concept dictionary.
• Negative concept weights are explicitly disabled (non-negativity flag) to maintain semantic relevance, even though they marginally improve cosine reconstruction.

Evidence (verbatim from paper)

We assess the performance of SpLiCE decompositions by evaluating the reconstruction error in terms of cosine similarity between SpLiCE representations and CLIP embeddings, the zero-shot performance of SpLiCE decompositions, and the retrieval performance of SpLiCE embeddings. ... We compute semantic relevance by tokenizing and filtering stop-words from the MSCOCO human-generated captions and embedding each token with CLIP. Then, we take all non-zero concepts output by SpLiCE and compute the Hausdorff distance between the sets of SpLiCE concepts and caption token embeddings.

Citation

@misc{bhalla2024splice,
  title={Interpreting CLIP with Sparse Linear Concept Embeddings (SpLiCE)},
  author={Bhalla et al. (2024)},
  year={2024},
  note={arXiv:2402.10376}
}

• arXiv: 2402.10376