Svo Probes Eval

svo-probes-eval

Scalable Performance Analysis for Vision-Language Models — Castro et al. (2023) (arXiv:2305.18786, 2023)

What this evaluates

Evaluates how well CLIP aligns text and images by measuring its preference for positive over negative image-text pairs, and analyzes how semantic features (part-of-speech, concreteness, length, frequency, ambiguity) influence this alignment.

Datasets

• SVO-Probes — total ?; splits: test (-1); repo https://github.com/MichiganNLP/Scalable-VLM-Probing

Metrics

• accuracy (primary) — range: percent
  • Computed as the percentage of instances where CLIP assigns a higher similarity score to the positive image-text pair than to the negative image-text pair. Evaluated separately for subjects, objects, and verbs.
• score difference (D) — range: other
  • D = P - N, where P is the CLIP similarity score for the positive pair and N is the score for the negative pair. Used to measure relative preference strength and feature importance via mean difference.

Input / output format

Input: Pairs of images and text captions (one positive match, one negative mismatch) fed into CLIP to compute similarity scores.

Output: Continuous similarity scores from CLIP for each image-text pair.

Scoring recipe

P = clip_similarity(image_positive, text_positive)
N = clip_similarity(image_negative, text_negative)
D = P - N
is_correct = 1 if P > N else 0

# Accuracy per POS category:
accuracy = sum(is_correct for category in POS) / total_instances_in_category

# Feature importance:
feature_importance = mean(D when feature_present) - mean(D when feature_absent)

Common pitfalls

• Treating the negative image as purely random rather than adversarial, as it shares common elements with the positive image, which inflates N scores.
• Interpreting high absolute CLIP scores as good semantic understanding, when the model often behaves like a bag-of-words model and scores both positive and negative pairs highly if they share vocabulary.
• Confusing the relative preference metric D with absolute retrieval accuracy; D measures preference strength, not correctness probability.

Evidence (verbatim from paper)

When computing the number of times CLIP assigns a higher score to the similarity between the text and the positive image as compared to the similarity between the text and the negative image, the verbs obtain 81.45% accuracy while the subjects get 86.87% and the objects 88.78%.

Citation

@misc{castro2023scalable,
  title={Scalable Performance Analysis for Vision-Language Models},
  author={Castro et al. (2023)},
  year={2023},
  note={arXiv:2305.18786}
}

• arXiv: 2305.18786