qhjqhj00 contamination_rate
Dynamic Multimodal Evaluation with Flexible Complexity by Vision-Language Bootstrapping — Yang et al. (2024) (arXiv:2410.08695, 2024)
What this evaluates
Measures the extent to which multimodal evaluation benchmarks are contaminated by pre-training data, assessing both visual similarity and textual inference leakage to quantify data contamination risks.
Datasets
- SEEDBench — total ?; splits: eval (-1)
- MMBench — total ?; splits: eval (-1)
- MME — total ?; splits: eval (-1)
Metrics
image-only contamination rate(primary) — range: [0, 1]- Ratio of evaluation images with a CLIPScore exceeding 0.9 against any pre-training image to the total number of evaluation images.
image-text contamination rate— range: [0, 1]- Ratio of evaluation samples where the answer can be directly inferred from pre-training image captions (detected via GPT-4) to the total number of evaluation samples.
Input / output format
Input: Evaluation benchmark images and their corresponding questions/answers, compared against pre-training dataset images and captions.
Output: A binary contamination flag per image or image-text pair, aggregated into a contamination rate.
Scoring recipe
def calculate_contamination(eval_images, eval_samples, train_images, train_captions):
contaminated_img_count = 0
for img in eval_images:
if max(clip_score(img, t_img) for t_img in train_images) > 0.9:
contaminated_img_count += 1
img_only_rate = contaminated_img_count / len(eval_images)
contaminated_pair_count = 0
for q, a, img in eval_samples:
if gpt4_inferred(a, train_captions):
contaminated_pair_count += 1
img_text_rate = contaminated_pair_count / len(eval_samples)
return img_only_rate, img_text_rate
Common pitfalls
- The CLIPScore threshold of 0.9 is a heuristic and may miss semantically similar but visually distinct images.
- GPT-4 inference for text contamination is non-deterministic and heavily depends on the prompt design used for caption comparison.
- Analysis is limited to the maximum pre-training dataset size checked (100M), which underestimates actual contamination in modern large-scale models.
Evidence (verbatim from paper)
We adopt 0.9 as the threshold to determine visual contamination. The image-only contamination rate is calculated as the ratio of the number of contaminated images and the number of total images in the evaluation set.
Citation
@misc{yang2024dynamicmultimodalevaluation,
title={Dynamic Multimodal Evaluation with Flexible Complexity by Vision-Language Bootstrapping},
author={Yang et al. (2024)},
year={2024},
note={arXiv:2410.08695}
}
- arXiv: 2410.08695