Evalmuse 40k Eval

evalmuse-40k-eval

EvalMuse-40K: A Reliable and Fine-Grained Benchmark with Comprehensive Human Annotations for Text-to-Image Generation Model Evaluation — Han et al. (2024) (arXiv:2412.18150, 2024)

What this evaluates

Evaluates the fine-grained alignment and structural fidelity between generated images and their corresponding text prompts. It probes a model's ability to match specific visual elements (e.g., objects, colors, counts) and overall composition against human-annotated ground truth.

Datasets

• EvalMuse-40K — total 40000; splits: train (30000), test (1000)

Metrics

• SRCC (primary) — range: [-1, 1]
  • Spearman Rank Correlation Coefficient measuring the monotonic relationship between predicted alignment scores and human annotations.
• Accuracy — range: percent
  • Percentage of correctly predicted element-wise matches (or mismatches) in fine-grained evaluation.

Input / output format

Input: Image-text pair, optionally accompanied by element-level annotations or structural labels for fine-grained evaluation.

Output: Continuous alignment score (0–1) for overall evaluation, or element-wise match probability/label for fine-grained evaluation.

Scoring recipe

# Overall alignment (SRCC)
preds = [model.predict(img, txt) for img, txt in test_set]
gold = [human_score for _, human_score in test_set]
srcc = spearmanr(preds, gold).correlation

# Fine-grained evaluation (Accuracy)
preds_elem = [model.predict_element(img, elem_prompt) for img, elem_prompt in test_set]
gold_elem = [elem_label for _, elem_label in test_set]
acc = sum(p == g for p, g in zip(preds_elem, gold_elem)) / len(gold_elem)

Common pitfalls

• Confusing overall alignment scoring with fine-grained element-wise scoring; the benchmark requires separate evaluation pipelines and metrics for each.
• Failing to apply the fixed-step (0.01) threshold search for binary classification methods, which is explicitly required to maximize overall accuracy.
• Using prompts that overlap between training and test splits, violating the no-overlap constraint and artificially inflating correlation metrics.

Evidence (verbatim from paper)

For overall alignment scores, we compare FGA-BLIP2 with the state-of-the-art models and report the Spearman Rank Correlation Coefficient (SRCC) and Pearson Linear Correlation Coefficient (PLCC) to measure the correlation between model predictions and human annotations. For fine-grained evaluation, we compare PN-VQA with TIFA and VQ2, and use several advanced MLLMs for the VQA task. On one hand, we average the fine-grained scores of the image-text pairs and compare them with the overall alignment scores from human annotations. On the other hand, we conduct fine-grained evaluation by reporting the accuracy of the method’s element-wise predictions.

Citation

@misc{han2024evalmuse40k,
  title={EvalMuse-40K: A Reliable and Fine-Grained Benchmark with Comprehensive Human Annotations for Text-to-Image Generation Model Evaluation},
  author={Han et al. (2024)},
  year={2024},
  note={arXiv:2412.18150}
}

• arXiv: 2412.18150