Creation Mmbench Eval

creation-mmbench-eval

Creation-MMBench: Assessing Context-Aware Creative Intelligence in MLLM — Fang et al. (2025) (arXiv:2503.14478, 2025)

What this evaluates

Evaluates context-aware creative intelligence in multimodal and text-only models by assessing their ability to generate creative, contextually relevant content while maintaining visual factuality across diverse functional and creative writing tasks.

Datasets

• Creation-MMBench — total ?; splits: (unstated); repo https://github.com/open-compass/Creation-MMBench

Metrics

• VFS (primary) — range: other
  • Visual Factuality Score assigned by GPT-4o judging the factual accuracy of visual elements in the generated response relative to the input image/context.
• Reward (primary) — range: other
  • Creative alignment score assigned by GPT-4o judging the creativity, contextual relevance, and human preference alignment of the generated content.
• MAE — range: other
  • Mean Absolute Error between model-as-judge scores and human pairwise comparison scores, used to measure alignment with human preference.
• Cons. — range: percent
  • Consistency percentage between model-as-judge decisions and human preferences in pairwise comparisons.

Input / output format

Input: Multimodal input consisting of one or more images and a text prompt/question describing a creative task. For the text-only variant (Creation-MMBench-TO), images are replaced by GPT-4o-generated textual descriptions.

Output: Text-only creative response generated by the model.

Scoring recipe

# Dual Evaluation Strategy using GPT-4o as judge
def score_response(response, input_image, prompt):
    # Step 1: Visual Factuality Score (VFS)
    vfs_prompt = f"Judge the visual factuality of this response given the image and prompt. Score: [0-10]"
    vfs_score = gpt4o_judge(vfs_prompt, response, input_image)
    
    # Step 2: Creative Reward Score
    reward_prompt = f"Judge the creativity and contextual relevance of this response. Score: [0-10]"
    reward_score = gpt4o_judge(reward_prompt, response, input_image)
    
    return vfs_score, reward_score

Common pitfalls

• Negative Reward scores are common and indicate performance degradation, often due to visual instruction tuning or lack of creative training data, rather than a scoring error.
• Proprietary models are evaluated against themselves (using their own image descriptions), which may inflate scores compared to open-source models evaluated with GPT-4o descriptions.
• High objective performance (e.g., OC Score) does not correlate with high creative/reward scores, requiring separate evaluation for creativity.

Evidence (verbatim from paper)

Table 3: LLM performance on Creation-MMBench-TO and Visual Instruction Tuning Impact on VLM creation capability. ... | VFS | Reward | ... Finally, we selected Dual Evaluation, and GPT-4o as the evaluation strategy for Creation-MMBench.

Citation

@misc{fang2025creationmmbench,
  title={Creation-MMBench: Assessing Context-Aware Creative Intelligence in MLLM},
  author={Fang et al. (2025)},
  year={2025},
  note={arXiv:2503.14478}
}

• arXiv: 2503.14478