Multibanana Eval

multibanana-eval

MultiBanana: A Challenging Benchmark for Multi-Reference Text-to-Image Generation — Oshima et al. (2025) (arXiv:2511.22989, 2025)

What this evaluates

Evaluates text-to-image generation models on their ability to synthesize images from multiple reference images and text prompts. It probes adherence to complex instructions, consistency with reference attributes, and robustness to domain mismatches, scale discrepancies, rare concepts, and multilingual text.

Datasets

• MultiBanana — total ?; splits: test (-1); repo https://github.com/matsuolab/multibanana

Metrics

• MultiBanana score (primary) — range: percent
  • A composite score computed by LLM judges (Gemini-2.5 and GPT-5) evaluating generated images against reference images and prompts. It aggregates sub-scores including Instruction Alignment, Reference Consistency, background consistency, physical realism, and visual quality into a Total Score.

Input / output format

Input: A set of reference images (2 to 8) and a text prompt describing the target image.

Output: A single generated image.

Scoring recipe

def compute_multibanana_score(gen_img, ref_imgs, prompt):
    # LLM judges evaluate generation against references and prompt
    score_gemini = gemini_2_5_judge(gen_img, ref_imgs, prompt)
    score_gpt = gpt_5_judge(gen_img, ref_imgs, prompt)
    # Average across evaluators to get final MultiBanana score
    return (score_gemini + score_gpt) / 2

Common pitfalls

• OmniGen2 is restricted to a maximum of 6 reference images, unlike other models tested up to 8.
• Closed-source models may achieve high reference adherence but suffer from compositional collapse or over-crowding, while open-source models may omit subjects but maintain visual coherence.
• The benchmark relies entirely on LLM-as-a-judge evaluation, which may introduce bias or inconsistency compared to human evaluation.

Evidence (verbatim from paper)

As described in Section[3.4], the evaluation is conducted using Gemini-2.5 and GPT-5. In the main paper, we report the average scores from Gemini-2.5 and GPT-5, while the individual results for each evaluator are provided in the supplementary material. We compute the MultiBanana scores for each task type defined in Section[3.1], and present the results in [Table 3] and Section[E.1].

Citation

@misc{oshima2025multibanana,
  title={MultiBanana: A Challenging Benchmark for Multi-Reference Text-to-Image Generation},
  author={Oshima et al. (2025)},
  year={2025},
  note={arXiv:2511.22989}
}

• arXiv: 2511.22989