Echo 4o Eval

echo-4o-eval

Echo-4o: Harnessing the Power of GPT-4o Synthetic Images for Improved Image Generation — Junyan Ye et al. (arXiv:2508.09987, 2025)

What this evaluates

Evaluates text-to-image generation models on instruction-following accuracy, surreal/fantasy creativity, and multi-reference composition. It probes the model's ability to align complex textual prompts with visual outputs, handle long-tail attributes, and integrate multiple reference images.

Datasets

• GenEval — total ?; splits: test (-1)
• DPG-Bench — total ?; splits: test (-1)
• GenEval++ — total ?; splits: test (-1)
• Imagine-Bench — total ?; splits: test (-1)
• OmniContext — total ?; splits: test (-1)

Metrics

• GenEval Overall (primary) — range: [0, 1]
  • Proportion of prompts where all specified objects, attributes, colors, and spatial relationships are correctly generated, measured via object detection and attribute matching.
• DPG-Bench Overall — range: [0, 100]
  • LLM-based scoring across five categories (Global, Entity, Attribute, Relation, Other), averaged to produce a score out of 100.
• GenEval++ Overall — range: [0, 1]
  • Extended version of GenEval with more complex, long-text instructions and additional objects/attributes; scored as the proportion of fully correct generations.
• Imagine-Bench Overall — range: [0, 10]
  • LLM-based evaluation of surreal and imaginative generation across four categories (Attribute shift, Spatiotemporal, Hybridization, Multi-Object), averaged on a 0-10 scale.
• OmniContext Overall — range: [0, 10]
  • LLM-based scoring of multi-reference image generation fidelity across MULTIPLE and SCENE settings, averaged on a 0-10 scale.

Input / output format

Input: Text prompt describing the target image (and optionally reference images for OmniContext multi-reference tasks).

Output: A single RGB image generated by the model.

Scoring recipe

def evaluate(predictions, gold, benchmark):
    if benchmark == 'GenEval':
        correct = sum(1 for p, g in zip(predictions, gold) if check_attributes(p, g))
        return correct / len(gold)
    elif benchmark == 'DPG-Bench':
        scores = llm_score(predictions, gold, categories=['Global', 'Entity', 'Attribute', 'Relation', 'Other'])
        return mean(scores) * 100
    elif benchmark == 'GenEval++':
        return complex_attribute_match(predictions, gold)
    elif benchmark == 'Imagine-Bench':
        return llm_score_imagination(predictions, gold) / len(gold) * 10
    elif benchmark == 'OmniContext':
        return llm_score_multi_ref(predictions, gold, ref_images) / len(gold) * 10
    return 0

Common pitfalls

• GenEval++ tasks differ from GenEval by adding only 1-2 objects or attributes, but this minor change substantially raises difficulty, causing many models to fail completely.
• OmniContext requires native multi-reference support; many models lack this capability entirely or are incompatible with multi-reference scenarios.
• Metrics rely on automated LLM or detection-based scoring, which may not perfectly correlate with human aesthetic judgments or subtle instruction adherence.

Evidence (verbatim from paper)

We evaluate the instruction-following capability of our model on two widely used benchmarks—GenEval and DPG-Bench, as well as our newly proposed benchmark, GenEval++. As shown in Table[1], Echo-4o achieves a score of 0.89 on GenEval, outperforming prior state-of-the-art unified models such as Bagel and OmniGen2. On DPG-Bench (Table[2]), Echo-4o attains an overall score of 86.07, outperforming strong competitors including SD3 and UniWorld.

Citation

@misc{ye2025echo4o,
  title={Echo-4o: Harnessing the Power of GPT-4o Synthetic Images for Improved Image Generation},
  author={Junyan Ye et al.},
  year={2025},
  note={arXiv:2508.09987}
}

• arXiv: 2508.09987