Echo Eval

echo-eval

Constantly Improving Image Models Need Constantly Improving Benchmarks — Ge et al. (2025) (arXiv:2510.15021, 2025)

What this evaluates

This benchmark probes an image generation model's ability to follow complex, real-world user prompts and produce high-quality outputs that preserve specific attributes like identity and color. It evaluates how well models handle non-standard, community-driven inputs and context-dependent instructions often found in social media discussions.

Datasets

• ECHO — total 31000; splits: benchmark (-1), analysis (-1), trash (-1)

Metrics

• quality_label (primary) — range: categorical
  • LLM-assigned categorical label ("Benchmark", "Analysis", or "Trash") based on prompt coherence, clear user intent, and data quality.
• identity preservation — range: percent
  • Assessed via VLM analysis to determine if the generated image maintains the subject's identity from the input or prompt context.
• color shift — range: percent
  • Measured by VLM to detect unintended color changes between the input/reference and the generated output.

Input / output format

Input: Text prompt (optionally reconstructed from social media reply trees), optional input image(s), and optional community feedback/comments.

Output: A single generated image corresponding to the provided prompt and context.

Scoring recipe

def score_echo(predictions, gold):
    results = []
    for pred, g in zip(predictions, gold):
        label = g.get('quality_label') or llm_classify_quality(g['prompt'], pred)
        id_pres = vlm_check_identity(g.get('input_images'), pred)
        color_shift = vlm_check_color(g.get('input_images'), pred)
        results.append({'label': label, 'id_pres': id_pres, 'color_shift': color_shift})
    return {
        'benchmark_rate': sum(1 for r in results if r['label'] == 'Benchmark') / len(results),
        'avg_id_pres': mean([r['id_pres'] for r in results]),
        'avg_color_shift': mean([r['color_shift'] for r in results])
    }

Common pitfalls

• Assuming input and output images are always explicitly labeled or appear in a fixed order; users often embed them in screenshots or collages without standard markers.
• Treating social media posts as isolated instances; prompts and context are frequently split across parent posts and reply trees, requiring recursive reconstruction.
• Relying solely on automated metrics without accounting for qualitative community feedback, which captures nuanced failures like identity drift or color shifts.

Evidence (verbatim from paper)

operationalizes qualitative community insights into measurable metrics like identity preservation and color shift—enabling adaptive, real-time evaluation of model capabilities.

Citation

@misc{ge2025echo,
  title={Constantly Improving Image Models Need Constantly Improving Benchmarks},
  author={Ge et al. (2025)},
  year={2025},
  note={arXiv:2510.15021}
}

• arXiv: 2510.15021