Seed X Eval

seed-x-eval

SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generation — Ge et al. (2024) (arXiv:2404.14396, 2024)

What this evaluates

Evaluates a multimodal model's ability to understand images and text (comprehension) and generate images from text instructions (generation). It probes fine-grained visual perception, reasoning, and compositional image synthesis.

Datasets

• VQAv2 — total ?; splits: test (-1)
• GQA — total ?; splits: test (-1)
• POPE — total ?; splits: test (-1)
• MME — total ?; splits: test (-1)
• SEED — total ?; splits: test (-1)
• MMB — total ?; splits: test (-1)
• MM-Vet — total ?; splits: test (-1)
• MMMU — total ?; splits: test (-1)
• GenEval — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • Standard accuracy or benchmark-specific scoring metric used across all listed benchmarks. For GenEval, it measures the percentage of generated images that correctly satisfy all compositional constraints in the prompt.

Input / output format

Input: Image and text prompt (for comprehension benchmarks), or text prompt only (for GenEval generation benchmark).

Output: Text answer (for comprehension), or generated image (for GenEval).

Scoring recipe

def compute_accuracy(predictions, gold):
    correct = 0
    for pred, gold_item in zip(predictions, gold):
        if evaluate_match(pred, gold_item):
            correct += 1
    return (correct / len(gold)) * 100

def evaluate_match(pred, gold):
    # For comprehension: exact string match or normalized answer match
    # For GenEval: strict attribute-level verification of generated image against prompt constraints
    return pred == gold

Common pitfalls

• Benchmarks like MME and MM-Vet contain diverse sub-tasks (e.g., perception, reasoning, OCR) that should be reported separately rather than averaged into a single score.
• GenEval uses strict attribute-level verification; missing a single constraint (e.g., color, position, or count) causes the entire instance to be marked incorrect.
• Models may rely on dataset memorization or shortcut heuristics rather than true visual reasoning, especially on VQAv2 and GQA.

Evidence (verbatim from paper)

We evaluate the multimodal comprehension capabilities of SEED-X-I on widely recognized image-based vision-language benchmarks, which include VQAv2 [43], GQA [44], POPE [45], MME [46], SEED [47], MMB [48], MM-Vet [49], and MMMU [50]. As listed in Tab. 2, SEED-X-I achieves competitive performance across various benchmarks, even when compared to MLLMs specifically designed for multimodal comprehension. Image Generation. We evaluate the image generation capabilities of SEED-X-I on GenEval [51], which is a challenging benchmark to evaluate compositional image properties such as object cooccurrence, position, count, and color. As shown in Tab. 3, SEED-X obtains 51% overall accuracy, demonstrating the model's excellent instruction-following capabilities for image generation.

Citation

@misc{ge2024seedx,
  title={SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generation},
  author={Ge et al. (2024)},
  year={2024},
  note={arXiv:2404.14396}
}

• arXiv: 2404.14396