Dreamomni2 Eval

dreamomni2-eval

DreamOmni2: Multimodal Instruction-based Editing and Generation — Bin Xia et al. (arXiv:2510.06679, 2025)

What this evaluates

Evaluates a model's ability to perform multimodal instruction-based image editing and generation, specifically testing adherence to text instructions while manipulating concrete objects and abstract attributes (e.g., texture, style) using multiple reference images.

Datasets

• DreamOmni2 benchmark — total ?; splits: test (-1); repo https://github.com/dvlab-research/DreamOmni2

Metrics

• success editing ratio (primary) — range: [0, 1]
  • The proportion of successfully edited or generated images out of the total test cases. Calculated as (number of successful cases) / (total cases). Evaluated separately for concrete objects and abstract attributions using VLM judges (Gemini 2.5, Doubao 1.6) and human experts.

Input / output format

Input: A set of reference images (one or multiple) and a natural language instruction specifying the desired edit or generation (e.g., changing an object's attribute, adding an object, or modifying style).

Output: A single edited or generated image matching the instruction.

Scoring recipe

def calculate_success_rate(predictions, instructions, evaluator):
    success_count = 0
    for pred_img, instruction in zip(predictions, instructions):
        # Evaluator (VLM or human) checks if pred_img matches instruction
        # and maintains consistency with reference images
        is_success = evaluator.evaluate(pred_img, instruction)
        if is_success:
            success_count += 1
    return success_count / len(predictions)

Common pitfalls

• VLM evaluators may struggle with abstract attributes (e.g., texture, style) and can introduce visual biases (e.g., color shifts like yellowing).
• The metric is a binary success ratio rather than a continuous quality score, so it does not capture partial compliance or subtle aesthetic degradation.
• Human evaluation is used as a proxy for ground truth, but inter-rater agreement and specific rubric details are not provided.

Evidence (verbatim from paper)

The models were evaluated for success rates by Gemini 2.5*(Google, [2025a])* and Doubao 1.6*(ByteDance, [2025]), and several professional engineers manually assessed the results. Table 3: Quantitative comparison of multimodal instruction-based generation. We use Gemini(Google, [2025a])* and Doubao*(ByteDance, [2025])* to evaluate the success editing ratio on concrete objects and abstract attributions, respectively. In addition, “Human” refers to professional engineers assessing the editing success rates of all models.

Citation

@misc{xia2025dreamomni2,
  title={DreamOmni2: Multimodal Instruction-based Editing and Generation},
  author={Bin Xia et al.},
  year={2025},
  note={arXiv:2510.06679}
}

• arXiv: 2510.06679