Stablei2i Eval

stablei2i-eval

StableI2I: Spotting Unintended Changes in Image-to-Image Transition — Li et al. (2026) (arXiv:2605.04453, 2026)

What this evaluates

Evaluates multimodal models' ability to detect unintended content, structural, and low-level appearance changes in image-to-image transitions. It probes fine-grained visual reasoning and pixel-level alignment capabilities by asking models to assess fidelity across three distinct dimensions.

Datasets

• StableI2I-Bench — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Proportion of correctly predicted answers (and problem types for Binary & Type QA) out of the total number of samples. Reported per dimension (Structure, Semantic, Low-level) and as an average.

Input / output format

Input: Image pair $(I_{in}, I_{out})$ and an I2I control instruction $x$, formatted within a fixed task template/prompt.

Output: QA response indicating fidelity status (e.g., 'Yes'/'No' for error presence) and a problem type description.

Scoring recipe

def compute_accuracy(predictions, gold):
    correct = 0
    for pred, g_ans, g_type in zip(predictions, gold['answer'], gold['type']):
        if pred['answer'] == g_ans and pred['type'] == g_type:
            correct += 1
    return correct / len(predictions)

Common pitfalls

• Model performance is highly sensitive to the prompt template; removing structured priors causes significant drops in accuracy for general-purpose MLLMs.
• Structure-level evaluation requires pixel-level alignment, which is inherently difficult and often leads to low scores even when semantic content is preserved.

Evidence (verbatim from paper)

The reported values are the accuracy of samples where both the answer and the problem type in Binary & Type QA are predicted correctly.

Citation

@misc{li2026stablei2i,
  title={StableI2I: Spotting Unintended Changes in Image-to-Image Transition},
  author={Li et al. (2026)},
  year={2026},
  note={arXiv:2605.04453}
}

• arXiv: 2605.04453