Opening Eval

opening-eval

OpenING: A Comprehensive Benchmark for Judging Open-ended Interleaved Image-Text Generation — Zhou et al. (2024) (arXiv:2411.18499, 2024)

What this evaluates

Evaluates the quality of open-ended interleaved image-text generation by comparing model outputs against human-annotated references. It probes multimodal coherence, visual fidelity, and text-image alignment through pairwise battles and automated judge agreement.

Datasets

• OpenING — total 5400; splits: test (5400)

Metrics

• agreement (primary) — range: percent
  • Measures the consistency between an automated judge and human evaluators. Calculated as the proportion of pairwise judgments where the judge's win/loss/tie decision matches the human's decision under a specified tie-handling strategy.
• win rate — range: percent
  • Indicates how often a model wins in pairwise comparisons. Computed using four tie-handling strategies: Force Dividing Tie (FDT), Without Tie (w/o Tie), With Tie counted as 0 (w/ Tie (0)), and With Tie counted as 0.5 (w/ Tie (.5)).

Input / output format

Input: A task prompt requiring interleaved image-text generation, paired with a reference human-annotated output. Models generate outputs that are then presented in pairwise battles against the reference or other models.

Output: A pairwise judgment (Model A wins, Model B wins, or Tie) or a numerical score per criterion for the generated interleaved content.

Scoring recipe

def compute_win_rate(judgments, tie_strategy='FDT'):
    wins = 0
    total = 0
    for j in judgments:
        if tie_strategy == 'FDT':
            wins += 1 if j == 'A' else 0
            total += 1
        elif tie_strategy == 'w/o Tie':
            if j != 'Tie':
                wins += 1 if j == 'A' else 0
                total += 1
        elif tie_strategy == 'w/ Tie (0)':
            wins += 1 if j == 'A' else 0
            total += 1
        elif tie_strategy == 'w/ Tie (0.5)':
            wins += 0.5 if j == 'Tie' else (1 if j == 'A' else 0)
            total += 1
    return wins / total if total > 0 else 0

def compute_agreement(judge_judgments, human_judgments):
    matches = sum(1 for j, h in zip(judge_judgments, human_judgments) if j == h)
    return matches / len(human_judgments)

Common pitfalls

• Win rates are highly sensitive to the tie-handling strategy; results using FDT, w/o Tie, and w/ Tie (0.5) are not directly comparable.
• Agreement scores depend heavily on the sampling size of battle pairs (4,320 pairs used); smaller samples yield unstable rankings.
• The 'seen' vs 'unseen' model split significantly impacts agreement, with unseen models showing lower agreement due to lack of training exposure.

Evidence (verbatim from paper)

Model performance are evaluated using two key metrics: win rate and agreement. Win rate indicates how often a model wins in pairwise comparisons. Four methods used to handle ties include 1) Force Dividing Tie (FDT): We force judges to assign ties with a more leaning model in rules and prompts, ensuring that every comparison round results in a decisive outcome. A win is attributed to A if a tie favors model A (Tie(A)), likewise for B. This metric allows for clear rankings without ambiguity. 2) Without Tie (w/o Tie): Tied comparisons are excluded; only matches with a clear winner are considered; 3) With Tie counted as 0 (w/ Tie (0)): Ties are included but do not contribute to the win count of either model; 4) With Tie counted as 0.5 (w/ Tie (.5)): Each tie contributes half a win to both models. Agreement measures the consistency between different evaluators (e.g., automated pipelines and human judgments) under the same tie-handling strategies.

Citation

@misc{zhou2024opening,
  title={OpenING: A Comprehensive Benchmark for Judging Open-ended Interleaved Image-Text Generation},
  author={Zhou et al. (2024)},
  year={2024},
  note={arXiv:2411.18499}
}

• arXiv: 2411.18499