Weavebench Eval

weavebench-eval

WEAVE: Unleashing and Benchmarking the In-context Interleaved Comprehension and Generation — Chow et al. (2025) (arXiv:2511.11434, 2025)

What this evaluates

Evaluates multi-turn, context-aware image comprehension and generation in an interleaved setting. It probes a model's ability to maintain visual consistency, follow iterative editing instructions, and integrate historical context across multiple turns.

Datasets

• WEAVEBench — total ?; splits: test (-1)

Metrics

• WEAVEBench (primary) — range: [0, 1]
  • Average score across four categories: ΔScience, Creation, Logic, and Game. Scores are computed via a hybrid VLM judge (GPT-4.1) and normalized to [0, 1].

Input / output format

Input: Multi-turn dialogue containing interleaved text instructions and images. Models are evaluated under three in-context conditions: (1) no in-context (single-turn), (2) partial in-context (only self-generated images with explicit visual context), and (3) complete in-context (all previous interactions visible). Images can be placed at first mention ('yes-first') or consolidated at the beginning ('yes-front').

Output: Generated image(s) corresponding to the final instruction in the dialogue, or text response for comprehension tasks.

Scoring recipe

def compute_weavebench_score(predictions, gold, judge='GPT-4.1'):
    scores = []
    for pred, gold_inst in zip(predictions, gold):
        score = judge.evaluate(pred, gold_inst)
        scores.append(score)
    return sum(scores) / len(scores)

Common pitfalls

• Confusing the three in-context conditions (no, partial, complete) which significantly impact performance, especially for open-source vs proprietary models.
• Image placement strategy ('yes-first' vs 'yes-front') affects results; concatenation can cause up to 10.3% degradation compared to sequential input.
• Relying solely on automated VLM judges without verifying human correlation, though the paper shows >0.8 Pearson correlation.

Evidence (verbatim from paper)

We evaluated 4 LLMs, 7 Edit models, and 11 UMMs on WEAVEBench as presented in Table 2. Evaluations were conducted under three distinct in-context conditions: (1) no in-context (single-turn generation without contextual information), (2) partial in-context (using only self-generated images with explicitly mentioned visual context, excluding other historical interactions), and (3) complete in-context (with all previous interactions visible).

Citation

@misc{chow2025weave,
  title={WEAVE: Unleashing and Benchmarking the In-context Interleaved Comprehension and Generation},
  author={Chow et al. (2025)},
  year={2025},
  note={arXiv:2511.11434}
}

• arXiv: 2511.11434