Ikea Bench Eval

ikea-bench-eval

Benchmarking and Mechanistic Analysis of Vision-Language Models for Cross-Depiction Assembly Instruction Alignment — Liu et al. (2026) (arXiv:2604.00913, 2026)

What this evaluates

Evaluates vision-language models on cross-depiction assembly instruction alignment, testing their ability to match, verify, locate, and predict steps from diagrams and videos. It also probes mechanistic properties like representational alignment and modality reliance to diagnose the 'depiction gap'.

Datasets

• IKEA-Bench — total ?; splits: test (-1)

Metrics

• Accuracy (primary) — range: [0, 1]
  • Fraction of correctly parsed and matched answers across multiple-choice tasks. Chance is 25% for 4-way MC and 50% for binary tasks.
• Centered Kernel Alignment (CKA) — range: [0, 1]
  • Measures geometric similarity between diagram and video feature sets: CKA(X,Y) = ||Y^T X||_F^2 / (||X^T X||_F * ||Y^T Y||_F). Ranges from 0 (unrelated) to 1 (identical structure).
• Recall@1 / Recall@10 — range: [0, 1]
  • Proportion of diagrams for which the nearest video frame in a gallery belongs to the matching step, evaluated at top-1 and top-10 retrievals.
• Modality Influence Score — range: [-1, 1]
  • Cosine similarity between the final-layer prediction hidden state and the average hidden state of a specific modality: s_m = cos(h_last, h_bar^m).
• Attention Allocation Score — range: [0, 1]
  • Sum of attention weights from the last input token to tokens belonging to each modality, averaged across heads and layers.

Input / output format

Input: Interleaved image-text prompts with task-specific system context. For mechanistic probing: frozen ViT feature vectors, final-layer hidden states, and attention weights.

Output: Multiple-choice answer (A/B/C/D) extracted via multi-priority regex matching; unparseable outputs are counted as incorrect.

Scoring recipe

# Accuracy
correct = sum(1 for p, g in zip(predictions, golds) if p == g)
accuracy = correct / len(golds)

# CKA
X = center(diagram_vectors)
Y = center(video_vectors)
cka = (norm(Y.T @ X)**2) / (norm(X.T @ X) * norm(Y.T @ Y))

# Recall@K
matches = [1 for q, gal in zip(queries, galleries) if q in gal[:K]]
recall = sum(matches) / len(queries)

Common pitfalls

• Regex-based answer extraction has a ~5.9% failure rate; these failures are explicitly counted as incorrect, which can artificially deflate reported accuracy.
• Zero-shot greedy decoding is used exclusively, which may not reflect optimal model performance under other decoding strategies or prompt variations.
• Mechanistic analysis relies on frozen ViT features, which may not capture how the full VLM adapts representations during inference.

Evidence (verbatim from paper)

Accuracy is the primary metric (4-way MC: chance = 25%; binary: chance = 50%). Answer extraction uses multi-priority regex matching; the overall parse rate is 94.1% across all runs, with failures counted as incorrect.

Citation

@misc{liu2026ikeabench,
  title={Benchmarking and Mechanistic Analysis of Vision-Language Models for Cross-Depiction Assembly Instruction Alignment},
  author={Liu et al. (2026)},
  year={2026},
  note={arXiv:2604.00913}
}

• arXiv: 2604.00913