Image2struct Eval

image2struct-eval

Image2Struct: Benchmarking Structure Extraction for Vision-Language Models — Somerville Roberts et al. (2024) (arXiv:2410.22456, 2024)

What this evaluates

Evaluates vision-language models' ability to extract structural code (HTML, LaTeX, LilyPond) from images. It uses a round-trip validation pipeline where generated code is rendered back to an image and compared to the original using automated similarity metrics.

Datasets

• Image2Struct — total ?; splits: test (-1); repo https://github.com/stanford-crfm/image2struct

Metrics

• EMS (primary) — range: [0, 1]
  • Automated image similarity metric comparing the original input image to the rendered output image, designed to correlate strongly with structural edit distances.
• compilation success rate — range: [0, 1]
  • Fraction of generated code snippets that successfully compile and render into an image without errors.

Input / output format

Input: An image of a structure (webpage, LaTeX document, or musical score) paired with a zero-shot prompt specifying the expected structural code format.

Output: The structural code (HTML, LaTeX, or LilyPond) intended to reproduce the input image.

Scoring recipe

compilation_success = 0
ems_scores = []
for pred_code, gold_img in zip(predictions, gold_images):
    rendered_img = render_and_screenshot(pred_code)
    if rendered_img is not None:
        compilation_success += 1
        ems_scores.append(earth_mover_similarity(gold_img, rendered_img))
    else:
        ems_scores.append(0.0)
return {
    'compilation_success_rate': compilation_success / len(predictions),
    'earth_mover_similarity': sum(ems_scores) / len(ems_scores)
}

Common pitfalls

• Models frequently refuse to generate code for musical scores due to false copyright infringement flags.
• Generated code often contains syntax errors or missing packages, requiring automated post-processing before rendering.
• Evaluation runs only once per model per instance (temperature=0) to minimize cost, potentially overlooking stochastic response variability.

Evidence (verbatim from paper)

We consider rendering success rate as a metric for model performance. ... We rank models with the mean win rate—which is the average fraction of other models that a model outperforms across scenarios—using the compilation success rates and EMS scores. Any compilation failure counts as zero EMS in the mean win rate calculation.

Citation

@misc{somerville2024image2struct,
  title={Image2Struct: Benchmarking Structure Extraction for Vision-Language Models},
  author={Somerville Roberts et al. (2024)},
  year={2024},
  note={arXiv:2410.22456}
}

• arXiv: 2410.22456