Figedit Chart Editing Eval

figedit-chart-editing-eval

Charts Are Not Images: On the Challenges of Scientific Chart Editing — Li et al. (2025) (arXiv:2512.00752, 2025)

What this evaluates

This benchmark evaluates the ability of vision-language and image-editing models to perform semantically correct, structure-aware modifications to scientific charts. It probes whether models can follow precise editing instructions while preserving data-encoding consistency, axis coherence, and legend integrity, rather than merely producing pixel-level visual similarity.

Datasets

• FigEdit — total ?; splits: test (-1); repo https://github.com/adobe-research/figure-editing

Metrics

• SSIM — range: [0, 1]
  • Structural Similarity Index measuring luminance, contrast, and structure correlation between the original and edited chart images.
• PSNR — range: dB
  • Peak Signal-to-Noise Ratio in decibels, calculated from the mean squared error between the original and edited images.
• Instruction-following score (primary) — range: percent
  • LLM-based evaluation score measuring whether the model successfully applied the requested semantic edit to the chart.
• Content-preservation score — range: percent
  • LLM-based evaluation score measuring whether unedited chart elements (axes, labels, data points) remain intact after the edit.
• OCR accuracy — range: percent
  • Character-level or word-level accuracy of text recognition in the edited chart compared to ground truth text.

Input / output format

Input: A scientific chart image paired with a standardized text instruction specifying the desired edit (e.g., add/remove data points, change colors, modify labels).

Output: An edited chart image intended to reflect the requested semantic transformation.

Scoring recipe

def score_edit(original_img, edited_img, instruction, gt_text=None):
    ssim = compute_ssim(original_img, edited_img)
    psnr = compute_psnr(original_img, edited_img)
    ocr_acc = compute_ocr_accuracy(edited_img, gt_text)
    llm_prompt_if = f"Did the edit '{instruction}' correctly apply to the chart? Check data/axis integrity."
    instruction_score = llm_evaluate(llm_prompt_if, original_img, edited_img)
    llm_prompt_pres = "Are unedited elements preserved?"
    preservation_score = llm_evaluate(llm_prompt_pres, original_img, edited_img)
    return {'SSIM': ssim, 'PSNR': psnr, 'OCR': ocr_acc, 'Instruction': instruction_score, 'Preservation': preservation_score}

Common pitfalls

• Relying exclusively on SSIM or PSNR, which remain high even when semantic instructions are ignored or data is corrupted.
• Assuming visual smoothness or high pixel similarity indicates successful instruction following, which masks structural or data-encoding failures.
• Failing to verify axis coherence, legend integrity, and exact data-point placement, which are critical for scientific chart validity.

Evidence (verbatim from paper)

Classic metrics such as SSIM and PSNR exaggerate the performance of pixel-oriented models like Imagen 4, while LLM-based scores and OCR accuracy reveal significant semantic errors.

Citation

@misc{li2025chartsarenotimages,
  title={Charts Are Not Images: On the Challenges of Scientific Chart Editing},
  author={Li et al. (2025)},
  year={2025},
  note={arXiv:2512.00752}
}

• arXiv: 2512.00752