Vincicoder Code Gen Eval

vincicoder-code-gen-eval

VinciCoder: Unifying Multimodal Code Generation via Coarse-to-fine Visual Reinforcement Learning — Xuanle Zhao et al. (arXiv:2511.00391, 2025)

What this evaluates

Probes a model's ability to generate executable, visually faithful code (HTML, SVG, LaTeX, SMILES) from input images across diverse domains. It evaluates both syntactic correctness via execution rate and perceptual alignment with target images using coarse-to-fine visual similarity metrics.

Datasets

• ChartMimic — total ?; splits: test (-1)
• Design2Code — total ?; splits: test (-1)
• UniSVG — total ?; splits: test (-1)
• Image2Struct — total ?; splits: test (-1)
• Cosyn-400k — total 128; splits: test (128)

Metrics

• UniSVG Final Score (primary) — range: [0, 1]
  • Weighted combination of low-level and high-level visual scores. Low-level = average of SSIM and (1 - LPIPS). High-level = perceptual alignment score (e.g., DINOv2). Final = 0.4 * Low-level + 0.6 * High-level.
• Execution Rate — range: [0, 1]
  • Fraction of generated code instances that run without syntax or runtime errors. Non-executable code is penalized with a zero reward.
• Tanimoto Similarity — range: [0, 1]
  • Measures chemical similarity between generated and ground-truth SMILES strings using the Tanimoto coefficient on molecular fingerprints.
• Earth Mover Similarity (EMS) — range: [0, 1]
  • Computes structural similarity between generated and target LaTeX code using the Earth Mover's Distance metric from the official Image2Struct repository.

Input / output format

Input: An input image (chart, webpage layout, SVG, chemical structure, or document) optionally accompanied by a text prompt or instruction.

Output: A generated code string (HTML, SVG, LaTeX, or SMILES) intended to render or execute to match the input image and/or ground-truth structure.

Scoring recipe

def compute_metrics(pred_code, gold_code, target_img, rendered_img):
    exec_ok = try_execute(pred_code)
    exec_rate = 1.0 if exec_ok else 0.0
    ssim = compute_ssim(rendered_img, target_img)
    lpips = compute_lpips(rendered_img, target_img)
    low_level = (ssim + (1.0 - lpips)) / 2.0
    high_level = compute_dino_similarity(rendered_img, target_img)
    univsvg_final = 0.4 * low_level + 0.6 * high_level
    tanimoto = compute_tanimoto(pred_code, gold_code)
    ems = compute_ems(pred_code, gold_code)
    return {'exec_rate': exec_rate, 'univsvg_final': univsvg_final, 'tanimoto': tanimoto, 'ems': ems}

Common pitfalls

• Treating UniSVG's low-level and high-level scores as the final metric instead of applying the 40/60 weighted combination.
• Ignoring the execution rate penalty: non-executable code receives a zero reward, which drastically lowers the overall score.
• Assuming visual similarity metrics (SSIM/LPIPS) apply to all domains; LaTeX and SMILES require structural/chemical metrics (EMS, Tanimoto) instead.

Evidence (verbatim from paper)

For the UniSVG, the final score is a weighted combination of two components. The low-level score, which is the average of SSIM and (1 - LPIPS), contributes 40% to the final score, while the high-level score contributes the remaining 60%. For the Image2Struct, we utilize Earth Mover Similarity (EMS) in their GitHub repository for evaluation. For the molecule-to-code task, we evaluate performance on the Cosyn-400k chemistry test set, which consists of 128 molecular images paired with their ground-truth SMILES strings. We report two metrics: the execution rate and the average Tanimoto similarity (Tani. Sim.) of the generated SMILES.

Citation

@misc{zhao2025vincicoder,
  title={VinciCoder: Unifying Multimodal Code Generation via Coarse-to-fine Visual Reinforcement Learning},
  author={Xuanle Zhao et al.},
  year={2025},
  note={arXiv:2511.00391}
}

• arXiv: 2511.00391