Svg Sophia Refinement Eval

svg-sophia-refinement-eval

Reliable Reasoning in SVG-LLMs via Multi-Task Multi-Reward Reinforcement Learning — Haomin Wang et al. (2026) (arXiv:2603.16189, 2026)

What this evaluates

Evaluates a model's ability to refine and correct imperfect SVG code, measuring structural accuracy, visual fidelity, and code efficiency.

Datasets

• SVG-Sophia Code Refinement Benchmark — total ?; splits: test (-1)

Metrics

• SR (primary) — range: percent
  • Success Rate: percentage of refined SVGs that successfully render and match the target structure.
• DINO — range: [0, 1]
  • DINO similarity score between the refined SVG image and the ground truth.
• SSIM — range: [0, 1]
  • Structural Similarity Index measuring perceptual similarity between images.
• LPIPS — range: [0, 1]
  • Learned Perceptual Image Patch Similarity measuring perceptual distance between images.
• Tokens — range: other
  • Number of tokens in the refined SVG code, measuring code efficiency.

Input / output format

Input: Imperfect SVG code draft.

Output: Refined SVG code string.

Scoring recipe

def compute_metrics(predictions, gold):
    rendered_imgs = [render_svg(svg) for svg in predictions]
    sr = sum(1 for img in rendered_imgs if img.is_valid()) / len(predictions)
    dino = dino_similarity(gold_imgs, rendered_imgs)
    ssim = structural_similarity(gold_imgs, rendered_imgs)
    lpips = perceptual_distance(gold_imgs, rendered_imgs)
    tokens = [count_tokens(svg) for svg in predictions]
    return {'SR': sr, 'DINO': dino, 'SSIM': ssim, 'LPIPS': lpips, 'Tokens': tokens}

Common pitfalls

• Metrics compare the rendered output of the refined SVG against the ground truth image.
• Token count measures the length of the generated SVG code, not the number of characters.

Evidence (verbatim from paper)

Table 2: SVG refinement results on SVG-Sophia Code Refinement Benchmark. Metrics: DINO, SSIM, LPIPS, SR, Tokens. CTRL-S achieves the best performance across all evaluation metrics compared with state-of-the-art proprietary models, including GPT-5.2, Claude-Sonnet-4.5, and Gemini-3-Pro.

Citation

@misc{wang2026ctrls,
  title={Reliable Reasoning in SVG-LLMs via Multi-Task Multi-Reward Reinforcement Learning},
  author={Haomin Wang et al. (2026)},
  year={2026},
  note={arXiv:2603.16189}
}

• arXiv: 2603.16189