Sarena Icon Eval

sarena-icon-eval

Reliable Reasoning in SVG-LLMs via Multi-Task Multi-Reward Reinforcement Learning — Haomin Wang et al. (2026) (arXiv:2603.16189, 2026)

What this evaluates

Evaluates a model's ability to generate scalable vector graphics (SVG) from text prompts and reference images, measuring visual fidelity, semantic alignment, structural success, and code efficiency.

Datasets

• SArena-Icon — total ?; splits: test (-1)

Metrics

• SR (primary) — range: percent
  • Success Rate: percentage of generated SVGs that successfully render without syntax or structural errors.
• CLIP-T2I — range: other
  • CLIP Text-to-Image similarity score between the text prompt and the rendered SVG image.
• CLIP-I2I — range: other
  • CLIP Image-to-Image similarity score between the reference image and the rendered SVG image.
• FID — range: other
  • Fréchet Inception Distance measuring the distribution difference between real and generated SVG images.
• DINO — range: [0, 1]
  • DINO similarity score between the reference/generated image and the ground truth.
• SSIM — range: [0, 1]
  • Structural Similarity Index measuring perceptual similarity between images.
• LPIPS — range: [0, 1]
  • Learned Perceptual Image Patch Similarity measuring perceptual distance between images.
• Tokens — range: other
  • Number of tokens in the generated SVG code, measuring code efficiency.

Input / output format

Input: Text prompt (for Text-to-SVG) or reference image (for Image-to-SVG).

Output: SVG code string.

Scoring recipe

def compute_metrics(predictions, gold):
    rendered_imgs = [render_svg(svg) for svg in predictions]
    sr = sum(1 for img in rendered_imgs if img.is_valid()) / len(predictions)
    clip_t2i = clip_similarity(text_prompts, rendered_imgs)
    clip_i2i = clip_similarity(ref_images, rendered_imgs)
    fid = frechet_distance(real_imgs, rendered_imgs)
    dino = dino_similarity(ref_imgs, rendered_imgs)
    ssim = structural_similarity(ref_imgs, rendered_imgs)
    lpips = perceptual_distance(ref_imgs, rendered_imgs)
    tokens = [count_tokens(svg) for svg in predictions]
    return {'SR': sr, 'CLIP-T2I': clip_t2i, 'CLIP-I2I': clip_i2i, 'FID': fid, 'DINO': dino, 'SSIM': ssim, 'LPIPS': lpips, 'Tokens': tokens}

Common pitfalls

• Metrics are computed on the rasterized/rendered output of the SVG, not the raw code.
• Success Rate (SR) requires a parser or renderer to verify SVG validity before computing visual metrics.

Evidence (verbatim from paper)

CTRL-S achieves leading performance on both the Text-to-SVG and Image-to-SVG tasks on the SArena-Icon benchmark. For Text-to-SVG, CTRL-S attains the highest CLIP-T2I score of 25.944... For Image-to-SVG, our model obtains the best results across the DINO, SSIM, and LPIPS metrics compared to mainstream general VLMs and SVG-LLMs.

Citation

@misc{wang2026ctrls,
  title={Reliable Reasoning in SVG-LLMs via Multi-Task Multi-Reward Reinforcement Learning},
  author={Haomin Wang et al. (2026)},
  year={2026},
  note={arXiv:2603.16189}
}

• arXiv: 2603.16189