Dualtoken Eval

dualtoken-eval

DualToken: Towards Unifying Visual Understanding and Generation with Dual Visual Vocabularies — Wei Song et al. (arXiv:2503.14324, 2025)

What this evaluates

Evaluates a unified vision tokenizer's capacity to decouple and jointly optimize low-level perceptual reconstruction and high-level semantic understanding. It probes zero-shot classification, cross-modal retrieval, image reconstruction fidelity, and downstream multimodal reasoning capabilities.

Datasets

• ImageNet-1K — total ?; splits: val (-1)
• Flickr8K — total ?; splits: test (-1)
• VQAv2 — total ?; splits: test (-1)
• POPE — total ?; splits: test (-1)
• MME — total ?; splits: test (-1)
• SEED-IMG — total ?; splits: test (-1)
• MMBench — total ?; splits: dev (-1)
• MM-Vet — total ?; splits: test (-1)

Metrics

• Top-1 accuracy (primary) — range: percent
  • Percentage of correctly classified images in zero-shot classification on ImageNet-1K validation set. Computed as (correct predictions / total samples) * 100.
• R@1 — range: percent
  • Recall at rank 1 for text-to-image and image-to-text retrieval on Flickr8K. Measures the fraction of queries where the correct match appears as the top-ranked result.
• rFID — range: other
  • Reconstruction Fréchet Inception Distance. Computes the FID between the distribution of real images and the distribution of reconstructed images from predicted tokens. Lower is better.
• PSNR — range: other
  • Peak Signal-to-Noise Ratio in decibels, measuring the ratio between the maximum possible power of a signal and the distorting noise introduced by reconstruction. Higher is better.
• SSIM — range: [-1, 1]
  • Structural Similarity Index, measuring perceived change in structural information between original and reconstructed images. Ranges from -1 to 1, where 1 indicates identical images.

Input / output format

Input: Images resized to 384×384 pixels, paired with text prompts, classification labels, or retrieval queries.

Output: Discrete semantic and pixel tokens (27×27×8 grid), generated images, or classification/retrieval labels.

Scoring recipe

acc = sum(1 for p, g in zip(preds, gold) if p == g) / len(gold)
r1 = sum(1 for q in queries if gold_ids[q] in top_1[q]) / len(queries)
rfid = frechet_inception_distance(real_imgs, recon_imgs)
psnr = peak_signal_noise_ratio(real_imgs, recon_imgs)
ssim = structural_similarity(real_imgs, recon_imgs, channel_axis=-1)

Common pitfalls

• Naively combining separate specialized encoders (e.g., MoVQGAN for pixels + SigLIP for semantics) without unified training severely degrades generation quality.
• Using only semantic tokens for downstream tasks underperforms on reasoning benchmarks (MMBench, MME) compared to concatenating semantic and perceptual tokens.
• Classifier-free guidance (CFG) must be fixed at 3.0 for generation evaluation to ensure fair comparison across methods.

Evidence (verbatim from paper)

We evaluate our model against widely used vision-language understanding benchmarks, including VQAv2, POPE, MME, SEED-IMG, MMBench, and MM-Vet. For evaluating the semantic capabilities of our unified vision tokenizer, we report the Top-1 accuracy for zero-shot image classification on ImageNet-1K (validation set), along with text-to-image and image-to-text retrieval performance (R@1) on Flickr8K. To evaluate reconstruction capability, we measured reconstruction FID (rFID), PSNR, and SSIM on the ImageNet-1K validation set.

Citation

@misc{song2025dualtoken,
  title={DualToken: Towards Unifying Visual Understanding and Generation with Dual Visual Vocabularies},
  author={Wei Song et al.},
  year={2025},
  note={arXiv:2503.14324}
}

• arXiv: 2503.14324