Svdquant Eval

svdquant-eval

SVDQuant: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Models — Li et al. (2024) (arXiv:2411.05007, 2024)

What this evaluates

Evaluates the visual fidelity and text-image alignment of quantized diffusion models by generating images from text prompts and comparing them against reference outputs. It probes whether low-bit quantization preserves distributional similarity, perceptual quality, and human-preferred aesthetics compared to full-precision baselines.

Datasets

• MJHQ-30K — total ?; splits: test (5000)
• sDCI — total ?; splits: test (5000)

Metrics

• FID (primary) — range: other
  • Fréchet Inception Distance measures the distributional distance between generated and reference images using features from an Inception network. Lower values indicate better image quality.
• Image Reward — range: other
  • A learned reward model that predicts human preference scores for generated images. Higher values indicate better alignment with human ratings.
• LPIPS — range: [0, 1]
  • Learned Perceptual Image Patch Similarity measures perceptual similarity using deep network features between generated and reference images. Lower values indicate higher fidelity.
• PSNR — range: other
  • Peak Signal-to-Noise Ratio measures numerical similarity in decibels between generated and reference images. Higher values indicate better reconstruction quality.

Input / output format

Input: Text prompts sampled from benchmark datasets

Output: Generated images

Scoring recipe

# predictions: list of generated images
# gold: list of reference images (16-bit outputs or ground truth)
fid = compute_fid(predictions, gold)
lpips = compute_lpips(predictions, gold)
psnr = compute_psnr(predictions, gold)
ir = compute_image_reward(predictions) # averaged over dataset
return {"FID": fid, "LPIPS": lpips, "PSNR": psnr, "IR": ir}

Common pitfalls

• Image Reward is aggregated over the entire dataset rather than computed per-prompt, requiring careful averaging.
• Comparisons must strictly match precision formats (e.g., W4A4 vs W4A16) and sampling steps (e.g., 4 vs 50 steps), as results vary significantly across configurations.
• LPIPS and PSNR require pixel-aligned image pairs; stochastic generation necessitates fixed seeds or multiple runs for stable scores.

Evidence (verbatim from paper)

Following previous works, we evaluate image quality and image similarity with respect to the 16-bit models’ results. For image quality assessment, we use Fréchet Inception Distance (FID, lower is better) to measure the distribution distance between the generated images and the ground-truth images. Besides, we employ Image Reward (higher is better) to approximate the human rating of the generated images. We use LPIPS (lower is better) to measure the perceptual similarity and Peak Signal Noise Ratio (PSNR, higher is better) to measure the numerical similarity of the images from the 16-bit models.

Citation

@misc{li2024svdquant,
  title={SVDQuant: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Models},
  author={Li et al. (2024)},
  year={2024},
  note={arXiv:2411.05007}
}

• arXiv: 2411.05007