Mv Adapter Multi View Eval

mv-adapter-multi-view-eval

MV-Adapter: Multi-view Consistent Image Generation Made Easy — Huang et al. (2024) (arXiv:2412.03632, 2024)

What this evaluates

Evaluates a diffusion adapter's ability to generate geometrically consistent multi-view images conditioned on text prompts or reference images with camera parameters. It measures visual fidelity, image-text alignment, and multi-view structural similarity against ground-truth 3D scans.

Datasets

• Objaverse — total ?; splits: (unstated)
• Google Scanned Objects (GSO) — total ?; splits: (unstated)

Metrics

• FID (primary) — range: [0, ∞)
  • Fréchet Inception Distance. Computes the Fréchet distance between multivariate Gaussians fitted to feature representations of real and generated images.
• PSNR — range: [0, ∞) dB
  • Peak Signal-to-Noise Ratio. Measures pixel-level reconstruction fidelity in decibels based on the mean squared error between reference and generated images.
• SSIM — range: [0, 1]
  • Structural Similarity Index. Quantifies perceived change in structural information, luminance, and contrast between two images.
• LPIPS — range: [0, 1]
  • Learned Perceptual Image Patch Similarity. Computes perceptual distance using deep features from a pre-trained network (e.g., VGG or AlexNet).
• CLIP Score — range: [-1, 1]
  • Cosine similarity between the text prompt embedding and the image embedding in the CLIP latent space.
• IS — range: [0, ∞)
  • Inception Score. Measures image quality and diversity by training a classifier on generated images and computing the KL divergence between conditional and marginal label distributions.
• KID — range: [0, ∞)
  • Kernel Inception Distance. Estimates the distance between real and generated image distributions using polynomial kernels on Inception features.

Input / output format

Input: Text prompt + camera pose parameters (text-to-multiview); Reference image + camera pose/depth map (image-to-multiview)

Output: A set of multi-view images (typically 4 or 8 views) corresponding to specified camera poses

Scoring recipe

def evaluate(pred_imgs, gt_imgs, text_prompts=None):
    fid = frechet_inception_distance(pred_imgs, gt_imgs)
    psnr = peak_signal_noise_ratio(gt_imgs, pred_imgs)
    ssim = structural_similarity(gt_imgs, pred_imgs)
    lpips = learned_perceptual_image_patch_similarity(pred_imgs, gt_imgs)
    clip = cosine_similarity(text_embeds, image_embeds) if text_prompts else None
    return {'FID': fid, 'PSNR': psnr, 'SSIM': ssim, 'LPIPS': lpips, 'CLIP': clip}

Common pitfalls

• Standard 2D metrics (FID, PSNR, LPIPS) are computed per-view and do not directly quantify 3D geometric consistency or cross-view alignment.
• The exact subset of Objaverse used for training and the 1,000-prompt test split are not specified, hindering exact reproduction.
• Inference time is reported on a single RTX 4090 but varies with batch size and resolution, limiting fair cross-method timing comparisons.

Evidence (verbatim from paper)

Table 3: Quantitative comparison on 3D texture generation. FID and KID (×10−4) are evaluated on multi-view renderings. Our models achieves best texture quality with faster inference.

Citation

@misc{huang2024mvadapter,
  title={MV-Adapter: Multi-view Consistent Image Generation Made Easy},
  author={Huang et al. (2024)},
  year={2024},
  note={arXiv:2412.03632}
}

• arXiv: 2412.03632