Mvpaint T2t Eval

mvpaint-t2t-eval

MVPaint: Synchronized Multi-View Diffusion for Painting Anything 3D — Wei Cheng et al. (2024) (arXiv:2411.02336, 2024)

What this evaluates

This evaluation probes a model's ability to generate high-quality, multi-view consistent 3D textures on arbitrary meshes conditioned on text instructions. It measures visual fidelity, distributional similarity to ground truth, and cross-view consistency through both automated generative metrics and human preference studies.

Datasets

• Objaverse T2T benchmark — total 1000; splits: train (102000), test (1000)
• GSO T2T benchmark — total 1032; splits: test (1032)

Metrics

• FID (primary) — range: other
  • Fréchet Inception Distance between the feature distributions of rendered images from generated textures versus ground truth textures. Lower is better.
• KID — range: other
  • Kernel Inception Distance, scaled by 10^3 in reported tables. Lower is better.
• CLIP score — range: other
  • Cosine similarity between image and text embeddings in CLIP space. Higher is better.
• User Study (Overall/Seamless/Consistency) — range: other
  • Average rating on a 1-5 Likert scale provided by 10 human participants evaluating rendered 3D meshes.

Input / output format

Input: Text instruction (prompt) and a 3D mesh geometry (with or without UVs).

Output: A UV texture map or synchronized multi-view RGB images that are baked onto the input mesh.

Scoring recipe

def score(predictions, gold, text_prompt):
    gen_imgs = render_mesh(predictions.mesh, predictions.texture, viewpoints=16, elevation=15)
    gt_imgs = render_mesh(gold.mesh, gold.texture, viewpoints=16, elevation=15)
    fid = compute_fid(gen_imgs, gt_imgs)
    kid = compute_kid(gen_imgs, gt_imgs) * 1000
    clip = compute_clip_similarity(gen_imgs, text_prompt)
    return {'FID': fid, 'KID': kid, 'CLIP': clip}

Common pitfalls

• Janus multi-view inconsistency artifacts can artificially inflate CLIP scores, so high CLIP alone does not guarantee visual quality or multi-view consistency.
• Reported KID values in tables are scaled by 10^3; raw KID values are orders of magnitude smaller and should not be compared directly to the table numbers.
• User study ratings use a 1-5 scale, not a percentage or 0-100 scale, which can cause misinterpretation of subjective performance.

Evidence (verbatim from paper)

After generating 3D textures, 512-resolution images of the mesh are rendered with the generated textures from 16 fixed viewpoints at the same elevation, i.e., 15° for fairness. Then, we compare the rendered images with the true image distribution generated using the ground truth textures. For a thorough evaluation, we use common generative metrics: Fréchet Inception Distance (FID), Kernel Inception Distance (KID), and CLIP score to assess image distribution, quality, and richness.

Citation

@misc{cheng2024mvpaint,
  title={MVPaint: Synchronized Multi-View Diffusion for Painting Anything 3D},
  author={Wei Cheng et al. (2024)},
  year={2024},
  note={arXiv:2411.02336}
}

• arXiv: 2411.02336