Bootstrap3d Eval

bootstrap3d-eval

Bootstrap3D: Improving Multi-view Diffusion Model with Synthetic Data — Zeyi Sun et al. (2024) (arXiv:2406.00093, 2024)

What this evaluates

Evaluates text-to-multi-view diffusion models on their ability to generate prompt-aligned, high-quality 4-view images and reconstruct consistent 3D objects. It measures image-text alignment and visual fidelity against a synthetic ground-truth distribution.

Datasets

• GPTeval3D — total 110; splits: test (110)
• Synthetic GT Distribution — total 30000; splits: gt (30000)

Metrics

• FID (primary) — range: other (float)
  • Fréchet Inception Distance computes the Wasserstein-2 distance between two multivariate Gaussians fitted to feature distributions of generated and reference images. Lower is better.
• CLIP Score — range: other (float)
  • Mean cosine similarity between text prompt embeddings and generated image embeddings from a CLIP model. Higher is better.
• CLIP-R Score — range: other (float)
  • Mean cosine similarity between text prompt embeddings and generated image embeddings, typically using a different CLIP variant or resolution than standard CLIP Score. Higher is better.

Input / output format

Input: Text prompt. (Baselines also condition on a single-view image generated by PixArt).

Output: 4-view images arranged in a 2x2 grid, or 9 rendered views of reconstructed 3D objects.

Scoring recipe

def evaluate(predictions, prompts, gt_images):
    clip_scores = []
    for img, txt in zip(predictions, prompts):
        img_emb = clip.encode_image(img)
        txt_emb = clip.encode_text(txt)
        clip_scores.append(cosine_similarity(img_emb, txt_emb))
    clip_score = mean(clip_scores)
    fid = frechet_distance(clip_features(predictions), clip_features(gt_images))
    return {'clip_score': clip_score, 'fid': fid}

Common pitfalls

• FID is computed against a synthetic distribution of CAD-style images rather than real 3D object renders, which may not reflect true 3D quality.
• CLIP scores are highly dependent on the specific CLIP model variant (L/14 vs bigG) and prompt phrasing.
• View consistency is not measured directly but inferred from the FID of reconstructed 3D objects.

Evidence (verbatim from paper)

We primarily benchmark the quantitative results of our approach and other methods from two main dimensions: 1). Image-text alignment measured by CLIP score and CLIP-R score indicating the prompt follow ability of text-to-multi-view (T2MV) diffusion model. 2). Quality of generated images measured by FID(Heusel et al., 2017). Given the trend of decoupling multi-view image generation and sparse view reconstruction, we conduct tests separately on multi-view images by T2MV and rerendered images from generated 3D objects.

Citation

@misc{sun2024bootstrap3d,
  title={Bootstrap3D: Improving Multi-view Diffusion Model with Synthetic Data},
  author={Zeyi Sun et al. (2024)},
  year={2024},
  note={arXiv:2406.00093}
}

• arXiv: 2406.00093