T3bench Eval

t3bench-eval

T$^3$Bench: Benchmarking Current Progress in Text-to-3D Generation — Yuze He et al. (2023) (arXiv:2310.02977, 2023)

What this evaluates

Evaluates the visual quality and text-3D alignment of generated 3D scenes across varying prompt complexities (single object, object with surroundings, multiple objects). It specifically probes multi-view consistency (detecting the Janus problem) and the ability of 2D diffusion guidance to translate into coherent 3D structures.

Datasets

• T$^3$ Bench — total 300; splits: test (300)

Metrics

• Multi-view Quality (ImageReward) (primary) — range: [0, 100]
  • Renders the 3D mesh from 5 focal lengths (3.0, 4.0, 5.0, 6.0, 7.5) at 512x512 resolution. Each view is scored using ImageReward, then aggregated via regional convolution to produce a global quality score. Normalized to [0, 100].
• Alignment (GPT-4) (primary) — range: [0, 100]
  • Generates a 3D caption for the scene, then uses GPT-4 to score the alignment between the caption and the original text prompt. Normalized to [0, 100].

Input / output format

Input: Text prompt describing a 3D scene (categorized as single object, single object with surroundings, or multiple objects).

Output: 3D mesh/scene (rendered as multi-view images for evaluation).

Scoring recipe

# Quality Metric
views = render_mesh(mesh, focal_lengths=[3.0, 4.0, 5.0, 6.0, 7.5], res=512)
view_scores = [image_reward_score(v) for v in views]
quality_score = regional_convolution_aggregate(view_scores)
quality_score = normalize(quality_score, min=-2.5, max=2.5, target_min=0, target_max=100)

# Alignment Metric
caption = generate_3d_caption(mesh)
alignment_score = gpt4_score(prompt=caption, reference=original_text_prompt)
alignment_score = normalize(alignment_score, min=1, max=5, target_min=0, target_max=100)

Common pitfalls

• Single-view metrics (e.g., CLIP R-Precision) fail to detect multi-view inconsistencies like the Janus problem.
• Retrieval-based metrics provide comparative rather than absolute scores, misaligning with human perceptual processes.
• SDS-based methods often suffer from density collapse or sparse floating density, degrading mesh extraction quality.

Evidence (verbatim from paper)

We observe that Multi-view capturing + ImageReward and 3D captioning + GPT-4 scoring align most closely with quality and alignment aspects as annotated by human experts, respectively. We thus employ these combinations as the default quality and alignment metrics in our benchmark, throughout the rest of the paper.

Citation

@misc{he2023t3bench,
  title={T$^3$Bench: Benchmarking Current Progress in Text-to-3D Generation},
  author={Yuze He et al. (2023)},
  year={2023},
  note={arXiv:2310.02977}
}

• arXiv: 2310.02977