Gt23d Bench Eval

gt23d-bench-eval

GT23D-Bench: A Comprehensive General Text-to-3D Generation Benchmark — Sitong Su et al. (2024) (arXiv:2412.09997, 2024)

What this evaluates

Evaluates the quality and alignment of generated 3D assets against text prompts across multiple dimensions, including textual alignment, texture fidelity, geometry correctness, and multi-view consistency. It measures how well automated metrics correlate with human preferences to provide a reliable assessment of general text-to-3D generation methods.

Datasets

• GT23D-Bench — total 400000; splits: test (380)

Metrics

• Textual-PointCloud — range: [0, 1] or arbitrary score scale
  • Measures alignment between the text prompt and the generated point cloud representation using cross-modal matching.
• Textual-MultiView — range: [0, 1] or arbitrary score scale
  • Measures alignment between the text prompt and the generated multi-view images.
• Textual-Attribute — range: [0, 1] or arbitrary score scale
  • Measures alignment between the text prompt and specific semantic attributes of the generated 3D object.
• Texture Fidelity (primary) — range: [0, 1] or arbitrary score scale
  • Evaluates the richness, clarity, and visual quality of textures on the generated 3D surface. Comprises sub-metrics CC, TR, and Mv-IQ.
• Geometry Correctness — range: [0, 1] or arbitrary score scale
  • Assesses the structural accuracy, shape completeness, and geometric validity of the generated 3D asset. Comprises sub-metrics 3D-Ali, Shape-C, and Geo-V.
• Multi-View Consistency — range: [0, 1] or arbitrary score scale
  • Measures how consistent the generated 3D asset appears across different camera viewpoints, penalizing view-inconsistent artifacts.

Input / output format

Input: Text prompt (caption) describing a 3D object or scene.

Output: 3D representation (e.g., point cloud, multi-view images, or implicit representation).

Scoring recipe

def evaluate_generation(prompts, generated_assets, human_scores=None):
    scores = {}
    for prompt, asset in zip(prompts, generated_assets):
        scores[prompt] = {
            'Textual-PointCloud': compute_alignment(asset, prompt, modality='pointcloud'),
            'Textual-MultiView': compute_alignment(asset, prompt, modality='multiview'),
            'Textual-Attribute': compute_alignment(asset, prompt, modality='attribute'),
            'Texture Fidelity': compute_texture_quality(asset),
            'Geometry Correctness': compute_geometry_quality(asset),
            'Multi-View Consistency': compute_consistency(asset)
        }
    if human_scores is not None:
        return {dim: pearson_spearman_kendall(metric_scores, human_scores) for dim, metric_scores in scores.items()}
    return scores

Common pitfalls

• Methods that only generate multi-view images (e.g., MVDream) cannot be evaluated on point-cloud-based metrics like Textual-PointCloud or 3D-Ali, leading to missing scores.
• Aesthetic scores may favor visually appealing but geometrically inconsistent outputs, leading to misleading quality assessments.
• Correlation metrics (Pearson/Spearman/Kendall) measure human alignment reliability, not absolute generation quality.

Evidence (verbatim from paper)

Multi-view image-based methods like MVDream achieve high visual quality in single-view images (high Texture Fidelity score) but lack 3D supervision, resulting in poor 3D quality (low Multi-View Consistency score).

Citation

@misc{su2024gt23dbench,
  title={GT23D-Bench: A Comprehensive General Text-to-3D Generation Benchmark},
  author={Sitong Su et al. (2024)},
  year={2024},
  note={arXiv:2412.09997}
}

• arXiv: 2412.09997