T23d Compbench Eval

t23d-compbench-eval

Towards Fine-Grained Text-to-3D Quality Assessment: A Benchmark and A Two-Stage Rank-Learning Metric — Cui et al. (2025) (arXiv:2509.23841, 2025)

What this evaluates

Evaluates the fine-grained quality of text-to-3D generated meshes across multiple dimensions including textual alignment, visual quality, and authenticity. It measures how well generative models adhere to complex compositional prompts and produce structurally sound, aesthetically pleasing 3D assets.

Datasets

• T23D-CompBench — total 3600; splits: test (3600)

Metrics

• Mean Opinion Score (MOS) (primary) — range: [1, 5]
  • Average of the 5-point subjective ratings provided by three human annotators for each mesh across a specific quality dimension.

Input / output format

Input: A text prompt and a generated textured 3D mesh (e.g., OBJ/GLTF format).

Output: A 5-point subjective rating score (1–5) per quality dimension.

Scoring recipe

def compute_mos(annotator_scores):
    # annotator_scores: list of 3 floats (ratings from 1 to 5)
    if len(annotator_scores) != 3:
        raise ValueError('Expected 3 annotator scores')
    mos = sum(annotator_scores) / 3.0
    return mos

Common pitfalls

• Failing to re-annotate samples where the range of the three annotator scores is ≥ 2, as the protocol mandates re-annotation until agreement is reached.
• Reporting only the Overall Quality (OQ) score without breaking down results into the 12 sub-dimensions, which obscures the fine-grained analysis the benchmark is designed for.

Evidence (verbatim from paper)

To obtain the MOS of each mesh, we invite participants to evaluate the samples across the defined quality dimensions. We adopt the 5-point rating scale recommended by ITU-T P.910 as the voting methodology. An interactive evaluation protocol is employed, allowing participants to freely adjust their viewing angles according to their preferences. Following [wu2024gpt4v, huang2023t2i, sun2025t2v, han2024evalmuse], each textured mesh is rated by three annotators across twelve evaluation dimensions. In cases where the scores from the three annotators exhibit significant disagreement (i.e., range ≥ 2), we conduct re-annotation until the scores fall within an acceptable agreement threshold. Ultimately, each sample is annotated by 3 annotators, and the average score is used as the MOS.

Citation

@misc{cui2025t23dcompbench,
  title={Towards Fine-Grained Text-to-3D Quality Assessment: A Benchmark and A Two-Stage Rank-Learning Metric},
  author={Cui et al. (2025)},
  year={2025},
  note={arXiv:2509.23841}
}

• arXiv: 2509.23841