qhjqhj00 manitwin-asset-quality-eval
ManiTwin: Scaling Data-Generation-Ready Digital Object Dataset to 100K — Wang et al. (2026) (arXiv:2603.16866, 2026)
What this evaluates
Evaluates the semantic alignment, geometric fidelity, and visual appearance of automatically generated 3D assets against input images or text. It also assesses the accuracy of VLM-generated annotations across five dimensions and the physical validity of simulated grasp poses for robotic manipulation readiness.
Datasets
- ManiTwin-100K — total 100000; splits: test (100000)
Metrics
CLIP(N-I/T)— range: [0, 1]- Computes the CLIP cosine similarity between rendered normal maps of the 3D asset and the input image or text description. Scores are averaged across 30 rendered views.
CLIP(I-I/T)(primary) — range: [0, 1]- Computes the CLIP cosine similarity between rendered RGB images of the 3D asset and the input image or text description. Scores are averaged across 30 rendered views.
ULIP(I/T)— range: [0, 1]- Measures 3D-to-image or 3D-to-text alignment using the ULIP latent space representation. Scores are averaged across 30 rendered views.
Grasp verification success rate— range: percent- Percentage of generated grasp candidates that pass physics-based simulation verification for stability and collision constraints.
Annotation accuracy— range: percent- Human-judged correctness rate across five annotation types (category, language, functional points, physical properties, grasp points) on a random sample of 500 objects.
Input / output format
Input: Input image or text description for 3D generation; rendered normal/RGB views (30 views) for latent metric computation; generated 3D assets with VLM annotations for human evaluation; grasp candidates for physics simulation.
Output: Latent similarity scores averaged over 30 views; pass/fail flags from simulation; human-annotated ground truth labels for accuracy calculation.
Scoring recipe
def compute_clip_metric(rendered_views, input_condition):
scores = [clip_similarity(view, input_condition) for view in rendered_views]
return mean(scores)
def compute_grasp_rate(candidates, sim_engine):
verified = [g for g in candidates if sim_engine.verify(g).is_valid]
return len(verified) / len(candidates) * 100
def compute_annotation_accuracy(sample_500, human_judge):
correct = sum(1 for obj in sample_500 if human_judge(obj) == obj.ground_truth)
return correct / 500 * 100
Common pitfalls
- Latent metrics must be averaged over exactly 30 rendered views, not a single view or different count.
- Human evaluation accuracy is reported on a random sample of 500 objects, not the full 100K dataset.
- Grasp verification success rate is computed per-object before averaging across the dataset, not globally across all candidates.
Evidence (verbatim from paper)
We evaluate the quality of generated 3D assets using latent-based metrics that measure semantic alignment between generated 3D assets and their input conditions (images or text). ... CLIP(N-I) and CLIP(N-T) compute the CLIP similarity between rendered normal maps and the input image or text description, respectively, measuring how well the generated 3D geometry aligns with the input conditions. For each metric, we compute the average score across all 30 rendered views.
Citation
@misc{wang2026manitwin,
title={ManiTwin: Scaling Data-Generation-Ready Digital Object Dataset to 100K},
author={Wang et al. (2026)},
year={2026},
note={arXiv:2603.16866}
}
- arXiv: 2603.16866