Meshfleet Eval

meshfleet-eval

MeshFleet: Filtered and Annotated 3D Vehicle Dataset for Domain Specific Generative Modeling — Boborzi et al. (2025) (arXiv:2503.14002, 2025)

What this evaluates

This evaluation probes the effectiveness of automated, quality-filtered 3D vehicle datasets for text-to-3D generative modeling. It measures how fine-tuning a base model on curated meshes improves multi-view consistency and perceptual alignment compared to caption- or aesthetic-score-based filtering.

Datasets

• MeshFleet — total 1620; splits: train (1620); repo https://github.com/FeMa42/MeshFleet
• CarCaption3K — total 3326; splits: train (3326)
• CarCaption800 — total 834; splits: train (834)

Metrics

• CLIP-S (primary) — range: [0, 1]
  • CLIP similarity score measuring perceptual alignment between generated multi-view images and reference images.
• MSE — range: [0, ∞)
  • Mean Squared Error between generated and reference images, measuring pixel-level reconstruction error.

Input / output format

Input: Text prompts describing specific vehicle types (e.g., sports car, SUV).

Output: Multi-view images or 3D asset representations generated by the fine-tuned SV3D model.

Scoring recipe

def score(predictions, references):
    clip_scores = [clip_similarity(p, r) for p, r in zip(predictions, references)]
    mse_scores = [mean_squared_error(p, r) for p, r in zip(predictions, references)]
    return {'CLIP-S': mean(clip_scores), 'MSE': mean(mse_scores)}

Common pitfalls

• Aesthetic-score filtering often includes non-realistic objects like toys or fictional vehicles, degrading generation quality.
• The held-out test set is extremely small (12 instances), which may lead to high variance in metric scores.
• Caption-based filtering relies on prompt engineering without exhaustive VLM fine-tuning, potentially missing edge cases.

Evidence (verbatim from paper)

Fine-tuning SV3D with the MeshFleet dataset yielded the highest CLIP-S of 0.925, surpassing even the 0.923 CLIP-S achieved with the Label 4 subset (Table [1]). While the MSE for the MeshFleet fine-tuned model was slightly higher than that of the model trained on the Label 4 subset, the superior CLIP-S indicates improved overall perceptual quality.

Citation

@misc{boborzi2025meshfleet,
  title={MeshFleet: Filtered and Annotated 3D Vehicle Dataset for Domain Specific Generative Modeling},
  author={Boborzi et al. (2025)},
  year={2025},
  note={arXiv:2503.14002}
}

• arXiv: 2503.14002