Material Palette Eval

material-palette-eval

Material Palette: Extraction of Materials from a Single Image — Lopes et al. (2023) (arXiv:2311.17060, 2023)

What this evaluates

Evaluates the ability of a diffusion-based pipeline to extract physically-based rendering (PBR) materials, specifically spatially varying BRDFs (SVBRDFs), from single real-world images. It measures decomposition accuracy against ground-truth material maps and assesses the perceptual and semantic coherence of the extracted materials compared to known PBR datasets.

Datasets

• AmbientCG — total 2000; splits: train (2000)
• PolyHaven — total 320; splits: test (320)
• CGBookcase — total 320; splits: test (320)
• OpenSurfaces — total ?; splits: test (-1)
• TexSD — total 9000; splits: train (9000)

Metrics

• MSE (primary) — range: other
  • Mean Squared Error between predicted and ground truth SVBRDF maps (Albedo, Normal, Roughness). Lower values indicate better decomposition accuracy.
• SSIM — range: [0, 1]
  • Structural Similarity Index measuring perceived structural change between predicted and ground truth maps. Higher values indicate better preservation of structural information.
• LPIPS — range: [0, 1]
  • Learned Perceptual Image Patch Similarity computed on deep network features between extracted and reference materials. Lower values indicate higher perceptual resemblance.
• CLIP Top-1 Accuracy — range: percent
  • Zero-shot classification accuracy of rendered materials using CLIP ViT-B/32 against known PBR dataset classes. Higher values indicate better semantic coherence.

Input / output format

Input: Single real-world image (or cropped region/mask) and optionally a text prompt or material class name.

Output: Tileable texture images (up to 1024px or higher) decomposed into Albedo, Normal, and Roughness (SVBRDF) maps.

Scoring recipe

def evaluate_materials(pred_svbrdf, gt_svbrdf, reference_dataset):
    mse = np.mean((pred_svbrdf - gt_svbrdf) ** 2)
    ssim = compute_ssim(pred_svbrdf, gt_svbrdf)
    lpips_score = compute_lpips(pred_svbrdf, reference_dataset)
    rendered_material = render_svbrdf(pred_svbrdf)
    clip_acc = clip_zero_shot_classify(rendered_material, reference_dataset.classes)
    return {'MSE': mse, 'SSIM': ssim, 'LPIPS': lpips_score, 'CLIP_Acc': clip_acc}

Common pitfalls

• SVBRDF ground truth is only available for synthetic datasets (ACG, PH, CGB); real-world evaluation relies on perceptual metrics (LPIPS) and CLIP classification rather than pixel-wise accuracy.
• The pipeline requires explicit region masks (e.g., from OpenSurfaces, SAM, or Materialistic) as input; evaluating on full unmasked images will fail due to entangled lighting and geometry.
• Outputs must be post-processed for tileability (via latent rolling and Poisson solving) to be valid for 3D rendering; skipping this step invalidates the evaluation.

Evidence (verbatim from paper)

In practice, the evaluation is conducted by sampling 100 {M_SD, M_tilde} pairs and evaluating LPIPS [64] between them (lower is better), for each of the 14 material classes. Considering class c ∈ C_m, Ours will evaluate {M_SD^c, M_tilde^c} pairs ∀ c ∈ C_m, while the upper bound will have {M_SD^c, M_tilde^c} where c_bar is a random c_bar ≠ c. The reported LPIPS values are averaged over all classes.

Citation

@misc{lopes2023materialpalette,
  title={Material Palette: Extraction of Materials from a Single Image},
  author={Lopes et al. (2023)},
  year={2023},
  note={arXiv:2311.17060}
}

• arXiv: 2311.17060