qhjqhj00 sim3d-eval
Modulate-and-Map: Crossmodal Feature Mapping with Cross-View Modulation for 3D Anomaly Detection — Costanzino et al. (2026) (arXiv:2604.02328, 2026)
What this evaluates
Evaluates 3D anomaly detection and segmentation capabilities in industrial settings using multiview and multimodal (image + depth) inputs. It probes a model's ability to identify and localize defects across multiple object categories under both in-domain (real-to-real) and out-of-domain (synthetic-to-real) conditions.
Datasets
- SiM3D — total ?; splits: test (-1)
Metrics
I-AUROC(primary) — range: [0, 1]- Area under the Receiver Operating Characteristic curve for image-level anomaly detection. Computed per object category and averaged across all categories.
V-AUPRO@1%— range: [0, 1]- Area under the Precision-Recall curve at a fixed 1% false positive rate for voxel-level anomaly segmentation. Computed per category and averaged.
Input / output format
Input: Multiview image and depth/point cloud data for industrial objects. Features are extracted per view and projected into a unified 3D space for anomaly scoring.
Output: 3D anomaly volumes containing per-voxel anomaly scores and per-category binary segmentation masks, plus a global anomaly score per object.
Scoring recipe
def compute_metrics(gold_labels, pred_scores, gold_masks, pred_masks):
# Detection: AUROC per category, then mean
i_auroc_per_cat = [compute_auroc(gold_labels[c], pred_scores[c]) for c in categories]
mean_i_auroc = np.mean(i_auroc_per_cat)
# Segmentation: AUPRO at 1% FPR per category, then mean
v_aupro_per_cat = [compute_aupro(gold_masks[c], pred_masks[c], fpr=0.01) for c in categories]
mean_v_aupro = np.mean(v_aupro_per_cat)
return mean_i_auroc, mean_v_aupro
Common pitfalls
- Synthetic-to-real domain shift causes severe performance drops and erratic behavior in some baselines, making cross-domain robustness a key differentiator.
- Baselines must be adapted from 2D to 3D by processing each view independently and aggregating via projection, which can introduce alignment errors if not handled correctly.
- Missing table entries indicate some competitors were not originally evaluated on SiM3D, so direct comparison requires careful adaptation of their original 2D pipelines.
Evidence (verbatim from paper)
We evaluate our method against state-of-the-art anomaly detection approaches adapted to the multiview and multimodal 3D anomaly detection scenario set forth by SiM3D [12]. The considered competitors include memory bank methods (PatchCore [27], BTF [19], M3DM [36]), teacher-student approaches (EfficientAD [1], AST [29]), and the original, single-view Crossmodal Feature Mapping (CFM [11]). Hence, as proposed in [12], all the competitors are adapted to produce 3D anomaly volumes by processing each view independently and aggregating the resulting 2D anomaly maps into the 3D space using the projection strategy described in the SiM3D paper. Tab. 1 and Tab. 2 report results on the real-to-real and synthetic-to-real setups of SiM3D, respectively, with qualitative results shown in Fig. 5. Real-to-real Setup. In the real-to-real setup (Tab. 1), our method (MODMAP) achieves the best performance in both the detection and segmentation tasks, with a mean I-AUROC of 0.844 and a mean V-AUPRO@ 1% of 0.804.
Citation
@misc{costanzino2026modulateandmap,
title={Modulate-and-Map: Crossmodal Feature Mapping with Cross-View Modulation for 3D Anomaly Detection},
author={Costanzino et al. (2026)},
year={2026},
note={arXiv:2604.02328}
}
- arXiv: 2604.02328