Uniem3m Segmentation Eval

uniem3m-segmentation-eval

UniEM-3M: A Universal Electron Micrograph Dataset for Microstructural Segmentation and Generation — Wang et al. (2025) (arXiv:2508.16239, 2025)

What this evaluates

Evaluates an instance segmentation model's ability to accurately delineate and separate individual microstructural objects in high-resolution electron micrographs, particularly under varying instance densities.

Datasets

• UniEM-3M — total 5091; splits: test (-1)

Metrics

• mAP@0.5 (primary) — range: [0, 1]
  • Mean Average Precision computed at an IoU threshold of 0.5. AP for a single image is defined as |TP| / (|TP| + |FP| + |FN|), where predicted masks matching ground truth masks over the threshold are counted as TP. Mean AP is the average across all images.
• PQ@0.5 — range: [0, 1]
  • Panoptic Quality at an IoU threshold of 0.5, measuring combined segmentation and recognition quality for instance separation.

Input / output format

Input: Raw electron micrograph images (original resolution for evaluation; 1024×1024 crops used during training).

Output: Predicted instance masks and associated bounding boxes/labels per image.

Scoring recipe

def compute_ap(pred_masks, gt_masks, iou_thresh=0.5):
    tp, fp, fn = 0, 0, 0
    for pred, gt in match_instances(pred_masks, gt_masks, iou_thresh):
        if iou(pred, gt) >= iou_thresh:
            tp += 1
        else:
            fp += 1
    fn = len(gt_masks) - tp
    return tp / (tp + fp + fn)

def compute_map(predictions, ground_truths):
    return mean([compute_ap(p, g) for p, g in zip(predictions, ground_truths)])

Common pitfalls

• The paper uses a simplified AP definition (single IoU threshold precision) rather than the standard COCO multi-threshold AP.
• Anchor-based methods fail in dense scenes due to NMS bottlenecks, so flow/vector-field methods are required for high-instance-density micrographs.
• Evaluation is performed on original resolution images, not the 1024×1024 training crops.

Evidence (verbatim from paper)

Evaluation metrics. We adopt standard metrics for instance segmentation: mean Average Precision (mAP) for mask accuracy, and Panoptic Quality (PQ) (Kirillov et al. 2019) at 0.5 for combined segmentation and recognition quality. Following stardist (Schmidt et al. 2018), we use AP calculated by AP = |TP| / (|TP| + |FP| + |FN|) at an IoU threshold T, where the predicted masks matching true masks over T are regarded as TP. Mean AP is the average AP of all images.

Citation

@misc{wang2025uniem3m,
  title={UniEM-3M: A Universal Electron Micrograph Dataset for Microstructural Segmentation and Generation},
  author={Wang et al. (2025)},
  year={2025},
  note={arXiv:2508.16239}
}

• arXiv: 2508.16239