Mushroom Segmentation Eval

mushroom-segmentation-eval

A Scalable Pipeline Combining Procedural 3D Graphics and Guided Diffusion for Photorealistic Synthetic Training Data Generation in White Button Mushroom Segmentation — Károly et al. (2025) (arXiv:2512.08747, 2025)

What this evaluates

This benchmark evaluates the zero-shot instance segmentation capability of models trained on synthetic data when applied to real-world agricultural scenes. It probes the model's ability to generalize across domain gaps, handling varying lighting, mushroom densities, and developmental stages without fine-tuning on real annotations.

Datasets

• Real-world on-field dataset — total 195; splits: test (195)
• M18K — total 20; splits: test (20)

Metrics

• F1 score (primary) — range: [0, 1]
  • Harmonic mean of precision and recall: 2 * (Precision * Recall) / (Precision + Recall). Used as the primary metric for model selection and comparison.
• Average Precision (AP) — range: [0, 1]
  • Area under the precision-recall curve across multiple IoU thresholds or confidence thresholds, measuring detection accuracy.
• Average Recall (AR) — range: [0, 1]
  • Area under the recall curve across multiple IoU thresholds or confidence thresholds, measuring the model's ability to find all ground-truth instances.
• mean Intersection over Union (mIoU) — range: [0, 1]
  • Average IoU across all predicted and ground-truth instance masks: IoU = |Prediction ∩ Ground Truth| / |Prediction ∪ Ground Truth|.
• Fréchet Inception Distance (FID) — range: other
  • Statistical distance between feature distributions of real and generated images using a pre-trained Inception network. Lower values indicate higher visual similarity.
• Kernel Inception Distance (KID) — range: other
  • Squared maximum mean discrepancy between feature distributions of real and generated images using a polynomial kernel. Lower values indicate higher visual similarity.

Input / output format

Input: RGB images of white button mushrooms at 512x512 or 1024x1024 resolution, captured in real-world agricultural settings or from the M18K dataset.

Output: Instance-level segmentation masks and bounding boxes for each detected mushroom, along with confidence scores.

Scoring recipe

def compute_metrics(predictions, ground_truth):
    ious = compute_iou(predictions, ground_truth)
    tp = count_true_positives(ious)
    fp = len(predictions) - tp
    fn = len(ground_truth) - tp
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    ap = average_precision_curve(predictions, ground_truth)
    ar = average_recall_curve(predictions, ground_truth)
    miou = mean(ious)
    return {'F1': f1, 'AP': ap, 'AR': ar, 'mIoU': miou}

Common pitfalls

• Incomplete labeling in the M18K test set causes correctly detected mushrooms to be counted as false positives, artificially lowering AP.
• FID and KID scores measure visual similarity but do not reliably correlate with downstream segmentation performance or human preference.
• Models trained on synthetic data may exhibit conservative detection behavior, favoring precision over recall on datasets with many small or ambiguous objects.

Evidence (verbatim from paper)

For model comparison, we report standard instance segmentation metrics: Average Precision (AP), Average Recall (AR), F1 score, and mean Intersection over Union (mIoU). The results of the first experiment, comparing the MRCNN-B and MRCNN-SD models on our real-world dataset, are presented in Table [1].

Citation

@misc{karoly2025mushroomsegmentation,
  title={A Scalable Pipeline Combining Procedural 3D Graphics and Guided Diffusion for Photorealistic Synthetic Training Data Generation in White Button Mushroom Segmentation},
  author={Károly et al. (2025)},
  year={2025},
  note={arXiv:2512.08747}
}

• arXiv: 2512.08747