Raspgrade Eval

raspgrade-eval

The RaspGrade Dataset: Towards Automatic Raspberry Ripeness Grading with Deep Learning — Mekhalfi et al. (2025) (arXiv:2505.08537, 2025)

What this evaluates

Evaluates deep learning models for real-time instance segmentation and ripeness classification of raspberries and punnets in industrial conveyor settings. It probes the model's ability to distinguish between five ripeness grades (OK, Dark, Light, Second, Waste) and background objects under conditions of color similarity and occlusion.

Datasets

• RaspGrade — total ?; splits: train (-1), val (-1)

Metrics

• mAP50 (primary) — range: percent
  • Mean Average Precision at an Intersection over Union (IoU) threshold of 0.50. Computed as the mean of the average precision scores across all object classes, summarizing the precision-recall curve.
• mAP50-95 — range: percent
  • Average Precision averaged across multiple IoU thresholds ranging from 0.50 to 0.95 with a step size of 0.05. Provides a comprehensive evaluation of localization accuracy across different strictness levels.

Input / output format

Input: RGB images resized to 1280×800 resolution containing raspberries and punnets.

Output: Per-instance bounding boxes and pixel-level segmentation masks, each assigned a class label (0: Punnet, 1: OK, 2: Dark, 3: Light, 4: Second, 5: Waste).

Scoring recipe

For each class c:
  TP = true positives (IoU >= 0.5 for box/mask)
  FP = false positives
  FN = false negatives
  Precision_c = TP / (TP + FP)
  Recall_c = TP / (TP + FN)
  AP_c = trapezoidal integration of Precision vs Recall curve
mAP50 = mean(AP_c for all classes)
mAP50-95 = mean(AP_c for IoU in [0.50, 0.55, ..., 0.95] for all classes)
Report best validation scores across training epochs.

Common pitfalls

• Class imbalance heavily impacts performance; Waste (Grade 5) has only 75 training samples and suffers from occlusion/bruises, leading to very low mAP.
• Color similarity between Dark (Grade 2) and Light (Grade 3) grades causes frequent misclassification, requiring careful loss weighting to balance.
• Metrics are reported on the validation set during training (best checkpoint), not on a held-out test set, which may overestimate generalization.

Evidence (verbatim from paper)

To evaluate the model’s performance, we use standard metrics in object detection and instance segmentation tasks. Specifically: Precision (Pre): measures the proportion of true positive detections out of all predicted detections, serving as an indicator of the accuracy of the model’s predictions. Recall (Rec): quantifies the proportion of true positive detections out of all actual ground truth instances, highlighting how well the model captures all relevant objects. Mean Average Precision (mAP50): represents the mean of the average precision scores computed for each object class, summarizing the precision-recall curve by averaging precision at various recall levels. mAP50-95: A variant of mAP, it computes average precision across multiple Intersection over Union (IoU) thresholds, ranging from 0.50 to 0.95, with a step size of 0.05.

Citation

@misc{mekhalfi2025raspgrade,
  title={The RaspGrade Dataset: Towards Automatic Raspberry Ripeness Grading with Deep Learning},
  author={Mekhalfi et al. (2025)},
  year={2025},
  note={arXiv:2505.08537}
}

• arXiv: 2505.08537