Colo Dataset Eval

colo-dataset-eval

A Model Generalization Study in Localizing Indoor Cows with COw LOcalization (COLO) dataset — Das et al. (2024) (arXiv:2407.20372, 2024)

What this evaluates

Evaluates object detection and localization capabilities for indoor cows under varying camera viewpoints (top, side, external) and lighting conditions (day, night). It probes model generalization across domain shifts in perspective and illumination, testing whether pre-trained weights and model complexity transfer effectively to agricultural environments.

Datasets

• COLO — total 1254; splits: 0_all (-1), 1_top (-1), 2_side (-1), 3_external (-1), a1_t2s (-1), a2_s2t (-1), b_light (-1), c_external (-1); HF Niche-Squad/COLO; repo https://github.com/Niche-Squad/COLO

Metrics

• mAP@0.5:0.95 (primary) — range: [0, 1]
  • Mean Average Precision computed by averaging the AP at each IoU threshold from 0.5 to 0.95 in steps of 0.05. It requires both high localization accuracy and high classification confidence.
• mAP@0.5 — range: [0, 1]
  • Mean Average Precision computed at a single IoU threshold of 0.5. It measures detection accuracy with a lenient localization requirement.
• precision — range: [0, 1]
  • Ratio of true positive detections to the total number of positive predictions (TP / (TP + FP)).
• recall — range: [0, 1]
  • Ratio of true positive detections to the total number of actual ground truth objects (TP / (TP + FN)).

Input / output format

Input: RGB images of indoor farm environments containing cows, captured from top-view, side-view, external, or day/night lighting conditions.

Output: Per image: a list of bounding box predictions in [x_center, y_center, width, height] format, class label 'cow', and confidence score.

Scoring recipe

def compute_detection_metrics(preds, gts, iou_thresh=0.5):
    tp, fp, fn = 0, 0, 0
    for pred, gt in zip(preds, gts):
        best_iou = max([box_iou(p['bbox'], g['bbox']) for g in gt])
        if best_iou >= iou_thresh:
            tp += 1
        else:
            fp += 1
    fn = len(gts) - tp
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    return precision, recall
# mAP@0.5:0.95 averages AP across IoU thresholds [0.5, 0.55, ..., 0.95]

Common pitfalls

• Assuming larger model parameter counts consistently improve generalization; the study shows simpler models (e.g., YOLOv8n) often outperform complex ones in this domain.
• Overestimating the impact of lighting changes (Day2Night) compared to camera angle shifts; viewpoint changes cause significantly larger performance drops than illumination changes.
• Treating mAP@0.5 and mAP@0.5:0.95 as interchangeable; the former is sufficient for counting tasks, while the latter strictly evaluates precise localization.

Evidence (verbatim from paper)

To assess the performance of the YOLO models, we used four key metrics: mAP@0.5:0.95, mAP@0.5, precision, and recall. These metrics provide a comprehensive understanding of how well the models detect and localize cows in the images from the COLO dataset. The mAP@0.5:0.95 metric is the most stringent, requiring the model to achieve both high positioning accuracy (i.e., high IoU) and high precision across IoU thresholds from 0.5 to 0.95.

Citation

@misc{das2024colo,
  title={A Model Generalization Study in Localizing Indoor Cows with COw LOcalization (COLO) dataset},
  author={Das et al. (2024)},
  year={2024},
  note={arXiv:2407.20372}
}

• arXiv: 2407.20372