Dualcam Eval

dualcam-eval

DualCam: A Novel Benchmark Dataset for Fine-grained Real-time Traffic Light Detection — Jayarathne et al. (2022) (arXiv:2209.01357, 2022)

What this evaluates

This benchmark evaluates fine-grained real-time traffic light detection using synchronized dual-camera inputs. It probes a model's ability to accurately localize and classify multiple traffic light states across varying distances and sizes, while balancing detection speed and precision.

Datasets

• DualCam — total 2250; splits: test (-1); repo https://github.com/harinduravin/DualCam

Metrics

• F1-score (primary) — range: [0, 1]
  • Calculated per class as F1 = (2 * Precision * Recall) / (Precision + Recall), where Precision = TP / (TP + FP) and Recall = TP / (TP + FN). True positives are determined by an Intersection over Union (IoU) threshold of 0.3 between predicted and ground-truth bounding boxes.

Input / output format

Input: Synchronized image pairs from a narrow-angle camera and a wide-angle camera.

Output: Predicted bounding boxes with class labels and confidence scores for each traffic light instance.

Scoring recipe

def compute_f1(predictions, ground_truth, iou_threshold=0.3):
    tp, fp, fn = 0, 0, 0
    matched_gt = set()
    for pred in predictions:
        best_iou = 0
        best_idx = -1
        for i, gt in enumerate(ground_truth):
            if i not in matched_gt:
                iou = calculate_iou(pred.box, gt.box)
                if iou > best_iou:
                    best_iou, best_idx = iou, i
        if best_iou >= iou_threshold:
            tp += 1
            matched_gt.add(best_idx)
        else:
            fp += 1
    fn = len(ground_truth) - len(matched_gt)
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0

Common pitfalls

• IoU threshold for matching predictions to ground truth is explicitly set to 0.3 for dataset evaluation, not the standard 0.5.
• Class definitions require merging Green-left, Green-right, and Green-up into a single 'Green-arrows' super-class for evaluation.
• Ground truth includes very small bounding boxes (minimum 6 pixels for dataset eval), which heavily impacts recall for distant lights.

Evidence (verbatim from paper)

For the performance evaluation of object detectors on our dataset, we use $F_{1}$-score for each class. Recall is the proportion of correct predictions out of all ground truths. Precision is the proportion of correct predictions out of all predictions. The precision-recall curve is calculated from predictions ranked according to confidence score. TP indicates the total number of detected traffic lights (true positives), FN indicates the total number of undetected traffic lights (false negatives) and FP indicates the total number of predictions that cannot be attributed to any ground truth (false positives). A prediction is considered as a true positive based on the IoU value across the ground truth and predicted bounding boxes.

Citation

@misc{jayarathne2022dualcam,
  title={DualCam: A Novel Benchmark Dataset for Fine-grained Real-time Traffic Light Detection},
  author={Jayarathne et al. (2022)},
  year={2022},
  note={arXiv:2209.01357}
}

• arXiv: 2209.01357