Griffin Eval

griffin-eval

Griffin: Aerial-Ground Cooperative Detection and Tracking Dataset and Benchmark — Wang et al. (2025) (arXiv:2503.06983, 2025)

What this evaluates

Evaluates aerial-ground cooperative 3D object detection and multi-object tracking in simulated urban environments. Probes cross-view feature alignment, occlusion handling, and communication efficiency under dynamic drone altitudes.

Datasets

• Griffin — total 30000; splits: full (-1); repo https://github.com/wang-jh18-SVM/Griffin

Metrics

• AP (primary) — range: [0, 1]
  • Average Precision computed over multiple IoU thresholds to assess detection quality.
• AMOTA (primary) — range: [0, 1]
  • Average Multi-Object Tracking Accuracy assessing overall tracking quality across detection confidence thresholds.
• ATE — range: meters
  • Average Translation Error measuring the mean L2 distance between predicted and ground truth object positions.
• ASE — range: meters
  • Average Scale Error measuring the mean L2 distance between predicted and ground truth object dimensions.
• AOE — range: radians
  • Average Orientation Error measuring the mean angular difference between predicted and ground truth object headings.
• AVE — range: m/s
  • Average Velocity Error measuring the mean L2 distance between predicted and ground truth object velocities.
• BPS — range: bytes/s
  • Bytes per second, quantifying the communication overhead required for cooperative perception.
• AMOTP — range: meters
  • Average Multi-Object Tracking Precision measuring the mean L2 distance between tracked and ground truth positions.
• MT — range: count
  • Mostly Tracked trajectories, counting ground truth objects tracked for more than 80% of their lifespan.
• ML — range: count
  • Mostly Lost trajectories, counting ground truth objects tracked for less than 20% of their lifespan.
• IDS — range: count
  • ID Switches, counting the number of times a tracked object's identity label changes incorrectly.

Input / output format

Input: Sequential images from ground vehicle and aerial drone, plus relative pose matrix between agents.

Output: 3D bounding boxes (position, dimensions, orientation), semantic labels, confidence scores, and unique tracking IDs (for tracking task).

Scoring recipe

# Detection
matches = hungarian_match(pred_boxes, gt_boxes, iou_thresh=0.5)
AP = mean_precision_over_thresholds(matches)
ATE = mean(l2_norm(pred_pos - gt_pos))
ASE = mean(l2_norm(pred_scale - gt_scale))
AOE = mean(angle_diff(pred_orient - gt_orient))
AVE = mean(l2_norm(pred_vel - gt_vel))
BPS = total_bytes_transmitted / sequence_duration

# Tracking
tracks = associate_tracks(pred_ids, gt_ids, max_dist=0.5)
AMOTA = compute_tracking_accuracy(tracks, fp, fn, ids)
AMOTP = mean(l2_norm(pred_track_pos - gt_track_pos))
MT = count(tracks_tracked > 0.8 * seq_len)
ML = count(tracks_tracked < 0.2 * seq_len)
IDS = count(identity_switches_in_tracks)

Common pitfalls

• Using single-view ground truth instead of the union-intersection GT = (GT_g U GT_a) n R, causing false negatives in occluded regions.
• Ignoring the relative pose between agents when fusing aerial and ground views, leading to misaligned 3D boxes.
• Evaluating detection and tracking metrics independently without accounting for communication overhead (BPS), which is critical for cooperative systems.

Evidence (verbatim from paper)

For comprehensive evaluation, we employ standard metrics in 3D object detection [[26]]: Average Precision (AP) to assess detection quality at various thresholds, alongside Average Translation Error (ATE), Average Scale Error (ASE), Average Orientation Error (AOE), and Average Velocity Error (AVE) to measure prediction accuracy of object position, size, orientation, and velocity, respectively. To evaluate communication costs for cooperative methods, we additionally utilize Bytes per second (BPS) as a key metric. This suite of metrics provides a thorough assessment of detection performance.

Citation

@misc{wang2025griffin,
  title={Griffin: Aerial-Ground Cooperative Detection and Tracking Dataset and Benchmark},
  author={Wang et al. (2025)},
  year={2025},
  note={arXiv:2503.06983}
}

• arXiv: 2503.06983