M3cad Eval

m3cad-eval

M3CAD: Towards Generic Cooperative Autonomous Driving Benchmark — Zhu et al. (2025) (arXiv:2505.06746, 2025)

What this evaluates

Evaluates cooperative autonomous driving capabilities across perception, mapping, motion forecasting, occupancy prediction, and path planning. It probes whether multi-vehicle cooperation and realistic, non-straight trajectories improve ego-vehicle performance compared to single-vehicle baselines.

Datasets

• M3CAD — total 204; splits: test (-1); repo https://github.com/zhumorui/M3CAD

Metrics

• AMOTA (primary) — range: [0, 1]
  • Average Multi-Object Tracking Accuracy. Computed using the same protocol as UniAD, measuring tracking consistency and detection quality over sequences.
• minADE — range: meters
  • Minimum Average Displacement Error. The lowest average L2 distance between predicted and ground truth future trajectories across multiple sampled hypotheses.
• L2 — range: meters
  • Average L2 distance between the predicted ego-vehicle trajectory and the ground truth trajectory over the planning horizon.
• Col. — range: percent
  • Average collision probability. Measures the likelihood of the planned trajectory intersecting with predicted agent occupancy or static obstacles.

Input / output format

Input: Multi-view camera images, LiDAR point clouds, GPS/IMU data, and optionally shared Bird's Eye View (BEV) features from cooperating vehicles.

Output: Per-frame object detections (bounding boxes), tracking IDs, lane/road segmentation masks, future trajectories for surrounding agents, 3D occupancy grids, and ego-vehicle planned trajectory.

Scoring recipe

def compute_metrics(preds, gold):
    # Detection/Tracking (per UniAD protocol)
    mAP = compute_map(preds.dets, gold.dets)
    AMOTA = compute_amota(preds.tracks, gold.tracks)
    # Motion Forecasting
    minADE = min(np.mean(np.linalg.norm(preds.trajectories - gold.trajectories, axis=-1), axis=0))
    # Planning
    L2 = np.mean(np.linalg.norm(preds.ego_traj - gold.ego_traj, axis=1))
    Col = np.mean(preds.collision_mask)
    return {'mAP': mAP, 'AMOTA': AMOTA, 'minADE': minADE, 'L2': L2, 'Col': Col}

Common pitfalls

• Training cooperative models with only local ground truth instead of global ground truth leads to incomplete supervision and significant performance drops.
• Assuming straight-line ego trajectories (common in NuScenes) generalize to complex cooperative scenarios; M3CAD's realistic non-straight paths heavily depend on perception data rather than just ego state.
• Failing to transform shared BEV features into the ego vehicle's coordinate frame before fusion, causing misaligned perception inputs.

Evidence (verbatim from paper)

Table 3 shows single-vehicle (S.) and cooperative (C.) tracking and mapping performance, using the same metrics defined in UniAD. ... Specifically, M3CAD yields a 2.4x improvement in minADE (minimum Average Displacement Error), likely due to its inclusion of multi-vehicle interactions, which enhance the model's understanding of object dynamics.

Citation

@misc{zhu2025m3cad,
  title={M3CAD: Towards Generic Cooperative Autonomous Driving Benchmark},
  author={Zhu et al. (2025)},
  year={2025},
  note={arXiv:2505.06746}
}

• arXiv: 2505.06746