Code Merge Eval

code-merge-eval

CodeMerge: Codebook-Guided Model Merging for Robust Test-Time Adaptation in Autonomous Driving — Yang et al. (2025) (arXiv:2505.16524, 2025)

What this evaluates

Evaluates a model's ability to perform test-time adaptation (TTA) for 3D object detection and autonomous driving tasks under domain shifts and sensor corruptions. It probes robustness to environmental changes (weather, lighting) and hardware failures without full retraining.

Datasets

• KITTI — total ?; splits: test (-1)
• KITTI-C — total ?; splits: test (-1)
• Waymo — total ?; splits: test (-1)
• nuScenes — total ?; splits: train (-1), val (-1)
• nuScenes-C — total ?; splits: val (-1)

Metrics

• NDS (primary) — range: [0, 1]
  • NuScenes Detection Score, a weighted average of detection (mAP, mATE, mASE, mAOE, mAVE, mAAE) and tracking (AMOTA, AMOTP, Recall) metrics. Higher is better.
• mAP — range: [0, 1]
  • Mean Average Precision across object classes, computed using standard IoU thresholds. Higher is better.
• AP_3D / AP_BEV — range: [0, 1]
  • Average Precision for 3D bounding boxes and Bird's Eye View projections, reported at moderate difficulty. Higher is better.
• AMOTA — range: [0, 1]
  • Average Multi-Object Tracking Accuracy, measuring identity-consistent tracking performance. Higher is better.

Input / output format

Input: Raw sensor data (LiDAR point clouds and/or camera images) representing autonomous driving scenes.

Output: Predicted 3D bounding boxes, tracking IDs, and downstream task outputs (online mapping, motion prediction, trajectory planning).

Scoring recipe

def evaluate(preds, gold):
    det_scores = compute_coco_ap(preds, gold)  # mAP, mATE, mASE, mAOE, mAVE, mAAE
    track_scores = compute_tracking_metrics(preds, gold)  # AMOTA, AMOTP, Recall
    nds = weighted_average(det_scores, track_scores)
    return nds

# For corruptions: evaluate at highest severity level.
# For cross-dataset shifts: report AP_3D/AP_BEV and Closed Gap % relative to Oracle & No Adaptation.

Common pitfalls

• Evaluating on different corruption severity levels than the paper's specified 'highest severity'.
• Confusing AP_3D (3D bounding box precision) with AP_BEV (Bird's Eye View projection precision).
• Reporting 'Closed Gap' without normalizing against the Oracle and No Adaptation baselines as defined in the cross-dataset setup.

Evidence (verbatim from paper)

Table 1 shows CodeMerge consistently outperforms all baselines, including No Adapt, Tent, and the state-of-the-art MOS in averaged results. In 3D detection, we boost mAP by 33.6% over no adaptation (0.1747 → 0.2334) and by 13.3% over MOS. CodeMerge also reduces mASE by 4.4% relative to MOS, and lower mAVE by 19%.

Citation

@misc{yang2025codemerge,
  title={CodeMerge: Codebook-Guided Model Merging for Robust Test-Time Adaptation in Autonomous Driving},
  author={Yang et al. (2025)},
  year={2025},
  note={arXiv:2505.16524}
}

• arXiv: 2505.16524