qhjqhj00 cosmos-drive-dreams-eval
Cosmos-Drive-Dreams: Scalable Synthetic Driving Data Generation with World Foundation Models — Ren et al. (2025) (arXiv:2506.09042, 2025)
What this evaluates
Evaluates the effectiveness of a synthetic driving data generation pipeline by measuring performance gains in downstream autonomous driving perception tasks, including 3D lane detection, 3D object detection, and LiDAR-based detection, particularly under challenging conditions like extreme weather and nighttime.
Datasets
- Waymo Open Dataset — total ?; splits: train (504), test (144)
- RDS-HQ — total ?; splits: train (-1), test (2800)
- RDS-HQ-HL — total 12000; splits: train (10000), test (2000)
Metrics
F1-score(primary) — range: [0, 1]- Harmonic mean of precision and recall for 3D lane detection, computed per weather/time split.
Category Accuracy— range: [0, 1]- Percentage of correctly classified lane categories.
LET-AP— range: [0, 1]- Average Precision metric adapted for LiDAR/BEV object detection, evaluated at varying IoU thresholds for the vehicle category.
mAP— range: [0, 1]- Mean Average Precision across object categories for LiDAR-based 3D object detection.
Input / output format
Input: Rectified camera images (960x540 to 960x720), HDMap annotations, and optionally LiDAR point clouds (20 uniformly sampled frames per clip).
Output: Predicted 3D lane coordinates/segments, 3D bounding boxes for vehicles, or LiDAR point cloud detections.
Scoring recipe
def compute_f1(predictions, golds, iou_thresh=0.5):
tp = sum(1 for p, g in zip(predictions, golds) if match(p, g, iou_thresh))
fp = len(predictions) - tp
fn = len(golds) - tp
prec = tp / (tp + fp) if (tp + fp) > 0 else 0
rec = tp / (tp + fn) if (tp + fn) > 0 else 0
return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
def compute_map(predictions, golds, iou_thresh=0.5):
scores, preds, golds = sort_by_confidence(predictions, golds)
tp, fp = 0, 0
for s, p, g in zip(scores, preds, golds):
if match(p, g, iou_thresh): tp += 1
else: fp += 1
return tp / (tp + fp) if (tp + fp) > 0 else 0
Common pitfalls
- The synthetic-to-real ratio ($R_{s2r}$) controls data mixing per training epoch, not the total dataset size.
- Corner-case splits (extreme weather/night) are curated subsets of the test set, not independent datasets.
- Metrics are reported per weather/time split, requiring careful aggregation rather than a single global score.
Evidence (verbatim from paper)
We report the F1-score and category accuracy of 3D lane detection trained on the Waymo Open Dataset and RDS-HQ (2k) in Tab. 1. The results show that Cosmos-Drive-Dreams significantly improves detection performance in cases where Albumumentations provide limited benefit.
Citation
@misc{ren2025cosmosdrivedreams,
title={Cosmos-Drive-Dreams: Scalable Synthetic Driving Data Generation with World Foundation Models},
author={Ren et al. (2025)},
year={2025},
note={arXiv:2506.09042}
}
- arXiv: 2506.09042