Orad 3d Eval

orad-3d-eval

Advancing Off-Road Autonomous Driving: The Large-Scale ORAD-3D Dataset and Comprehensive Benchmarks — Chen Min et al. (2025) (arXiv:2510.16500, 2025)

What this evaluates

Evaluates off-road autonomous driving capabilities across perception, planning, and world modeling. It probes 2D free-space detection, 3D semantic occupancy prediction, GPS-guided trajectory planning, VLM-based scene understanding and path planning, and future video generation in unstructured, variable-terrain environments.

Datasets

• ORAD-3D — total 57000; splits: test (-1); repo https://github.com/chaytonmin/ORAD-3D

Metrics

• mIoU (primary) — range: [0, 1]
  • Mean Intersection over Union across all semantic classes for 2D free-space and 3D occupancy segmentation tasks.
• IoU — range: [0, 1]
  • Intersection over Union for the primary occupancy class in 3D prediction.
• FDE — range: other
  • Final Displacement Error: Euclidean distance between the predicted and ground truth endpoint of the trajectory.
• minADE1 — range: other
  • Minimum Average Displacement Error over 1 predicted trajectory: mean Euclidean distance between predicted and ground truth points over the horizon.
• HitRate1,2m — range: [0, 1]
  • Percentage of predictions where the endpoint falls within 1m or 2m of the ground truth trajectory.
• L2(m) — range: other
  • L2 trajectory error at specific time horizons (1s, 2s, 3s) and averaged, measuring Euclidean distance between predicted and actual vehicle positions.
• Failure Rate(%) — range: percent
  • Percentage of test cases where the VLM-generated trajectory deviates beyond a safety threshold or fails to reach the goal.
• FID — range: other
  • Fréchet Inception Distance: measures the distance between feature distributions of real and generated future driving videos.
• FVD — range: other
  • Fréchet Video Distance: temporal extension of FID that accounts for video dynamics and motion coherence.

Input / output format

Input: Synchronized RGB images and LiDAR point clouds per frame; rough GPS coordinates for path planning; image prompts for VLM tasks.

Output: Per instance: 2D/3D semantic segmentation masks, predicted trajectory waypoints, textual scene analysis with driving decisions (VLM), or generated future video frames.

Scoring recipe

def compute_metrics(preds, gold):
    res = {}
    # Segmentation
    res['mIoU'] = np.mean([np.sum((p & g) > 0) / np.sum((p | g) > 0) for p, g in zip(preds['seg'], gold['seg'])])
    res['IoU'] = np.mean([np.sum((p & g) > 0) / np.sum((p | g) > 0) for p, g in zip(preds['occ'], gold['occ'])])
    # Path Planning
    res['FDE'] = np.linalg.norm(preds['traj'][-1] - gold['traj'][-1], axis=1).mean()
    res['minADE1'] = np.mean(np.min([np.mean(np.linalg.norm(preds['traj'] - gold['traj'], axis=2), axis=1)]))
    res['HitRate1,2m'] = np.mean(np.linalg.norm(preds['traj'][-1] - gold['traj'][-1], axis=1) < 2.0)
    # VLM Driving
    res['L2(m)'] = np.mean(np.linalg.norm(preds['vlm_traj'] - gold['traj'], axis=1))
    res['Failure Rate(%)'] = 100 * np.mean(preds['vlm_fail_mask'])
    # World Model
    res['FID'] = compute_frechet_distance(real_videos=gold['videos'], fake_videos=preds['videos'])
    res['FVD'] = compute_frechet_video_distance(real_videos=gold['videos'], fake_videos=preds['videos'])
    return res

Common pitfalls

• Confusing 2D free-space mIoU with 3D occupancy mIoU, as both use identical metric names but operate on different modalities and spatial resolutions.
• VLM failure rate and L2 error are highly sensitive to the exact prompt template and time horizon (1s/2s/3s); results are not directly comparable across different prompt variations.
• GPS-guided path planning metrics (FDE, minADE) assume a fixed prediction horizon and degrade significantly if initial GPS localization errors exceed the model's tolerance.

Evidence (verbatim from paper)

Quantitative results are reported in Table VI, where the fusion of LiDAR and vision yields higher prediction accuracy for 3D occupancy estimation. TABLE VI 3D SEMANTIC OCCUPANCY PREDICTION RESULTS ON ORAD-3D TEST SET. Method | IoU↑ | mIoU↑

Citation

@misc{min2025orad3d,
  title={Advancing Off-Road Autonomous Driving: The Large-Scale ORAD-3D Dataset and Comprehensive Benchmarks},
  author={Chen Min et al. (2025)},
  year={2025},
  note={arXiv:2510.16500}
}

• arXiv: 2510.16500