Cam4docc Eval

cam4docc-eval

Cam4DOcc: Benchmark for Camera-Only 4D Occupancy Forecasting in Autonomous Driving Applications — Ma et al. (2023) (arXiv:2311.17663, 2023)

What this evaluates

Evaluates camera-only 4D occupancy forecasting for autonomous driving by predicting current and future voxel states in a 3D grid. It specifically probes the model's ability to distinguish between general movable objects (GMO) and general static objects (GSO) across multiple future time steps.

Datasets

• nuScenes — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/haomo-ai/Cam4DOcc
• nuScenes-Occupancy — total ?; splits: (unstated)
• Lyft-Level5 — total ?; splits: (unstated)

Metrics

• IoU_c — range: [0, 1]
  • Intersection over Union for current occupancy at t=0. Computed as the sum of predicted and ground truth voxel intersections divided by their union across H, W, L dimensions.
• IoU_f — range: [0, 1]
  • Average IoU over N_f future timestamps. Computed as the mean of per-timestep IoU scores across the forecasting horizon.
• ~IoU_f (primary) — range: [0, 1]
  • Weighted average IoU over the entire forecasting horizon. Closer timestamps contribute more via a 1/t weighting factor, aligning with the importance of near-term predictions for motion planning.

Input / output format

Input: A sequence of N_p past and current camera images I = {I_t}_{t=-N_p}^0.

Output: Current occupancy O_c in R^{1xHxWxL} and future occupancy O_f in R^{N_fxHxWxL}, where each voxel contains a binary state (free/occupied) for each of the N_f future timestamps.

Scoring recipe

def compute_iou(pred, gt):
    inter = np.sum(pred * gt)
    union = np.sum(pred) + np.sum(gt) - inter
    return inter / union if union > 0 else 0.0

def score(pred_Oc, pred_Of, gt_Oc, gt_Of, Nf):
    iou_c = compute_iou(pred_Oc, gt_Oc)
    iou_f = np.mean([compute_iou(pred_Of[t], gt_Of[t]) for t in range(1, Nf+1)])
    tilde_iou_f = 0.0
    for t in range(1, Nf+1):
        tilde_iou_f += (1.0/t) * np.mean([compute_iou(pred_Of[k], gt_Of[k]) for k in range(1, t+1)])
    tilde_iou_f /= Nf
    return {'IoU_c': iou_c, 'IoU_f': iou_f, '~IoU_f': tilde_iou_f}

Common pitfalls

• Lyft-Level5 lacks occupancy labels, so only the 'Forecasting inflated GMO' task is evaluated on it.
• Invalid instances are discarded if visibility is under 40% across 6 cameras, if they first appear in future frames, or if they move beyond the predefined spatial range.
• The weighted metric ~IoU_f intentionally down-weights distant future timestamps, so models optimized for long-horizon accuracy may score lower than expected.

Evidence (verbatim from paper)

Metrics. For all four tasks, we use intersection over union (IoU) as the performance metric. We separately evaluate the current moment ($t=0$) occupancy estimation and the future time ($t\in[1,N_{f}]$) forecasting by ... We also provide a singular quantitative indicator to evaluate forecasting performance within the whole time horizon using one value calculated by ... IoU of timestamps closer to the current moment contributes more to the final $\tilde{\text{IoU}}_{f}$. This aligns with the principle that occupancy predictions at near timestamps are more crucial for subsequent motion planning and decision making.

Citation

@misc{ma2023cam4docc,
  title={Cam4DOcc: Benchmark for Camera-Only 4D Occupancy Forecasting in Autonomous Driving Applications},
  author={Ma et al. (2023)},
  year={2023},
  note={arXiv:2311.17663}
}

• arXiv: 2311.17663