Nrc Downstream Eval

nrc-downstream-eval

Neural Radiance Field Codebooks — Wallingford et al. (2023) (arXiv:2301.04101, 2023)

What this evaluates

Evaluates the quality of learned object-centric 3D representations across unsupervised segmentation, embodied object navigation, and relative depth ordering tasks.

Datasets

• ProcTHOR — total ?; splits: train (-1), test (-1)
• RoboTHOR — total 89; splits: train (-1), val (-1), test (-1)
• CLEVR-3D — total ?; splits: val (500)
• NYU Depth — total ?; splits: test (-1)

Metrics

• ARI (primary) — range: [0, 1]
  • Adjusted Rand Index measures the agreement between predicted and ground truth pixel clusterings. FG-ARI restricts this calculation to foreground pixels only.
• SPL — range: [0, 1]
  • Success weighted by Path Length: (1/N) * sum(S_i * l_i / max(p_i, l_i)), where l_i is shortest path, p_i is taken path, S_i is binary success indicator.
• Success Rate — range: percent
  • Percentage of episodes where the agent stops within 1 meter of the goal object with it in view.
• Depth Order Accuracy — range: percent
  • Percentage of object pairs correctly ordered by depth (closer vs farther) based on mean pixel depth of predicted masks.

Input / output format

Input: RGB frames or video sequences from indoor scenes; ground truth segmentation masks and depth maps for evaluation.

Output: Predicted segmentation masks (cluster assignments per pixel), navigation policy actions (discrete move/rotate/stop), predicted object masks and depth values.

Scoring recipe

def compute_ari(predictions, ground_truth):
    return adjusted_rand_score(predictions, ground_truth)

def compute_spl(trajectories, goals):
    total = len(trajectories)
    spl_sum = 0.0
    for traj, goal in zip(trajectories, goals):
        success = 1.0 if traj.success_within_1m(goal) else 0.0
        shortest = compute_shortest_path(goal)
        taken = traj.length()
        spl_sum += success * (shortest / max(shortest, taken))
    return spl_sum / total

def compute_depth_acc(predicted_masks, gt_masks, gt_depths):
    correct = 0
    for pair in object_pairs:
        pred_mask = max(predicted_masks, key=lambda m: iou(m, gt_masks[pair]))
        pred_depth = mean_depth(pred_mask)
        gt_depth = gt_depths[pair]
        if (pred_depth < gt_depth) == (gt_depth < gt_depths[other]):
            correct += 1
    return correct / len(object_pairs)

Common pitfalls

• CLEVR-3D evaluation uses FG-ARI instead of standard ARI, restricting metrics to foreground pixels only.
• NYU Depth evaluation only considers object classes that were present in the ProcTHOR training set.
• Depth ordering accuracy relies on selecting the predicted mask with the largest IoU to the ground truth, which can amplify segmentation errors.

Evidence (verbatim from paper)

Adjusted random index (ARI) Yeung & Ruzzo (2001) measures the agreement between two clusterings and is a standard metric for unsupervised segmentation. In our case the two clusterings are the predicted and ground truth segmentations. Foreground adjusted random index only measures the ARI for pixels belonging to foreground objects. ... SPL is defined as $\frac{1}{N}\sum_{i=1}^{N}S_{i}\frac{\ell_{i}}{\max\left(p_{i},\ell_{i}\right)}$, where $l_{i}$ is the shortest possible path, $p_{i}$ is the taken path, & $S_{i}$ is the binary indicator of success for episode $i$.

Citation

@misc{wallingford2023nrc,
  title={Neural Radiance Field Codebooks},
  author={Wallingford et al. (2023)},
  year={2023},
  note={arXiv:2301.04101}
}

• arXiv: 2301.04101