Rio 3rscan Eval

rio-3rscan-eval

RIO: 3D Object Instance Re-Localization in Changing Indoor Environments — Wald et al. (2019) (arXiv:1908.06109, 2019)

What this evaluates

Evaluates a model's ability to match 3D object patches and re-localize object instances in dynamically changing indoor environments. It measures feature matching robustness and 6DoF pose estimation accuracy under partial observations and contextual shifts.

Datasets

• 3RScan — total ?; splits: test (-1)

Metrics

• Recall <0.1m, 10° (primary) — range: percent
  • Percentage of object instances successfully aligned where translation error ≤ 0.1 m and rotation error ≤ 10°.
• Recall <0.2m, 20° — range: percent
  • Percentage of object instances successfully aligned where translation error ≤ 0.2 m and rotation error ≤ 20°.
• MRE [deg] — range: other
  • Median Rotation Error computed from the axis-angle representation of the predicted vs ground-truth rotation matrices.
• MTE [m] — range: other
  • Median Translation Error computed as the Euclidean distance between predicted and ground-truth translation vectors.

Input / output format

Input: RGB-D patches (TSDF) around annotated keypoints on object instances in changing indoor scenes.

Output: Predicted 6DoF pose (rotation matrix R_p and translation vector t_p) for each object instance.

Scoring recipe

def compute_recall_and_errors(predictions, ground_truth, t_thresh=0.1, r_thresh=10.0):
    correct = 0
    r_errors = []
    t_errors = []
    for pred, gt in zip(predictions, ground_truth):
        t_delta = np.linalg.norm(pred['t'] - gt['t'])
        R_delta = np.dot(pred['R'].T, gt['R'])
        r_delta = np.arccos(np.clip((np.trace(R_delta) - 1) / 2, -1, 1)) * 180 / np.pi
        r_errors.append(r_delta)
        t_errors.append(t_delta)
        if t_delta <= t_thresh and r_delta <= r_thresh:
            correct += 1
    recall = correct / len(predictions)
    mre = np.median(r_errors)
    mte = np.median(t_errors)
    return recall, mre, mte

Common pitfalls

• Symmetries of object instances are explicitly considered in the error computation, which must be accounted for when calculating rotation/translation deltas.
• The top-1 matching metric uses 50 randomly chosen negative patches per positive keypoint, differing from standard 1:1 matching.
• Evaluation is performed on the test set of 3RScan, which contains dynamically changing environments, not static scenes.

Evidence (verbatim from paper)

We evaluate the predicted rotation R_p and translation t_p against the ground truth annotation R_GT and t_GT according to equation [5] and [4]. An instance has successfully been aligned if the alignment error for the translation t_Δ and rotation R_Δ are lower than t≤0.1m, r≤10° or t≤0.2m, r≤20°. Please note that respective symmetry are considered in the error computation:

Citation

@misc{wald2019rio,
  title={RIO: 3D Object Instance Re-Localization in Changing Indoor Environments},
  author={Wald et al. (2019)},
  year={2019},
  note={arXiv:1908.06109}
}

• arXiv: 1908.06109