Cimi4d Annotation Eval

cimi4d-annotation-eval

CIMI4D: A Large Multimodal Climbing Motion Dataset under Human-scene Interactions — Ming Yan et al. (2023) (arXiv:2303.17948, 2023)

What this evaluates

Probes the accuracy of automatically generated 3D human pose and translation annotations for rock climbing motions. It evaluates how well a LiDAR-IMU fusion and blending optimization pipeline reconstructs off-ground climbing poses compared to manual ground truth.

Datasets

• CIMI4D — total 180000; splits: eval (-1)

Metrics

• PMPJPE (primary) — range: other
  • Procrustes-Aligned Mean Per Joint Position Error. Computed by first aligning predicted and ground-truth joint sets via rigid Procrustes analysis, then averaging the Euclidean distances across all joints. Measured in millimeters.
• MPJPE — range: other
  • Mean Per Joint Position Error. Computed as the average Euclidean distance between predicted and ground-truth joint positions without alignment. Measured in millimeters.
• PCK0.5 — range: [0, 1]
  • Percentage of Correct Keypoints. The fraction of predicted joints whose distance to the ground-truth joint is below a threshold of 0.5. Reported as a ratio.
• PVE — range: other
  • Per Vertex Error. The average Euclidean distance between corresponding vertices of predicted and ground-truth 3D meshes. Measured in millimeters.
• ACCEL — range: other
  • Acceleration Error. The average difference in joint acceleration between predicted and ground-truth sequences. Measured in m/s².

Input / output format

Input: Synchronized RGB images, LiDAR point clouds, IMU pose/translation data, and high-precision static point clouds of climbing walls.

Output: Reconstructed 3D human joint positions and body translation, evaluated against manually annotated ground truth poses.

Scoring recipe

def compute_mpjpe(pred, gt):
    return np.mean(np.linalg.norm(pred - gt, axis=2))
def compute_pmpjpe(pred, gt):
    aligned_pred = procrustes_align(pred, gt)
    return np.mean(np.linalg.norm(aligned_pred - gt, axis=2))
def compute_pve(pred_mesh, gt_mesh):
    return np.mean(np.linalg.norm(pred_mesh - gt_mesh, axis=1))
def compute_accel(pred_j, gt_j):
    pred_a = np.diff(pred_j, axis=0, n=2)
    gt_a = np.diff(gt_j, axis=0, n=2)
    return np.mean(np.linalg.norm(pred_a - gt_a, axis=2))

Common pitfalls

• ACCEL is reported in m/s² while all other error metrics (PMPJPE, MPJPE, PVE) are in millimeters.
• PMPJPE requires Procrustes alignment of predicted and ground-truth poses before computing joint errors, unlike raw MPJPE.
• The reported metrics evaluate the quality of the dataset's annotation pipeline rather than downstream task performance like pose prediction or generation.

Evidence (verbatim from paper)

Evaluation metrics. In this section and in Sec. 5, we report Procrustes-Aligned Mean Per Joint Position Error (PMPJPE), Mean Per Joint Position Error (MPJPE), Percentage of Correct Keypoints (PCK), Per Vertex Error (PVE), and Acceleration error(m/s^{2}) (ACCEL). Except ACCEL, error metrics are measured in millimeters.

Citation

@misc{yan2023cimi4d,
  title={CIMI4D: A Large Multimodal Climbing Motion Dataset under Human-scene Interactions},
  author={Ming Yan et al. (2023)},
  year={2023},
  note={arXiv:2303.17948}
}

• arXiv: 2303.17948