qhjqhj00 panoptic-studio-eval
Panoptic Studio: A Massively Multiview System for Social Interaction Capture — Joo et al. (2016) (arXiv:1612.03153, 2016)
What this evaluates
Evaluates the accuracy and robustness of a massively multiview 3D motion capture system for reconstructing full-body skeletal trajectories of multiple interacting people under severe occlusions and natural social interactions.
Datasets
- Panoptic Studio — total ?; splits: (unstated); repo https://domedb.perception.cs.cmu.edu
Metrics
PCK(primary) — range: percent- Probability of Correct Keypoint. Computes the fraction of 3D joints whose Euclidean distance to the ground truth (full-camera reconstruction) falls within a specified physical threshold (in cm).
Node Acc.— range: percent- Percentage of correctly reconstructed 3D joint nodes. Calculated as (Total Nodes - Outlier Nodes) / Total Nodes, where outliers are manually annotated by verifying 2D projections.
Skel. Acc.— range: percent- Percentage of correctly reconstructed 3D skeletons without any incorrect joints. Calculated as Correctly Reconstructed Skeletons / Total Skeletons.
Input / output format
Input: Synchronized multi-view video feeds (VGA, HD, RGB-D) with camera calibration data.
Output: 3D skeletal trajectories / 3D pose reconstruction results (joint coordinates over time).
Scoring recipe
def compute_pck(pred_joints, gt_joints, threshold_cm):
dists = np.linalg.norm(pred_joints - gt_joints, axis=-1)
return (dists <= threshold_cm).mean() * 100
def compute_node_acc(total_nodes, outlier_nodes):
return (total_nodes - outlier_nodes) / total_nodes * 100
def compute_skel_acc(correct_skeletons, total_skeletons):
return correct_skeletons / total_skeletons * 100
Common pitfalls
- The PCK metric uses physical 3D distance (cm) derived from calibration data, not the standard 2D torso/head ratio threshold.
- The 'ground truth' for PCK is actually the system's own output using all 480 VGA cameras after manual outlier exclusion, not independent motion-capture markers.
- Evaluations are performed on specific held-out sequences (e.g., ultimatum1, mafia2) rather than a standardized train/val/test split.
Evidence (verbatim from paper)
As an evaluation metric, we use the PCK (Probability of Correct Keypoint) metric, which is commonly used to evaluate 2D pose detectors. Here, we use 3D distance in physical scale (cm) obtained from calibration data for the threshold of PCK, in contrast to the 2D ratio of torso/head as in 2D pose detection cases. Among about 1.2 million body joints, about 8.7K nodes are determined as outliers or missed (rejected by thresholds of our system), showing 99.29% accuracy in node reconstruction. And, 93.55% out of about 82K 3D skeletons are correctly reconstructed without any incorrect joints.
Citation
@misc{joo2016panopticstudio,
title={Panoptic Studio: A Massively Multiview System for Social Interaction Capture},
author={Joo et al. (2016)},
year={2016},
note={arXiv:1612.03153}
}
- arXiv: 1612.03153