Synthverse Eval

synthverse-eval

SynthVerse: A Large-Scale Diverse Synthetic Dataset for Point Tracking — Zhao et al. (2026) (arXiv:2602.04441, 2026)

What this evaluates

This benchmark evaluates 2D and 3D point tracking capabilities across diverse synthetic domains, including rapid camera motion, articulated objects, and occlusions. It probes a model's ability to maintain spatio-temporal correspondence, handle depth-adaptive spatial errors, and correctly classify occlusion or out-of-frame status under significant distribution shifts.

Datasets

• SynthVerse — total 5800000; splits: train (5800000), test (-1)

Metrics

• AJ_3D (primary) — range: percent
  • Calculates the proportion of points whose predicted 3D positions fall within a depth-adaptive error tolerance relative to ground truth, assessing spatial and visibility overlap in 3D coordinates.
• APD_3D — range: other
  • Measures the average Euclidean distance between predicted and ground truth 3D trajectories.
• AJ_2D — range: percent
  • Measures the spatio-temporal alignment of pixels across video sequences using the Intersection-over-Union metric after mapping 3D results back to 2D images.
• APD_2D — range: other
  • Measures whether the average Euclidean distance between predicted and ground truth trajectories on the pixel plane remains within minimal error bounds.
• OA — range: percent
  • Evaluates the accuracy of the model’s binary classification regarding whether a tracked point is occluded or out of frame.

Input / output format

Input: Video sequences with initial 2D/3D point coordinates or query points to track.

Output: Predicted 2D/3D trajectories and binary occlusion/out-of-frame labels per frame.

Scoring recipe

def compute_metrics(pred_2d, pred_3d, pred_occ, gt_2d, gt_3d, gt_occ):
    aj_3d = mean([1 if dist_3d(p, g) < depth_adaptive_tol(g) else 0 for p, g in zip(pred_3d, gt_3d)])
    apd_3d = mean([dist_3d(p, g) for p, g in zip(pred_3d, gt_3d)])
    aj_2d = iou(pred_2d, gt_2d)
    apd_2d = mean([dist_2d(p, g) for p, g in zip(pred_2d, gt_2d)])
    oa = accuracy(pred_occ, gt_occ)
    return {'AJ_3D': aj_3d, 'APD_3D': apd_3d, 'AJ_2D': aj_2d, 'APD_2D': apd_2d, 'OA': oa}

Common pitfalls

• Depth-adaptive tolerance for AJ_3D is not a fixed threshold but scales with scene depth, requiring careful implementation to avoid over/under-counting valid points.
• OA requires strict binary alignment with ground truth occlusion masks; partial occlusions or out-of-frame states must be explicitly handled to match evaluation conventions.
• 3D tracking performance degrades significantly under rapid viewpoint changes and large camera motion compared to 2D, so evaluating only on static scenes will overestimate capability.

Evidence (verbatim from paper)

Following TAPIP3D*[[35]]*, we adopt AJ_3D, APD_3D, AJ_2D, APD_2D, OA as the main evaluation metric. AJ_3D assesses tracking rigor by calculating the spatial and visibility overlap between predicted points and ground truth in 3D coordinates. AJ_3D quantifies the proportion of points whose predicted 3D positions fall within a depth-adaptive error tolerance relative to the actual coordinates. In addition, OA specifically evaluates the accuracy of the model’s binary classification regarding whether a tracked point is occluded or out of frame. After mapping results back to 2D images, AJ_2D measures the spatio-temporal alignment of pixels across video sequences using the Intersection-over-Union metric. Moreover, APD_2D measures whether the average Euclidean distance between the predicted trajectories and ground truth on the pixel plane remains within minimal error bounds.

Citation

@misc{zhao2026synthverse,
  title={SynthVerse: A Large-Scale Diverse Synthetic Dataset for Point Tracking},
  author={Zhao et al. (2026)},
  year={2026},
  note={arXiv:2602.04441}
}

• arXiv: 2602.04441