Panmatch Eval

panmatch-eval

PanMatch: Unleashing the Potential of Large Vision Models for Unified Matching Models — Zhang et al. (2025) (arXiv:2507.08400, 2025)

What this evaluates

Evaluates a unified vision model's ability to perform correspondence matching across stereo disparity estimation, optical flow, and feature matching in a zero-shot setting. It probes cross-domain generalization and robustness to challenging conditions like occlusion, lighting changes, and non-Lambertian surfaces.

Datasets

• Middlebury — total ?; splits: train-h (-1), half (-1), quarter (-1)
• ETH3D — total ?; splits: train (-1)
• KITTI — total ?; splits: 2012 (-1), 2015 (-1)
• Infinigen — total ?; splits: val (-1)
• Spring — total ?; splits: val (-1)
• Sintel — total ?; splits: clean (-1), final (-1)
• Booster — total ?; splits: test (-1)

Metrics

• PCA x (primary) — range: percent
  • Percentage of Accuracy in threshold x pixel. Computes the fraction of valid pixels where the absolute displacement error is below a dataset-specific threshold (1px for ETH3D, 2px for Middlebury, 3px for KITTI, 1px for Infinigen/Spring).
• EPE — range: pixels
  • Endpoint Error. The mean L2 distance between predicted and ground-truth displacement vectors across all valid pixels.
• Bad x — range: percent
  • Percentage of pixels where the absolute error exceeds threshold x (e.g., Bad 1.0, Bad 2.0).
• D1-all — range: percent
  • Percentage of pixels with disparity error greater than 1 pixel or 5% of the ground truth disparity.

Input / output format

Input: Paired two-view images (left/right for stereo, consecutive frames for flow, or arbitrary pairs for feature matching).

Output: Dense displacement field (optical flow), disparity map, or dense correspondence map.

Scoring recipe

def compute_pca(pred, gt, threshold):
    valid = gt > 0
    error = np.abs(pred[valid] - gt[valid])
    return np.mean(error < threshold) * 100

def compute_epe(pred, gt):
    valid = gt != 0
    error = np.sqrt(np.sum((pred[valid] - gt[valid])**2, axis=-1))
    return np.mean(error)

Common pitfalls

• Thresholds for PCA vary by dataset (1px ETH3D, 2px Middlebury, 3px KITTI).
• Feature matching evaluation requires forward-backward circular consistency filtering before pose estimation.
• Zero-shot models are compared against RVC methods that are fine-tuned on benchmark training splits, making direct leaderboard ranking misleading for real-world robustness.

Evidence (verbatim from paper)

For stereo matching, we assess disparity accuracy on the Middlebury (train-h), ETH3D (train), and KITTI 2012 (train) datasets. We use percentage of accuracy in threshold x pixel (PCA x) as the evaluation metric, setting threshold 1px for ETH3D, 2px for Middlebury and 3px for KITTI, respectively.

Citation

@misc{zhang2025panmatch,
  title={PanMatch: Unleashing the Potential of Large Vision Models for Unified Matching Models},
  author={Zhang et al. (2025)},
  year={2025},
  note={arXiv:2507.08400}
}

• arXiv: 2507.08400