Matchbench Eval

matchbench-eval

MatchBench: An Evaluation of Feature Matchers — Bian et al. (2018) (arXiv:1808.02267, 2018)

What this evaluates

Evaluates the matching ability, correspondence sufficiency, and computational efficiency of local feature matchers across short- and wide-baseline image pairs. It measures how well matchers recover camera pose and how many correct correspondences they produce, enabling fair comparison for real-time applications like SLAM.

Datasets

• SfM/SLAM datasets (sequences 01-08) — total ?; splits: test (-1)

Metrics

• AUC (SP curve) (primary) — range: [0, 1]
  • Area under the Success Ratio (SP) curve, which plots the fraction of correctly estimated camera poses against varying pose error thresholds.
• AP (correspondence sufficiency) — range: other
  • Average number of correspondences computed over image pairs that yield a correct pose estimate.
• Time consumption — range: other
  • Wall-clock time in milliseconds for feature detection, matching, and correspondence selection.

Input / output format

Input: Pairs of images (short-baseline or wide-baseline) from SfM/SLAM sequences.

Output: Set of matched keypoint correspondences, estimated relative camera pose (essential matrix), and execution time logs.

Scoring recipe

# For each image pair:
# 1. Extract features & match (FLANN+Euclidean for real-valued, BF+Hamming for binary)
# 2. Estimate pose (OpenCV 5-point RANSAC for sparse matchers, RepMatch estimator for rich matchers)
# 3. Compute pose error threshold vs success ratio -> plot SP curve -> compute AUC
# 4. Count matches for correctly matched pairs -> average -> AP
# 5. Record detection, matching, selection times in ms

Common pitfalls

• Using the default OpenCV pose estimator for rich matchers (CODE, RepMatch, GMS) instead of the RepMatch built-in estimator, which fails on dense correspondences.
• Not overriding ORB's default nfeatures=500 to 100,000, artificially capping feature detection.
• Evaluating slow matchers (KVLD, GAIM, CODE, RepMatch) on short-baseline tasks where real-time efficiency is required.
• Applying Euclidean distance to binary descriptors or Hamming distance to real-valued features.

Evidence (verbatim from paper)

The AUC score of matchers is shown in Tab.2, and the time consumption of matchers is shown in Tab.3. These results enable us to analyze the matching ability, correspondence sufficiency, as well as efficiency of different matchers.

Citation

@misc{bian2018matchbench,
  title={MatchBench: An Evaluation of Feature Matchers},
  author={Bian et al. (2018)},
  year={2018},
  note={arXiv:1808.02267}
}

• arXiv: 1808.02267