Mms Vpr Eval

mms-vpr-eval

MMS-VPR: Multimodal Street-Level Visual Place Recognition Dataset and Benchmark — Ou et al. (2025) (arXiv:2505.12254, 2025)

What this evaluates

Evaluates multimodal street-level visual place recognition by classifying pedestrian-view locations into graph-based spatial units (nodes, edges, or combined). It probes the model's ability to fuse image, video, and textual metadata for robust geolocalization in complex urban environments.

Datasets

• MMS-VPR — total ?; splits: (unstated)

Metrics

• Accuracy (primary) — range: [0, 1]
  • Fraction of correctly classified instances out of the total number of instances in the evaluation set. Macro-averaged across classes.
• F1-score — range: [0, 1]
  • Harmonic mean of Precision and Recall, computed per class and macro-averaged.
• MSE — range: [0, inf)
  • Mean Squared Error between predicted and ground-truth representations, used to compare unimodal vs multimodal fusion effectiveness.

Input / output format

Input: Multimodal features per location: video frames sampled and encoded via CLIP, static images converted to grayscale, resized to 32×32, and flattened to 1024-dim vectors, and textual metadata (store names/signage) embedded via BERT [CLS] token. These are fused via an MLP and concatenated before a shared classification head, optionally incorporating graph structural information (nodes/edges).

Output: Class label prediction corresponding to one of the 82 node classes, 125 edge classes, or combined classes, depending on the experimental scenario.

Scoring recipe

def compute_accuracy(preds, gold):
    correct = sum(p == g for p, g in zip(preds, gold))
    return correct / len(gold)
# Results are averaged over 5 runs; reported values are mean ± std × 1000
# e.g., 0.124±6 represents 0.124 ± 0.006

Common pitfalls

• Reported table values are scaled by 1000 (e.g., 0.124±6 means 0.124±0.006), which can be misread as raw percentages or unnormalized scores.
• The dataset uses a spatial graph where 'nodes' and 'edges' refer to physical road intersections and segments, not abstract GNN graph components, which may cause confusion during implementation.

Evidence (verbatim from paper)

We provide the results of place recognition on three variants: Edge-only, Node-only, and Combined. ... Results are averaged over 5 runs, where values are presented as (mean±std × 1000) (e.g., 0.124±6 represents 0.124±0.006). Table 2: Place recognition on Edge-only settings. ... | Accuracy | 0.132±5 |

Citation

@misc{ou2025mmsvpr,
  title={MMS-VPR: Multimodal Street-Level Visual Place Recognition Dataset and Benchmark},
  author={Ou et al. (2025)},
  year={2025},
  note={arXiv:2505.12254}
}

• arXiv: 2505.12254