Geolocalization Eval

geolocalization-eval

G3: An Effective and Adaptive Framework for Worldwide Geolocalization Using Large Multi-Modality Models — Pengyue Jia et al. (arXiv:2405.14702, 2024)

What this evaluates

Evaluates a model's ability to predict the geographic coordinates (latitude and longitude) of a query image from worldwide visual data. It probes fine-grained location-aware visual semantics and robustness to geographical heterogeneity across urban, regional, and continental scales.

Datasets

• IM2GPS3k — total ?; splits: test (-1)
• YFCC4K — total ?; splits: test (-1)

Metrics

• threshold metric (primary) — range: percent
  • Percentage of predictions where the geodesic distance to the ground truth falls within specified thresholds (1km, 25km, 200km, 750km, and 2500km).

Input / output format

Input: Query image (and optionally retrieved reference images/text for RAG prompts during inference).

Output: Predicted GPS coordinates (latitude, longitude).

Scoring recipe

def threshold_metric(predictions, ground_truths, thresholds=[1, 25, 200, 750, 2500]):
    results = {}
    for thresh_km in thresholds:
        correct = 0
        for pred, gt in zip(predictions, ground_truths):
            dist_km = geodesic_distance_meters(pred, gt) / 1000.0
            if dist_km <= thresh_km:
                correct += 1
        results[f'acc@{thresh_km}km'] = (correct / len(predictions)) * 100
    return results

Common pitfalls

• Geodesic distance must be calculated using the specified Earth radius (6378137.0 m) to match the paper's results.
• The thresholds are evaluated independently, not cumulatively; accuracy at 25km is not derived from the 1km metric.
• Inference uses GPT-4V with temperature 1.2, introducing stochasticity; results should be averaged over multiple runs or fixed seeds for reproducibility.

Evidence (verbatim from paper)

Following previous work[cepeda2023geoclip] ; [zhou2024img2loc], we evaluate G3 with public datasets (IM2GPS3k[hays2008im2gps]* and YFCC4K*[thomee2016yfcc100m]*) and a threshold metric. Given the predicted coordinates and the ground truths, this metric quantifies the percentage of predictions where the distance to the ground truth falls within specified thresholds (1km, 25km, 200km, 750km, and 2500km).

Citation

@misc{jia2024g3,
  title={G3: An Effective and Adaptive Framework for Worldwide Geolocalization Using Large Multi-Modality Models},
  author={Pengyue Jia et al.},
  year={2024},
  note={arXiv:2405.14702}
}

• arXiv: 2405.14702