Spatio Textual Retrieval Eval

spatio-textual-retrieval-eval

LIST: Learning to Index Spatio-Textual Data for Embedding based Spatial Keyword Queries — Yin et al. (2024) (arXiv:2403.07331, 2024)

What this evaluates

Evaluates the effectiveness of embedding-based spatial keyword retrieval models by measuring how well they rank relevant Points of Interest (POIs) based on combined location and textual query signals. It probes the model's ability to handle spatio-textual relevance without manual weighting of spatial and textual factors.

Datasets

• Beijing — total ?; splits: train (-1), val (-1), test (-1)
• Shanghai — total ?; splits: train (-1), val (-1), test (-1)
• Geo-Glue — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• Recall@k, NDCG@k (primary) — range: [0, 1]
  • Recall@k evaluates the proportion of positive objects contained in the top-k candidates for a given query. NDCG@k considers the order of ground-truth objects in the retrieved list, using graded relevance rel_i in {0, 1} where rel_i=1 if relevant, else 0.

Input / output format

Input: Query consisting of a location and a set of keywords, paired with candidate Points of Interest (POIs) having locations and textual attributes.

Output: A ranked list of candidate POIs (or relevance scores for reranking).

Scoring recipe

def recall_at_k(retrieved, gold, k):
    top_k = retrieved[:k]
    return len(set(top_k) & set(gold)) / len(gold)

def ndcg_at_k(retrieved, gold, k):
    dcg = sum(1.0 / math.log2(i + 2) for i, obj in enumerate(retrieved[:k]) if obj in gold)
    idcg = sum(1.0 / math.log2(i + 2) for i in range(min(len(gold), k)))
    return dcg / idcg if idcg > 0 else 0.0

Common pitfalls

• Ground truth relies on click-through logs rather than explicit ratings, which may introduce selection bias.
• The Geo-Glue dataset has modified coordinates for privacy, causing many objects to share identical geo-locations.
• Some baseline models (e.g., PALM, DrW) are too slow for brute-force evaluation on Geo-Glue, requiring results from original papers instead of direct reproduction.

Evidence (verbatim from paper)

Following previous studies [9, 33], we use two metrics, i.e., the Recall and Normalized Discounted Cumulative Gain (NDCG), to evaluate the effectiveness. Recall@k evaluates the proportion of positive objects contained in the top-k candidates for a given query. NDCG@k considers the order of ground-truth objects in the retrieved objects, reflecting the quality of the ranking in the retrieved list. We assign the graded relevance of the result at position i as $rel_{i} \in {0, 1}$, where $rel_{i} = 1$ when the object is relevant to the query, otherwise $rel_{i}=0$.

Citation

@misc{yin2024list,
  title={LIST: Learning to Index Spatio-Textual Data for Embedding based Spatial Keyword Queries},
  author={Yin et al. (2024)},
  year={2024},
  note={arXiv:2403.07331}
}

• arXiv: 2403.07331