Disenpoi Ctr Eval

disenpoi-ctr-eval

DisenPOI: Disentangling Sequential and Geographical Influence for Point-of-Interest Recommendation — Qin et al. (2022) (arXiv:2210.16591, 2022)

What this evaluates

Evaluates a model's ability to predict user check-in behavior (CTR) in location-based recommendation by disentangling sequential and geographical influences. It probes how well the model handles data sparsity and cold-start scenarios using real-world POI interaction logs.

Datasets

• Foursquare Tokyo — total 573703; splits: train (-1), val (-1), test (-1)
• Foursquare New York — total 227428; splits: train (-1), val (-1), test (-1)
• Meituan — total 470095; splits: train (-1), val (-1), test (-1)

Metrics

• AUC (primary) — range: [0, 1]
  • Area under the Receiver Operating Characteristic curve. Measures the probability that a randomly chosen positive interaction is ranked higher than a negative one.
• Logloss — range: [0, ∞)
  • Binary cross-entropy loss. Measures the performance of a classification model where the prediction is a probability value between 0 and 1. Lower values indicate better calibration.

Input / output format

Input: User's chronological check-in sequence (positive samples) and randomly sampled unvisited POIs (negative samples).

Output: Predicted probability score for each user-POI pair, used for CTR ranking.

Scoring recipe

def compute_auc(y_true, y_pred):
    # y_true: 1 for positive, 0 for negative
    # y_pred: predicted probabilities
    from sklearn.metrics import roc_auc_score
    return roc_auc_score(y_true, y_pred)

def compute_logloss(y_true, y_pred):
    from sklearn.metrics import log_loss
    return log_loss(y_true, y_pred)

Common pitfalls

• Uses chronological split with last interaction per user for test, not random split.
• Negative sampling is random unvisited POIs, not hard negatives or popularity-based.
• Cold-start evaluation divides sequences into 5 folds (20% to 100%), which differs from standard train/val/test.

Evidence (verbatim from paper)

We sort the recorded user interactions in each dataset in chronological order. All visited POIs and the corresponding previous visiting subsequences make up the positive samples. For each positive sample, we randomly select one of the user’s unvisited POIs as the negative sample. The last interaction of each user is reserved for evaluation, while the remaining part is used for training. The evaluation set is then randomly split into two equal-sized subsets as test and validation set respectively. Since we have constructed training and evaluation set with positive and negative samples, we adopt AUC and Logloss as evaluation metrics, which is a common practice in CTR prediction

Citation

@misc{qin2022disenpoi,
  title={DisenPOI: Disentangling Sequential and Geographical Influence for Point-of-Interest Recommendation},
  author={Qin et al. (2022)},
  year={2022},
  note={arXiv:2210.16591}
}

• arXiv: 2210.16591