Zs Xlt News Rec Eval

zs-xlt-news-rec-eval

News Without Borders: Domain Adaptation of Multilingual Sentence Embeddings for Cross-lingual News Recommendation — Iana et al. (2024) (arXiv:2406.12634, 2024)

What this evaluates

Evaluates zero-shot cross-lingual news recommendation by measuring how effectively a model recommends articles in a target language to users who only consume news in a source language. It probes the model's ability to leverage multilingual sentence embeddings and click behavior fusion without task-specific fine-tuning on the target language.

Datasets

• MIND (small) / xMIND (small) — total ?; splits: train (124229), val (29498), test (70938)

Metrics

• AUC — range: [0, 1]
  • Area Under the Receiver Operating Characteristic Curve. Measures the probability that a randomly chosen positive instance is ranked higher than a randomly chosen negative instance.
• MRR — range: [0, 1]
  • Mean Reciprocal Rank. For each query, takes the reciprocal of the rank of the first relevant item, then averages across all queries.
• nDCG@10 (primary) — range: [0, 1]
  • Normalized Discounted Cumulative Gain at cutoff 10. Computes the weighted sum of relevance scores at each position up to 10, normalized by the ideal DCG@10 for the same set of items.

Input / output format

Input: News article text (title and abstract), category information, named entities (for specific baselines), and a user's click history (capped at 50 items).

Output: A ranked list of news articles recommended to the user.

Scoring recipe

def evaluate(predictions, relevant_items):
    # predictions: list of recommended item IDs
    # relevant_items: set of ground-truth clicked items
    
    # AUC (Wilcoxon-Mann-Whitney)
    pos = len(relevant_items)
    neg = len(predictions) - pos
    auc = (sum(1 for r in predictions if r in relevant_items) - pos*(pos+1)/2) / (pos*neg) if pos > 0 and neg > 0 else 0.0
    
    # MRR
    rr = 0.0
    for rank, item in enumerate(predictions):
        if item in relevant_items:
            rr = 1.0 / (rank + 1)
            break
            
    # nDCG@10
    dcg = sum((1.0 if item in relevant_items else 0.0) / math.log2(rank + 2) for rank, item in enumerate(predictions[:10]))
    idcg = sum(1.0 / math.log2(i + 2) for i in range(min(pos, 10)))
    ndcg = dcg / idcg if idcg > 0 else 0.0
    
    return auc, rr, ndcg

Common pitfalls

• MIND test labels are not publicly released; the authors use the validation set as the test set due to this constraint.
• xMIND statistics in Table 1 are reported per language (14 languages total), not as aggregate counts across all languages.
• Models use different user modeling strategies: some employ parameterized attention/GRU networks, while others use non-parameterized late fusion of click behaviors, affecting direct comparison.

Evidence (verbatim from paper)

We repeat each experiment three times with the seeds {42,43,44}, set with PyTorch Lightning’s seed_everything, and report the mean and standard deviations for common metrics: AUC, MRR, and nDCG@10. ... Wu et al. [[60]] do not release test labels for MIND. Hence, we use the validation set for testing, and split the training set into temporarily disjoint portions for training (first four days) and validation (last day), as per Table [1].

Citation

@misc{iana2024news,
  title={News Without Borders: Domain Adaptation of Multilingual Sentence Embeddings for Cross-lingual News Recommendation},
  author={Iana et al. (2024)},
  year={2024},
  note={arXiv:2406.12634}
}

• arXiv: 2406.12634