Xmind Crosslingual Eval

xmind-crosslingual-eval

MIND Your Language: A Multilingual Dataset for Cross-lingual News Recommendation — Iana et al. (2024) (arXiv:2403.17876, 2024)

What this evaluates

This benchmark evaluates the cross-lingual transfer capability of neural news recommenders. It probes how well models trained monolingually on English news can generate accurate recommendations in 14 other languages under zero-shot and few-shot settings, with and without bilingual user consumption patterns.

Datasets

• xMIND — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/andreeaiana/xMIND

Metrics

• AUC (primary) — range: [0, 1]
  • Area Under the Receiver Operating Characteristic curve, measuring the probability that a randomly chosen positive instance is ranked higher than a randomly chosen negative instance.
• MRR — range: [0, 1]
  • Mean Reciprocal Rank, calculated as the average of the reciprocal of the rank of the first relevant item in the recommended list.
• nDCG@5 — range: [0, 1]
  • Normalized Discounted Cumulative Gain at cutoff 5, measuring ranking quality by discounting gains logarithmically based on position.
• nDCG@10 — range: [0, 1]
  • Normalized Discounted Cumulative Gain at cutoff 10, measuring ranking quality by discounting gains logarithmically based on position.

Input / output format

Input: User click history (up to 50 previously clicked news items with text, categories, and named entities) and a set of candidate news items (text, categories, named entities) to be scored.

Output: A ranked list of candidate news items, or a scalar recommendation score per candidate computed as the dot product between the user history representation and the candidate news representation.

Scoring recipe

import math

def compute_metrics(predictions, gold):
    # predictions: list of (score, item_id) sorted descending
    # gold: set of relevant item_ids
    pos = [s for s, _ in predictions if _ in gold]
    neg = [s for s, _ in predictions if _ not in gold]
    auc = sum(1 for p in pos for n in neg if p > n) / (len(pos) * len(neg)) if pos and neg else 0.0
    mrr = 0.0
    for rank, (_, item_id) in enumerate(predictions, 1):
        if item_id in gold:
            mrr = 1.0 / rank
            break
    k = 10
    dcg = sum(1.0 / math.log2(r + 1) for r, (_, item_id) in enumerate(predictions[:k], 1) if item_id in gold)
    idcg = sum(1.0 / math.log2(r + 1) for r in range(1, min(len(gold), k) + 1))
    ndcg = dcg / idcg if idcg > 0 else 0.0
    return {'AUC': auc, 'MRR': mrr, 'nDCG@10': ndcg}

Common pitfalls

• The test set uses the validation portion of MIND because official test labels were not released, which may not reflect true out-of-distribution performance.
• Hyperparameter tuning is performed exclusively on the English (MIND) dataset, potentially introducing distribution bias that does not generalize to target languages.
• Bilingual consumption is simulated by randomly replacing English news with translated versions, which may not capture real-world bilingual user behavior or language mixing patterns.

Evidence (verbatim from paper)

We repeat each experiment three times, with different random seeds, and report averages and standard deviations for the standard metrics: AUC, MRR, nDCG@5, and nDCG@10.

Citation

@misc{iana2024mindyourlanguage,
  title={MIND Your Language: A Multilingual Dataset for Cross-lingual News Recommendation},
  author={Iana et al. (2024)},
  year={2024},
  note={arXiv:2403.17876}
}

• arXiv: 2403.17876