qhjqhj00 crossnews-ua-eval
CrossNews-UA: A Cross-lingual News Semantic Similarity Benchmark for Ukrainian, Polish, Russian, and English — Dementieva et al. (2025) (arXiv:2510.19628, 2025)
What this evaluates
Evaluates cross-lingual semantic similarity between news article pairs across four dimensions (Who, What, Where, When) in Ukrainian, Polish, Russian, and English. It probes a model's ability to align event-level information across languages while ignoring publication dates.
Datasets
- CrossNews-UA — total ?; splits: test (-1); repo https://github.com/TUM-NLP/crossnews-ua
Metrics
macro-averaged F1-score(primary) — range: [0, 1]- Harmonic mean of precision and recall computed per similarity label, then averaged macroscopically across the four 4W dimensions (Who, What, Where, When) and language pairs.
Input / output format
Input: Pairs of news article texts in two different languages. Publication dates are explicitly excluded from the input.
Output: A similarity label for each of the four dimensions: Who, What, Where, When.
Scoring recipe
def compute_macro_f1(predictions, gold):
dims = ["Who", "What", "Where", "When"]
f1_scores = []
for dim in dims:
pred_labels = [p[dim] for p in predictions]
gold_labels = [g[dim] for g in gold]
f1_scores.append(f1_score(gold_labels, pred_labels, average="binary"))
return sum(f1_scores) / len(f1_scores)
Common pitfalls
- Models must not use publication dates as a proxy for temporal similarity, as annotators were explicitly instructed to exclude them.
- Performance varies significantly across the 4W dimensions, with 'When' consistently being the hardest, so reporting a single aggregate without dimension breakdown can be misleading.
Evidence (verbatim from paper)
Табл. 3: The detailed results for e5-large embeddings baseline per each language pair and dimension (macro-averaged F1-score).
Citation
@misc{dementieva2025crossnewsua,
title={CrossNews-UA: A Cross-lingual News Semantic Similarity Benchmark for Ukrainian, Polish, Russian, and English},
author={Dementieva et al. (2025)},
year={2025},
note={arXiv:2510.19628}
}
- arXiv: 2510.19628