qhjqhj00 neural-news-detection-eval
Crafting Tomorrow's Headlines: Neural News Generation and Detection in English, Turkish, Hungarian, and Persian — Cem Üyük et al. (arXiv:2408.10724, 2024)
What this evaluates
Evaluates the ability of classifiers and LLMs to detect machine-generated news headlines across four languages. It probes cross-lingual generalization, robustness to zero-shot vs fine-tuned generators, and the effectiveness of linguistic vs transformer-based features for authenticity verification.
Datasets
- Multilingual Neural News Detection Benchmark — total ?; splits: in-domain test (-1), out-of-domain test (-1)
Metrics
F1 score(primary) — range: [0, 1]- Harmonic mean of precision and recall: 2 * (precision * recall) / (precision + recall).
Input / output format
Input: A single news headline text in English, Turkish, Hungarian, or Persian.
Output: Binary classification label indicating whether the headline is human-written or machine-generated (LLM-generated).
Scoring recipe
def calculate_f1(predictions, gold):
tp = sum(1 for p, g in zip(predictions, gold) if p == 1 and g == 1)
fp = sum(1 for p, g in zip(predictions, gold) if p == 1 and g == 0)
fn = sum(1 for p, g in zip(predictions, gold) if p == 0 and g == 1)
precision = tp / (tp + fp) if (tp + fp) > 0 else 0
recall = tp / (tp + fn) if (tp + fn) > 0 else 0
return 2 * (precision * recall) / (precision + recall) if (precision + recall) > 0 else 0
Common pitfalls
- Performance is heavily language-dependent; Persian shows artificially high scores likely due to shorter text lengths.
- Models fine-tuned on in-domain generators often fail to generalize to out-of-domain zero-shot generators.
- LLMs struggle to detect their own generated texts despite high performance on other models.
Evidence (verbatim from paper)
Notably, for GPT-4 in English, Random Forest once again achieves the highest F1 score.
Citation
@misc{uyuk2024headlines,
title={Crafting Tomorrow's Headlines: Neural News Generation and Detection in English, Turkish, Hungarian, and Persian},
author={Cem Üyük et al.},
year={2024},
note={arXiv:2408.10724}
}
- arXiv: 2408.10724