Dt Pens Eval

dt-pens-eval

Improved Personalized Headline Generation via Denoising Fake Interests from Implicit Feedback — Liu et al. (2025) (arXiv:2508.07178, 2025)

What this evaluates

This benchmark evaluates a model's ability to generate personalized news headlines by accurately capturing user interests from implicit feedback (clicks and dwell times) while filtering out noise. It probes the system's capacity to align generated text with both lexical patterns and semantic meaning relative to ground-truth headlines tailored to specific user preferences.

Datasets

• DT-PENS — total 9823; splits: test (9823); repo https://github.com/liukejin-up/PHG-DIF

Metrics

• ROUGE-1 (primary) — range: percent
  • Measures unigram overlap between generated and reference headlines.
• ROUGE-2 — range: percent
  • Measures bigram overlap between generated and reference headlines.
• ROUGE-L — range: percent
  • Measures longest common subsequence overlap between generated and reference headlines.
• BLEURT — range: [0, 100]
  • Uses a BERT-based model to capture semantic similarity and provide robust quality judgments.
• BARTScore — range: [0, 100]
  • Assesses fluency, grammar, and alignment with input text by leveraging BART’s generation probabilities.

Input / output format

Input: User click history, dwell times, and candidate news article content/body.

Output: A single personalized news headline.

Scoring recipe

def compute_metrics(predictions, references):
    rouge1 = rouge_score(references, predictions, ngrams=1)
    rouge2 = rouge_score(references, predictions, ngrams=2)
    rougeL = rouge_score(references, predictions, ngrams='L')
    bleurt = bleurt_score(references, predictions, model='bleurt')
    bart = bart_score(references, predictions, model='bart')
    return {'ROUGE-1': rouge1, 'ROUGE-2': rouge2, 'ROUGE-L': rougeL, 'BLEURT': bleurt, 'BARTScore': bart}

Common pitfalls

• The dataset was constructed using LLMs and human filtering, so evaluators must not use the same LLMs for generation during testing to avoid data leakage or style overfitting.
• Dwell-time annotations are only available for the dataset construction phase; the evaluation itself relies on standard text generation metrics, so models cannot directly optimize against dwell time during inference.
• Baselines include both non-personalized (BART, T5) and personalized methods; comparing personalized models against non-personalized ones without accounting for the lack of user history will unfairly penalize the latter.

Evidence (verbatim from paper)

For lexical similarity between the generated and reference headlines, we employ ROUGE-n*(Lin, [2004]), which measures the overlap of n-grams and is widely used in text summarization evaluation. To evaluate the semantic quality of the generated headlines, we utilize two model-based evaluation methods: BLEURT(Sellam et al., [2020])* and BARTScore*(Yuan et al., [2021])*. BLEURT captures semantic similarity and provides robust quality judgments. BARTScore assesses fluency, grammar, and alignment with the input text by leveraging BART’s language understanding and generation probabilities.

Citation

@misc{liu2025improvedpersonalized,
  title={Improved Personalized Headline Generation via Denoising Fake Interests from Implicit Feedback},
  author={Liu et al. (2025)},
  year={2025},
  note={arXiv:2508.07178}
}

• arXiv: 2508.07178