Longlamp Eval

longlamp-eval

LongLaMP: A Benchmark for Personalized Long-form Text Generation — Kumar et al. (2024) (arXiv:2407.11016, 2024)

What this evaluates

Evaluates a model's ability to generate personalized long-form text by integrating retrieved user profiles into a retrieval-augmented generation framework. It probes how well models can adapt their writing style and content to specific user attributes across different domains like emails, abstracts, reviews, and topic-based writing.

Datasets

• LongLaMP — total ?; splits: val (-1), test (-1)

Metrics

• ROUGE-1 — range: [0, 1]
  • Unigram overlap between generated and reference text.
• ROUGE-L — range: [0, 1]
  • Longest common subsequence overlap between generated and reference text.
• METEOR (primary) — range: [0, 1]
  • Precision, recall, and fragmentation penalty based on aligned unigrams.

Input / output format

Input: Task prompt + retrieved user profile snippets (k items) from a retriever (BM25 or Contriever).

Output: Generated long-form text corresponding to the task (e.g., email, abstract, review, or topic writing).

Scoring recipe

import rouge
import nltk
nltk.download('punkt')

def compute_metrics(preds, refs):
    rouge = rouge.Rouge()
    meteor = nltk.translate.meteor_score.single_meteor_score
    scores = {'ROUGE-1': [], 'ROUGE-L': [], 'METEOR': []}
    for p, r in zip(preds, refs):
        r1 = rouge.get_scores(p, r)[0]['rouge-1']['f']
        rL = rouge.get_scores(p, r)[0]['rouge-l']['f']
        m = meteor([r.split()], p.split())
        scores['ROUGE-1'].append(r1)
        scores['ROUGE-L'].append(rL)
        scores['METEOR'].append(m)
    return {k: sum(v)/len(v) for k, v in scores.items()}

Common pitfalls

• Retriever choice (BM25 vs Contriever) and k value significantly impact scores; optimal k varies by domain and setting.
• Temporal vs User settings require different train/test splits based on profile recency or user identity, which must be strictly separated to avoid data leakage.
• GPT-3.5 was not evaluated on the Email domain due to dataset privacy restrictions.

Evidence (verbatim from paper)

For zero-shot experiments, GPT-3.5 and LLaMA2 are utilized, and the evaluation metrics used are ROUGE-1, ROUGE-L, and METEOR. To evaluate the results, we assessed the generated output corresponding to each input against the expected output, as described in Section [3].

Citation

@misc{kumar2024longlamp,
  title={LongLaMP: A Benchmark for Personalized Long-form Text Generation},
  author={Kumar et al. (2024)},
  year={2024},
  note={arXiv:2407.11016}
}

• arXiv: 2407.11016