Email Subject Line Generation Eval

email-subject-line-generation-eval

This Email Could Save Your Life: Introducing the Task of Email Subject Line Generation — Zhang et al. (2019) (arXiv:1906.03497, 2019)

What this evaluates

Evaluates a model's ability to generate highly abstractive, ultra-concise email subject lines from email body text. It probes extreme compression, informativeness, and fluency in a real-world email triaging context, distinguishing the task from standard text summarization.

Datasets

• AESLC — total ?; splits: dev (-1), test (-1)

Metrics

• ROUGE-1 (primary) — range: percent
  • F1 score of unigram overlap between the generated subject line and the reference subject line(s). Computed as 2 * (precision * recall) / (precision + recall).
• ROUGE-2 — range: percent
  • F1 score of bigram overlap between the generated subject line and the reference subject line(s).
• ROUGE-L — range: percent
  • F1 score based on the longest common subsequence between the generated subject line and the reference subject line(s), capturing sentence-level structure.
• METEOR — range: percent
  • Metric for Evaluation of Translation with Explicit ORdering; aligns words using synonymy and stemming, penalizes for fragmentation and ordering differences.
• ESQE — range: other
  • Email Subject Quality Estimator; a reference-less neural metric specifically trained to score subject line quality without requiring ground-truth references.
• Human Rating — range: [1, 4]
  • Average score on a 1-4 scale (Poor, Fair, Good, Great) across three aspects: overall quality, informativeness, and fluency. Each sample is rated by 3 independent judges.

Input / output format

Input: Email body text (including greeting and main content) provided as the source document.

Output: A single sentence or short phrase representing the generated email subject line.

Scoring recipe

def compute_rouge_f1(pred, refs, n=1):
    pred_ngrams = set(ngrams(pred.split(), n))
    ref_ngrams = set()
    for r in refs:
        ref_ngrams.update(ngrams(r.split(), n))
    if not pred_ngrams or not ref_ngrams:
        return 0.0
    overlap = len(pred_ngrams & ref_ngrams)
    prec = overlap / len(pred_ngrams)
    rec = overlap / len(ref_ngrams)
    return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0

def compute_human_rating(predictions, judges=3):
    scores = []
    for pred in predictions:
        ratings = [judge.assign_score(pred) for _ in range(judges)]
        scores.append(sum(ratings) / len(ratings))
    return sum(scores) / len(scores)

Common pitfalls

• Evaluating against a single reference when the dataset actually provides multiple human annotations per email (Table 3b shows evaluation against two human annotations).
• Relying solely on n-gram overlap metrics (ROUGE/METEOR) without considering that email subject line generation requires extreme abstraction and compression, which these metrics may penalize unfairly.
• Ignoring the reference-less ESQE metric, which is specifically designed for this task and often correlates better with human judgment than standard summarization metrics.

Evidence (verbatim from paper)

We first use automatic metrics from text summarization and machine translation: (1) ROUGE (Lin, 2004) including F1 scores of ROUGE-1, ROUGE-2, and ROUGE-L. (2) METEOR (Denkowski and Lavie, 2014). They all rely on one or more references and measure the similarity between the output and the reference. In addition, we include ESQE, which is a reference-less metric.

Citation

@misc{zhang2019email,
  title={This Email Could Save Your Life: Introducing the Task of Email Subject Line Generation},
  author={Zhang et al. (2019)},
  year={2019},
  note={arXiv:1906.03497}
}

• arXiv: 1906.03497