Mention Detection Eval

mention-detection-eval

What did you Mention? A Large Scale Mention Detection Benchmark for Spoken and Written Text — Mass et al. (2018) (arXiv:1801.07507, 2018)

What this evaluates

Evaluates a model's ability to detect all types of entity mentions (named and general) in both clean written text and noisy spoken/transcribed speech. It specifically probes handling of ambiguous, nested, and context-dependent terms across different data modalities.

Datasets

• Wikipedia — total 1000; splits: dev (500), test (500)
• Transcribed Speech — total 1000; splits: dev (500), test (500)
• ASR Output — total 1000; splits: dev (500), test (500)

Metrics

• Precision — range: [0, 1]
  • The proportion of detected mentions that are correct: TP / (TP + FP).
• Recall — range: [0, 1]
  • The proportion of gold mentions that are correctly detected: TP / (TP + FN).
• F1-measure (primary) — range: [0, 1]
  • The harmonic mean of Precision and Recall: 2 * (Precision * Recall) / (Precision + Recall).

Input / output format

Input: A single sentence (clean text, transcribed speech, or ASR-generated text).

Output: A list of detected mention spans (phrases) within the input sentence.

Scoring recipe

def compute_metrics(pred_mentions, gold_mentions):
    tp = len(set(pred_mentions) & set(gold_mentions))
    fp = len(set(pred_mentions) - set(gold_mentions))
    fn = len(set(gold_mentions) - set(pred_mentions))
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0.0
    return precision, recall, f1

Common pitfalls

• Handling nested mentions: The benchmark guidelines explicitly address nesting, and baselines must be configured to return only the longest phrases to avoid penalizing valid nested structures.
• Modality noise impact: Performance drops significantly on ASR data compared to clean text and transcribed speech due to automatic speech recognition errors, which is expected but often overlooked.
• Split naming inconsistency: The text initially describes the 500-sentence halves as 'training and testing' but subsequently labels them 'dev' and 'test' in the evaluation table.

Evidence (verbatim from paper)

Table 1 shows the results of a state-of-the-art system TagMe (Ferragina and Scaiella, 2012) on the three test datasets. ... We can see that the precision and recall on Wiki are higher than on the two spoken datasets Trans and ASR. For example the recall on Wiki is 0.523 compared with 0.436 on Trans and 0.421 on ASR.

Citation

@misc{mass2018mention,
  title={What did you Mention? A Large Scale Mention Detection Benchmark for Spoken and Written Text},
  author={Mass et al. (2018)},
  year={2018},
  note={arXiv:1801.07507}
}

• arXiv: 1801.07507