Character Level F1

character-level-f1

From Jack of All Trades to Master of One: Specializing LLM-based Autoraters to a Test Set — Finkelstein et al. (2024) (arXiv:2411.15387, 2024)

What this evaluates

This evaluation probes an LLM-based autorater's ability to predict fine-grained machine translation errors (spans, severities, categories) without using human references. It specifically tests how well the model can specialize to a given test set by leveraging in-context examples of human ratings from other systems on the same inputs.

Datasets

• WMT'23 and WMT'24 MQM datasets — total ?; splits: test (-1)

Metrics

• character-level F1 (primary) — range: [0, 1]
  • Standard character-level precision, recall, and F1 for error span prediction. Partial credit of 0.5 is awarded if a character is correctly identified as an error but the predicted severity is incorrect.
• Acc23 — range: [0, 1]
  • Segment-level pairwise accuracy with tie calibration, rewarding correct ranking of translations and correct tie prediction.

Input / output format

Input: Source sentence, hypothesis translation, and a set of in-context learning examples containing other translations from the same test set paired with their human MQM ratings (error spans, severity, category). No human reference is provided.

Output: JSON object containing a list of predicted errors, where each error includes "span" (character indices), "severity", and "category" fields.

Scoring recipe

def compute_char_f1(gold_spans, pred_spans):
    gold_chars = set()
    for s in gold_spans: gold_chars.update(range(s.start, s.end))
    pred_chars = set()
    for s in pred_spans: pred_chars.update(range(s.start, s.end))
    tp = len(gold_chars & pred_chars)
    fp = len(pred_chars - gold_chars)
    fn = len(gold_chars - pred_chars)
    sev_mismatch = sum(0.5 * (s.end - s.start) for g, p in zip(gold_spans, pred_spans) if g.severity != p.severity and g.start == p.start and g.end == p.end)
    prec = (tp + sev_mismatch) / (tp + fp + sev_mismatch) if (tp + fp + sev_mismatch) > 0 else 0
    rec = (tp + sev_mismatch) / (tp + fn + sev_mismatch) if (tp + fn + sev_mismatch) > 0 else 0
    return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0

Common pitfalls

• Including the human reference in the input violates the reference-free (QE) constraint of the metric.
• Failing to use hold-one-out prompting when constructing ICL examples causes data leakage by including the target system's own ratings.
• Shuffling ICL examples across different source sentences or raters breaks the pseudo-SxS constraint, significantly degrading performance.

Evidence (verbatim from paper)

To meta-evaluate the quality of Specialist AutoMQM, we compute the character-level precision, recall, and F1 span tagging evaluation metrics (used by the WMT’23 QE Shared Task; Blain et al. (2023)). Given gold and predicted ratings, these metrics represent the precision, recall, and F1 of predicting whether a character in the hypothesis translation is included in an error span or not. Partial credit of 0.5 is given if the predicted rating marks a character as an error but predicts the incorrect severity.

Citation

@misc{finkelstein2024specializing,
  title={From Jack of All Trades to Master of One: Specializing LLM-based Autoraters to a Test Set},
  author={Finkelstein et al. (2024)},
  year={2024},
  note={arXiv:2411.15387}
}

• arXiv: 2411.15387