Finerumfact Eval

finerumfact-eval

Learning to Verify Summary Facts with Fine-Grained LLM Feedback — Oh et al. (2024) (arXiv:2412.10689, 2024)

What this evaluates

Evaluates a model's ability to perform sentence-level fact verification on generated summaries and localize specific factuality error types. It measures how well the model's judgments align with human annotations across sentence, summary, and system levels.

Datasets

• FineSumFact — total 26353; splits: train (25660), test (693); repo https://github.com/DISL-Lab/FineSumFact

Metrics

• balanced accuracy (bAcc) (primary) — range: [0, 1]
  • Standard balanced accuracy for binary classification: (TP/(TP+FN) + TN/(TN+FP)) / 2. Measures sentence-level verification accuracy.
• summary-level correlation (Pearson) — range: [-1, 1]
  • Pearson correlation coefficient between the model's predicted summary-level factuality scores and human-provided summary-level scores.
• system-level correlation (Rank/Spearman) — range: [-1, 1]
  • Spearman rank correlation between the model's ranking of summarizers and the human ranking of summarizers.
• error localization accuracy — range: [0, 1]
  • Accuracy of predicting the correct factuality error category among seven predefined types for misclassified sentences.

Input / output format

Input: A generated summary and its corresponding reference text.

Output: Per sentence: a binary label (factually correct/incorrect), optionally accompanied by reasoning and an error type/category.

Scoring recipe

def compute_metrics(preds, gold):
    # Binary sentence-level
    tp = sum(1 for p, g in zip(preds['sent'], gold['sent']) if p == 1 and g == 1)
    fp = sum(1 for p, g in zip(preds['sent'], gold['sent']) if p == 1 and g == 0)
    fn = sum(1 for p, g in zip(preds['sent'], gold['sent']) if p == 0 and g == 1)
    tn = sum(1 for p, g in zip(preds['sent'], gold['sent']) if p == 0 and g == 0)
    bAcc = (tp / (tp + fn) + tn / (tn + fp)) / 2

    # Correlations
    pearson_r = pearsonr(preds['summ_score'], gold['summ_score'])
    spearman_r = spearmanr(preds['sys_rank'], gold['sys_rank'])

    # Error localization (only on incorrect sentences)
    err_preds = [p for p, g in zip(preds['err_type'], gold['err_type']) if g == 0]
    err_gold = [g for p, g in zip(preds['err_type'], gold['err_type']) if g == 0]
    loc_acc = sum(1 for p, g in zip(err_preds, err_gold) if p == g) / len(err_gold)

    return {'bAcc': bAcc, 'pearson': pearson_r, 'spearman': spearman_r, 'loc_acc': loc_acc}

Common pitfalls

• Confusing sentence-level binary accuracy with summary-level or system-level correlation metrics, which require different aggregation strategies.
• Assuming human-annotated fine-grained error types are more reliable than LLM-generated ones; the paper notes human inter-annotator Kappa is often < 0.5 for fine-grained tasks.
• Evaluating error localization on correctly classified sentences; it should only be computed on sentences where the binary judgment is incorrect.

Evidence (verbatim from paper)

Metrics. We follow the widely used metrics in recent worksSong et al. ([2024]); Liu et al. ([2023]), verifying the agreement with human in three different levels: balanced accuracy (bAcc), an indicator of sentence-level verification accuracy; summary-level correlation, an indicator of agreement with humans’ summary-level scores; system-level correlation, an indicator of agreement with humans’ ranking across different summarizers.

Citation

@misc{oh2024learnstoverify,
  title={Learning to Verify Summary Facts with Fine-Grained LLM Feedback},
  author={Oh et al. (2024)},
  year={2024},
  note={arXiv:2412.10689}
}

• arXiv: 2412.10689