Finedialfact Eval

finedialfact-eval

FineDialFact: A benchmark for Fine-grained Dialogue Fact Verification — Chen et al. (2025) (arXiv:2508.05782, 2025)

What this evaluates

Evaluates a model's ability to perform fine-grained fact verification on dialogue responses by checking individual atomic facts against external knowledge. It probes whether models can correctly classify facts as supporting, refuting, or lacking sufficient information, particularly in the presence of hallucinations and imbalanced label distributions.

Datasets

• FineDialFact — total 1000; splits: test (1000); repo https://github.com/XiangyanChen/FineDialFact

Metrics

• F1-score (primary) — range: [0, 1]
  • Harmonic mean of precision and recall across all classes. Calculated as 2 * (precision * recall) / (precision + recall).
• Geometric Mean (G-Mean) — range: [0, 1]
  • Geometric mean of recall scores computed for each class individually. Designed to handle imbalanced datasets by penalizing poor performance on minority classes.
• Accuracy — range: [0, 1]
  • Ratio of correctly classified instances to the total number of instances.

Input / output format

Input: Dialogue history, retrieved external knowledge (Wikipedia passages), and the atomic fact(s) extracted from the dialogue response to be verified.

Output: A single categorical label: 'Supports', 'Refutes', or 'Not Enough Information'.

Scoring recipe

def compute_metrics(predictions, gold_labels):
    classes = ['Supports', 'Refutes', 'Not Enough Information']
    precision, recall, f1, _ = precision_recall_fscore_support(gold_labels, predictions, average='macro')
    recalls_per_class = [recall_score(gold_labels, predictions, pos_label=c) for c in classes]
    g_mean = np.prod(recalls_per_class) ** (1/len(classes))
    accuracy = np.mean(np.array(predictions) == np.array(gold_labels))
    return {'accuracy': accuracy, 'precision': precision, 'recall': recall, 'f1': f1, 'g_mean': g_mean}

Common pitfalls

• Accuracy is explicitly noted as misleading due to severe class imbalance (522 'Not Enough Info' vs 97 'Refutes'); relying solely on accuracy will overstate performance.
• Evaluation assumes high-quality atomic fact splitting; poor splitting degrades both annotation reliability and model evaluation scores.
• Models are expected to use external knowledge retrieval rather than parametric memory; evaluating without retrieval violates the benchmark's premise.

Evidence (verbatim from paper)

We use classification metrics to validate the performance of dialogue fact verification, including accuracy, precision, recall and F1-score. Accuracy reflects the overall performance of a classifier, but it may be misleading when dealing with imbalanced data. The F1-score and Geometric Mean (G-Mean) can more realistically reflect performance for imbalanced data.

Citation

@misc{chen2025finedialfact,
  title={FineDialFact: A benchmark for Fine-grained Dialogue Fact Verification},
  author={Chen et al. (2025)},
  year={2025},
  note={arXiv:2508.05782}
}

• arXiv: 2508.05782