Much Eval

much-eval

MUCH: A Multilingual Claim Hallucination Benchmark — Dentan et al. (2025) (arXiv:2511.17081, 2025)

What this evaluates

Evaluates the ability of logit-based uncertainty quantification (UQ) methods to predict claim-level hallucination (factuality) in multilingual LLM outputs. It measures how well token-level confidence scores, when aggregated, correlate with ground-truth factuality labels across different languages and model configurations.

Datasets

• MUCH — total 20751; splits: test (-1); repo https://github.com/orailix/much

Metrics

• ROC-AUC (primary) — range: [0, 1]
  • Area under the Receiver Operating Characteristic curve, measuring the trade-off between true positive rate and false positive rate across all classification thresholds.
• PR-AUC — range: [0, 1]
  • Area under the Precision-Recall curve, measuring the trade-off between precision and recall across all thresholds, particularly sensitive to class imbalance.

Input / output format

Input: Text segments (claims) produced by a deterministic segmenter, along with per-token logits from the LLM and ground-truth factuality labels.

Output: Claim-level uncertainty score (continuous) derived by aggregating token-level UQ scores using a specified aggregator (mean, max, geometric mean, or product).

Scoring recipe

def compute_metrics(predictions, gold):
    # predictions: list of aggregated claim-level UQ scores
    # gold: list of binary factuality labels (1=hallucinated, 0=factual)
    roc_auc = roc_auc_score(gold, predictions)
    pr_auc = average_precision_score(gold, predictions)
    return {"ROC-AUC": roc_auc, "PR-AUC": pr_auc}

Common pitfalls

• Aggregator choice drastically changes results; the product aggregator consistently outperforms mean/max/geometric mean.
• Baselines like SAR and CCP rely on NLI models optimized only for English, causing severe performance drops in Spanish and other languages.
• UQ computation time can exceed generation time (e.g., CCP takes ~197% of generation time), making real-time monitoring infeasible without optimization.

Evidence (verbatim from paper)

Following*(Fadeeva et al., [2024]), we evaluated four aggregators: the (arithmetic) mean of token values in the claim, the maximum, the geometric mean, and the product. Consistent with the observations of(Fadeeva et al., [2024])*, our best results are obtained when using the product as aggregator. We report ROC-AUC and Precision–Recall (PR) AUC.

Citation

@misc{dentan2025much,
  title={MUCH: A Multilingual Claim Hallucination Benchmark},
  author={Dentan et al. (2025)},
  year={2025},
  note={arXiv:2511.17081}
}

• arXiv: 2511.17081