Psiloqa Eval

psiloqa-eval

When Models Lie, We Learn: Multilingual Span-Level Hallucination Detection with PsiloQA — Rykov et al. (2025) (arXiv:2510.04849, 2025)

What this evaluates

Evaluates the ability of models to detect span-level hallucinations in multilingual question-answering contexts. It probes cross-lingual generalization and token-level inconsistency detection between generated answers and ground truth.

Datasets

• PsiloQA — total ?; splits: train (-1), test (-1); repo https://github.com/s-nlp/psiloqa

Metrics

• IoU (primary) — range: percent
  • Intersection over Union at the character level. Calculated as the size of the intersection divided by the size of the union between the set of binarized character-level gold annotations and the set of characters predicted as hallucinated.
• AP — range: percent
  • Average Precision for ranking-based character-level evaluation. Computed as the area under the precision-recall curve: sum of precision at each cut-off multiplied by the change in recall between consecutive items.

Input / output format

Input: Context-question-answer triples processed for token-level classification.

Output: Binary token-level labels indicating whether each character is hallucinated, or continuous confidence scores for ranking.

Scoring recipe

def compute_iou(pred_chars, gold_chars):
    intersection = len(pred_chars & gold_chars)
    union = len(pred_chars | gold_chars)
    return intersection / union if union > 0 else 0.0

def compute_ap(scores, gold_mask):
    ranks = sorted(range(len(scores)), key=lambda i: scores[i], reverse=True)
    tp, fp = 0, 0
    precisions, recalls = [], []
    for i in ranks:
        if gold_mask[i]: tp += 1
        else: fp += 1
        precisions.append(tp / (tp + fp))
        recalls.append(tp / sum(gold_mask))
    return sum(p * (recalls[i] - recalls[i-1]) for i, p in enumerate(precisions))

Common pitfalls

• IoU requires language-specific threshold calibration on the validation set rather than a global threshold.
• AP measures ranking quality at the character level, not span-level accuracy; confusing the two granularities leads to misinterpretation.
• LLM-based baselines like FActScore must be adapted from sentence-level to token-level by computing scores based on unsupported claim frequency.

Evidence (verbatim from paper)

Due to the complex nature of hallucination detection, we employ a dual-level evaluation approach combining span-level and character-level assessment. As with Mu-SHROOM, we selected the intersection over union (IoU) metric to evaluate span-level hallucination detection. Additionally, we use average precision (AP) for ranking-based evaluation on character-level.

Citation

@misc{rykov2025psiloqa,
  title={When Models Lie, We Learn: Multilingual Span-Level Hallucination Detection with PsiloQA},
  author={Rykov et al. (2025)},
  year={2025},
  note={arXiv:2510.04849}
}

• arXiv: 2510.04849