Vista Score Eval

vista-score-eval

VISTA Score: Verification In Sequential Turn-based Assessment — Ashley Lewis et al. (2025) (arXiv:2510.27052, 2025)

What this evaluates

Evaluates conversational factuality and hallucination detection in LLMs by decomposing dialogue turns into atomic claims, verifying them against reference texts and dialogue history, and categorizing unverifiable content. It measures how well models track factual consistency across sequential turns.

Datasets

• FaithDial — total ?; splits: test (-1)

Metrics

• claim-level accuracy (primary) — range: percent
  • Proportion of atomic claims correctly classified as VERIFIED or UNVERIFIABLE (or fine-grained subcategories) against gold annotations.
• macro-F1 — range: percent
  • Unweighted mean of the F1 score calculated per class across all verification categories.
• turn-level accuracy — range: percent
  • Proportion of dialogue turns correctly classified as verifiable or unverifiable.

Input / output format

Input: Dialogue context (prior turns), current turn text, and retrieved reference document. The model must decompose the turn into atomic claims and assign a verification label.

Output: A classification label per atomic claim: VERIFIED, CONTRADICTED, LACKING_EVIDENCE, SUBJECTIVE, or ABSTAIN (merged to UNVERIFIABLE for baseline comparison).

Scoring recipe

def score(predictions, gold):
    acc = sum(p == g for p, g in zip(predictions, gold)) / len(gold)
    classes = set(gold)
    f1s = []
    for c in classes:
        tp = sum(1 for p, g in zip(predictions, gold) if p == c and g == c)
        fp = sum(1 for p, g in zip(predictions, gold) if p == c and g != c)
        fn = sum(1 for p, g in zip(predictions, gold) if p != c and g == c)
        prec = tp / (tp + fp) if (tp + fp) else 0
        rec = tp / (tp + fn) if (tp + fn) else 0
        f1s.append(2 * prec * rec / (prec + rec) if (prec + rec) else 0)
    return {'claim_accuracy': acc, 'macro_f1': sum(f1s) / len(f1s)}

Common pitfalls

• Prior benchmarks often misclassify unverifiable or out-of-scope content as hallucinations; VISTA explicitly separates these into UNVERIFIABLE subcategories.
• Ignoring dialogue history or background context during verification significantly drops accuracy, as models rely on sequential grounding to resolve ambiguities.

Evidence (verbatim from paper)

We report turn-level accuracy (verifiable vs. unverifiable), claim-level accuracy, and macro-F1. The majority baseline predicts the most frequent claim label (VERIFIED).

Citation

@misc{lewis2025vista,
  title={VISTA Score: Verification In Sequential Turn-based Assessment},
  author={Ashley Lewis et al. (2025)},
  year={2025},
  note={arXiv:2510.27052}
}

• arXiv: 2510.27052