Granular Change Accuracy

granular-change-accuracy

Granular Change Accuracy: A More Accurate Performance Metric for Dialogue State Tracking — Aksu et al. (2024) (arXiv:2403.11123, 2024)

What this evaluates

Evaluates Dialogue State Tracking (DST) models by measuring performance based on per-turn belief state changes rather than raw slot or turn-level accuracy. It aims to provide partial credit for partially correct predictions and reduce bias from error timing and distribution across dialogue turns.

Datasets

• MultiWOZ 2.1 — total ?; splits: test (-1)
• SGD — total ?; splits: test (-1)

Metrics

• Granular Change Accuracy (primary) — range: [0, 1]
  • Computes accuracy by comparing predicted versus gold belief state changes per turn. Assigns partial credit for partially correct slot-value updates and avoids double-counting errors when incorrect values persist across multiple turns. Aggregated as an average score over all turns.

Input / output format

Input: A sequence of system and user utterance pairs per turn, along with the previous turn's belief state.

Output: Current turn's belief state represented as a set of active slot-value pairs, with inactive slots assigned a 'none' value.

Scoring recipe

score = 0.0
for turn in dialogue:
    gold_change = compute_belief_state_change(gold_bs, prev_gold_bs)
    pred_change = compute_belief_state_change(pred_bs, prev_pred_bs)
    turn_score = partial_match_score(gold_change, pred_change)
    score += turn_score
final_metric = score / total_turns

Common pitfalls

• Treating all turns equally regardless of how many slots actually changed, which masks model performance on dynamic belief states.
• Double-counting prediction errors when a model incorrectly predicts a slot value and fails to update it to 'none' in subsequent turns.
• Ignoring partial correctness, as traditional metrics often use a hard binary pass/fail for each turn or slot.

Evidence (verbatim from paper)

Granular Change Accuracy (GCA) introduces a novel DST evaluation metric that addresses three key flaws in existing metrics: turn-centric uniform scoring, lack of partial credit for partially correct turns, and double-counting of prediction errors. By focusing on per-turn belief state changes rather than raw slot or turn accuracy, GCA reduces bias from error timing and error distribution, particularly in few-shot/zero-shot settings where model errors are more concentrated.

Citation

@misc{aksu2024granularchangeaccuracy,
  title={Granular Change Accuracy: A More Accurate Performance Metric for Dialogue State Tracking},
  author={Aksu et al. (2024)},
  year={2024},
  note={arXiv:2403.11123}
}

• arXiv: 2403.11123