qhjqhj00 spokenwoz-dst-eval
AURA: Agent for Understanding, Reasoning, and Automated Tool Use in Voice-Driven Tasks — Maben et al. (2025) (arXiv:2506.23049, 2025)
What this evaluates
Evaluates dialog state tracking performance by measuring how accurately an agent predicts and maintains the current state of a multi-turn spoken conversation.
Datasets
- SpokenWOZ — total ?; splits: test (-1)
Metrics
Joint Goal Accuracy (JGA)(primary) — range: percent- Exact match between predicted and ground-truth dialog states across turns. Calculated as (number of turns with exact state match / total turns) * 100.
Input / output format
Input: Spoken dialogue turns with domain classification and slot-filling requirements.
Output: Predicted dialog state (set of slot-value pairs) per turn.
Scoring recipe
def compute_jga(predictions, gold):
exact_matches = sum(1 for p, g in zip(predictions, gold) if p == g)
return (exact_matches / len(gold)) * 100
Common pitfalls
- JGA requires exact match of all slots, making it sensitive to minor formatting or ordering differences.
- Domain classification is done via prompt-based inference prior to DST, which can propagate errors into state tracking.
- No fine-tuning is used, relying entirely on prompt-based inference, which may not reflect fine-tuned system capabilities.
Evidence (verbatim from paper)
We evaluate AURA’s dialog state tracking (DST) on the SpokenWOZ benchmark using Joint Goal Accuracy (JGA), which measures the exact match between predicted and ground-truth dialog states across turns. AURA outperforms the best baseline by over 3 points, achieving a JGA of 28.76 using LLaMA3.3-70B with prompt-based DST.
Citation
@misc{maben2025aura,
title={AURA: Agent for Understanding, Reasoning, and Automated Tool Use in Voice-Driven Tasks},
author={Maben et al. (2025)},
year={2025},
note={arXiv:2506.23049}
}
- arXiv: 2506.23049