Multi Turn Task Exec Eval

multi-turn-task-exec-eval

AURA: Agent for Understanding, Reasoning, and Automated Tool Use in Voice-Driven Tasks — Maben et al. (2025) (arXiv:2506.23049, 2025)

What this evaluates

Assesses an agent's ability to execute complex, multi-turn voice-driven tasks requiring tool invocation, memory, and reasoning across varying difficulty levels.

Datasets

• Custom Multi-Turn Voice Tasks — total 30; splits: test (30); repo https://github.com/Sentientia/Aura

Metrics

• Avg. Success (primary) — range: percent
  • Percentage of tasks successfully completed as rated by a human evaluator.
• Avg. Satisfaction — range: other
  • User satisfaction rated on a 1 to 5 scale by a human evaluator.

Input / output format

Input: Multi-turn spoken dialogue with compound goals requiring tool use and state tracking.

Output: Agent actions (tool calls, responses) and final task completion status.

Scoring recipe

def evaluate_tasks(interactions):
    success_scores = []
    satisfaction_scores = []
    for task in interactions:
        success_scores.append(human_evaluator.rate_success(task))
        satisfaction_scores.append(human_evaluator.rate_satisfaction(task))
    return sum(success_scores) / len(success_scores), sum(satisfaction_scores) / len(satisfaction_scores)

Common pitfalls

• Human evaluation by a single co-author may lack inter-rater reliability.
• Task difficulty categorization (Easy/Medium/Hard) is subjective and not formally defined in the text.
• Scores are averaged per difficulty level, which may mask performance on specific task types.

Evidence (verbatim from paper)

We evaluated AURA on 30 real-world tasks grouped by difficulty—Easy, Medium, and Hard—with harder tasks involving compound goals requiring multi-turn reasoning, memory, and tool use. A human evaluator (co-author) rates each interaction on difficulty, success, and satisfaction (1–5 scale); task details and scores are available in the repository.

Citation

@misc{maben2025aura,
  title={AURA: Agent for Understanding, Reasoning, and Automated Tool Use in Voice-Driven Tasks},
  author={Maben et al. (2025)},
  year={2025},
  note={arXiv:2506.23049}
}

• arXiv: 2506.23049