Travelplanner Eval

travelplanner-eval

TravelPlanner: A Benchmark for Real-World Planning with Language Agents — Xie et al. (2024) (arXiv:2402.01622, 2024)

What this evaluates

Evaluates language agents' ability to perform long-horizon, multi-constraint real-world travel planning. It probes their capacity for dynamic tool use, constraint tracking, commonsense reasoning, and maintaining task coherence across complex decision-making steps.

Datasets

• TravelPlanner — total 1225; splits: test (-1)

Metrics

• macro pass rate (primary) — range: percent
  • The percentage of tasks where the generated plan satisfies all hard constraints. Calculated as (number of fully satisfied tasks / total tasks) * 100.
• micro pass rate — range: percent
  • The average constraint satisfaction rate across all individual constraints in a task, measuring partial success even when the full plan fails.

Input / output format

Input: Natural language travel planning queries with multiple constraints. In two-stage mode, agents receive the query and access to six real-world tools for information collection. In sole-planning mode, agents are provided with the necessary pre-collected information.

Output: A structured travel plan (itinerary) specifying activities, bookings, and routes that must satisfy all specified constraints.

Scoring recipe

def compute_pass_rate(predictions, gold_constraints):
    satisfied = 0
    for pred, constraints in zip(predictions, gold_constraints):
        if all(check_constraint(pred, c) for c in constraints):
            satisfied += 1
    return (satisfied / len(predictions)) * 100

Common pitfalls

• Agents often fail to track multiple constraints holistically, satisfying some but missing others, leading to low macro pass rates despite decent micro scores.
• The two-stage mode (tool use + planning) causes significant performance degradation compared to sole-planning due to limited cognitive capacity and multitasking overhead.
• Greedy search optimizing only for cost may outperform LLMs on hard constraints, highlighting that LLMs struggle with constraint satisfaction rather than cost minimization.

Evidence (verbatim from paper)

In the two-stage mode, GPT-4-Turbo with ReAct achieves only 0.6% in the final pass rate, and none of the other LLMs can pass any of the tasks. ... Agents struggle in obtaining a high macro pass rate. While some agents achieve high micro scores, their macro scores remain low.

Citation

@misc{xie2024travelplanner,
  title={TravelPlanner: A Benchmark for Real-World Planning with Language Agents},
  author={Xie et al. (2024)},
  year={2024},
  note={arXiv:2402.01622}
}

• arXiv: 2402.01622