Tool Learning Eval

tool-learning-eval

Tool Learning with Foundation Models — Qin et al. (2023) (arXiv:2304.08354, 2023)

What this evaluates

Evaluates foundation models' ability to decompose complex instructions, reason over subgoals, and dynamically select/call external APIs or tools to complete tasks across diverse domains like translation, mathematics, web search, and data processing.

Datasets

• MLQA — total 200; splits: test (200)
• ASDiv — total 266; splits: test (266)
• MathQA — total ?; splits: test (-1)
• RealTimeQA — total 30; splits: test (30)
• HotpotQA — total 200; splits: test (200)
• WebShop — total 100; splits: test (100)
• ALFWorld — total 134; splits: valid (134)
• Curated (Map) — total 129; splits: test (129)
• Curated (Weather) — total 100; splits: test (100)
• Curated (Stock) — total 122; splits: test (122)
• Curated (Slides) — total 100; splits: test (100)
• Curated (Tables) — total 117; splits: test (117)
• Curated (KGs) — total 64; splits: test (64)
• Curated (Cooking) — total 50; splits: test (50)
• Curated (Movie) — total 60; splits: test (60)
• Curated (AI Painting) — total 25; splits: test (25)
• Curated (3D Model Construction) — total 10; splits: test (10)
• Curated (Chemical Properties) — total 100; splits: test (100)
• Curated (Database) — total 50; splits: test (50)

Metrics

• accuracy / success rate (primary) — range: percent
  • Percentage of instances where the model's final answer matches the ground truth, or where the sequence of API calls is judged correct and yields a reasonable result. For curated tool-use datasets, human judges verify that all API calls are accurate and produce valid outputs.
• F1-score — range: [0, 1]
  • Harmonic mean of precision and recall computed over token-level matches between predicted and gold answers. Used specifically for the MLQA multilingual QA benchmark.

Input / output format

Input: User query/instruction, optionally with context (e.g., English context for translation, question for QA). For tool-use tasks, the model receives the instruction and descriptions of available APIs.

Output: Final answer string (for QA/math) or a sequence of tool/API calls followed by a final answer/result. The model must generate tool calls in the framework's expected format.

Scoring recipe

def compute_metric(predictions, gold, dataset_name):
    if dataset_name == 'MLQA':
        return f1_score(predictions, gold)
    elif dataset_name in ['ASDiv', 'MathQA', 'RealTimeQA', 'HotpotQA', 'WebShop', 'ALFWorld']:
        return exact_match(predictions, gold)
    else: # Curated tool-use datasets
        return 1.0 if judge_api_trace_correct(predictions) else 0.0

Common pitfalls

• Many tasks (e.g., slides-making, 3D modeling) cannot be completed without tools, so 'No Tool' baselines are omitted, which may skew cross-task comparisons.
• Curated datasets rely on human judgment of API call traces rather than automated exact-match, introducing subjectivity and potential inconsistency.
• Zero-shot tool use sometimes degrades performance compared to no-tool baselines for tasks solvable with internal knowledge (e.g., calculator, search engine).

Evidence (verbatim from paper)

Unless otherwise specified, for these manually curated test sets, we employ the trace of API calls as the metric for evaluating the models’ performance. Specifically, if humans judge that all the API calls are accurate for the given task, and they yield a reasonable result, the task is deemed to be correctly completed.

Citation

@misc{qin2023toollearning,
  title={Tool Learning with Foundation Models},
  author={Qin et al. (2023)},
  year={2023},
  note={arXiv:2304.08354}
}

• arXiv: 2304.08354