Meta Tool Eval

meta-tool-eval

Meta-Tool: Efficient Few-Shot Tool Adaptation for Small Language Models — Kumar et al. (2026) (arXiv:2604.20148, 2026)

What this evaluates

Evaluates small language models' capability to adapt to and execute diverse tool-use tasks, including REST API calls, complex SQL queries, web navigation, and command-line interactions. It measures both functional correctness and efficiency under few-shot prompting and lightweight adaptation.

Datasets

• Gorilla APIBench (BFCL V4) — total ?; splits: test (-1)
• Spider 2.0 (Enterprise Subset) — total ?; splits: test (-1)
• WebArena — total ?; splits: test (-1)
• InterCode (Bash & CTF) — total ?; splits: test (-1)

Metrics

• Execution Success Rate (SR) (primary) — range: percent
  • Percentage of tasks where the agent’s action leads to the desired state change or returns the correct data. Calculated as (correct tasks / total tasks) * 100.
• Pass@1 — range: percent
  • Percentage of tasks solved on the first attempt. Calculated as (tasks solved on first try / total tasks) * 100.
• Adaptation Time — range: other
  • Wall-clock time required to prepare the agent for a new tool, from receiving documentation to readiness.
• Inference Latency — range: other
  • Time taken to generate the action.

Input / output format

Input: Task prompts containing tool documentation, few-shot examples, and environment states (API specs, database schemas, web pages, or shell environments) depending on the benchmark.

Output: Model-generated actions (e.g., API calls, SQL queries, web navigation steps, or bash commands) executed in their respective sandboxed environments.

Scoring recipe

correct_count = 0
first_try_count = 0
for task in tasks:
    action = model.generate(prompt)
    result = execute(action, task.environment)
    if result.matches_gold(task):
        correct_count += 1
        first_try_count += 1
sr = (correct_count / len(tasks)) * 100
pass_at_1 = (first_try_count / len(tasks)) * 100

Common pitfalls

• Confusing AST/syntax matching with actual execution success; the paper explicitly uses the 'Executable' subset where success depends on real API responses and state changes.
• Overlooking the distinction between adaptation time (preparation overhead) and inference latency (generation speed), which measure different efficiency bottlenecks.
• Assuming larger specialized models automatically outperform smaller adapted ones without accounting for context window limits and retrieval memory constraints.

Evidence (verbatim from paper)

Execution Success Rate (SR): The primary metric. The percentage of tasks where the agent’s action leads to the desired state change or returns the correct data. Pass@1: The percentage of tasks solved on the first attempt.

Citation

@misc{kumar2026metatool,
  title={Meta-Tool: Efficient Few-Shot Tool Adaptation for Small Language Models},
  author={Kumar et al. (2026)},
  year={2026},
  note={arXiv:2604.20148}
}

• arXiv: 2604.20148