Anytool Eval

anytool-eval

AnyTool: Self-Reflective, Hierarchical Agents for Large-Scale API Calls — Du et al. (2024) (arXiv:2402.04253, 2024)

What this evaluates

Evaluates an agent's ability to retrieve and invoke relevant APIs from a large-scale pool to resolve user queries. It probes hierarchical API retrieval, self-reflective error recovery, and the capacity to handle context limits when dealing with thousands of available tools.

Datasets

• ToolBench (filtered) — total 632; splits: G1-I (115), G1-T (132), G1-C (142), G2-I (107), G2-C (98), G3-I (38)
• AnyToolBench — total 400; splits: test (400)

Metrics

• pass rate (primary) — range: percent
  • The percentage of user queries for which the agent's generated solution successfully resolves the task. Calculated as (number of resolved queries / total queries) × 100%. Success is determined by a GPT-4-32K judge using the ToolBench evaluation prompt.

Input / output format

Input: User query describing a task, along with access to an API pool (either reference APIs, a candidate pool, or the full 16,000+ API collection).

Output: A sequence of API calls and parameters generated by the agent to solve the query.

Scoring recipe

def compute_pass_rate(predictions, gold_queries):
    resolved_count = 0
    for query, solution in zip(gold_queries, predictions):
        # GPT-4-32K judge evaluates if the solution resolves the query
        if gpt4_judge(solution, query) == 'success':
            resolved_count += 1
    return (resolved_count / len(gold_queries)) * 100

Common pitfalls

• Prior benchmarks artificially inflated pass rates by providing reference APIs or using simpler judges; this protocol requires the agent to retrieve APIs from a pool of 16,000+ without reference lists.
• GPT-4's context window limits prevent feeding all APIs at once; the evaluation relies on a hierarchical retriever to manage context, which must be accounted for when reproducing results.
• The GPT-4 judge alignment with humans is high (96.5%), but using GPT-3.5 drops alignment to 73.9%, so the exact judge model matters.

Evidence (verbatim from paper)

We employ the pass rate (as defined in Eq. 2) as our evaluation metric. To assess whether a solution generated by an agent can resolve the query, we use GPT-4-32K. The same prompt utilized in ToolBench is applied when GPT-4 serves as the judge.

Citation

@misc{du2024anytool,
  title={AnyTool: Self-Reflective, Hierarchical Agents for Large-Scale API Calls},
  author={Du et al. (2024)},
  year={2024},
  note={arXiv:2402.04253}
}

• arXiv: 2402.04253