Function Calling Eval

function-calling-eval

ToolACE: Winning the Points of LLM Function Calling — Liu et al. (2024) (arXiv:2409.00920, 2024)

What this evaluates

Evaluates an LLM's ability to correctly identify, retrieve, and invoke external APIs or functions based on a user query. It probes zero-shot and multi-turn function-calling capabilities, including handling live vs. non-live APIs, detecting irrelevant queries, and mitigating hallucinations.

Datasets

• BFCL-v3 — total ?; splits: test (-1)
• API-Bank — total ?; splits: test (-1)

Metrics

• AST (primary) — range: percent
  • Percentage of test cases where the model's predicted function call matches the ground truth in abstract syntax tree structure.
• Exec — range: percent
  • Percentage of test cases where the predicted function call is syntactically valid and executable.
• Overall — range: percent
  • Composite score combining AST and Exec accuracy across single-turn and multi-turn scenarios.
• Relevance — range: percent
  • Accuracy in correctly identifying whether a query requires a function call.
• Irrelevance — range: percent
  • Accuracy in correctly identifying whether a query does not require a function call.

Input / output format

Input: Natural language user query or instruction, optionally with conversation history for multi-turn settings.

Output: Structured function call specification (e.g., JSON or code) containing the function name and arguments.

Scoring recipe

def compute_metrics(predictions, gold):
    ast_correct = sum(1 for p, g in zip(predictions, gold) if p['ast'] == g['ast'])
    exec_correct = sum(1 for p, g in zip(predictions, gold) if p['exec'] == g['exec'])
    rel_correct = sum(1 for p, g in zip(predictions, gold) if p['rel'] == g['rel'])
    irrel_correct = sum(1 for p, g in zip(predictions, gold) if p['irrel'] == g['irrel'])
    n = len(predictions)
    return {
        'AST': ast_correct / n * 100,
        'Exec': exec_correct / n * 100,
        'Overall': (ast_correct + exec_correct) / (2 * n) * 100,
        'Relevance': rel_correct / n * 100,
        'Irrelevance': irrel_correct / n * 100
    }

Common pitfalls

• Models often hallucinate non-existent APIs or invent arguments when faced with out-of-distribution queries.
• Multi-turn evaluation requires strict context management; forgetting previous turns or mixing up API states leads to severe accuracy drops.
• Live API benchmarks require handling real-time network calls, which can introduce latency and non-determinism not present in non-live evaluations.

Evidence (verbatim from paper)

The findings in BFCL indicate that API-based models demonstrate significant advantages over open-source models, such as the Claude series and the GPT-4 series. Open-source models fine-tuned for function calling, such as Functionary and xLAM, exhibit competitive performance, but still fall short of the leading models. Our ToolACE-8B model outperforms most API-based and open-source models in both the AST and Exec categories of BFCL, and continues to exhibit substantial advantages over all the open-source models in the context of API-Bank, demonstrating the effectiveness of our training data for functional calling.

Citation

@misc{liu2024toolace,
  title={ToolACE: Winning the Points of LLM Function Calling},
  author={Liu et al. (2024)},
  year={2024},
  note={arXiv:2409.00920}
}

• arXiv: 2409.00920