Bfcl Eval

bfcl-eval

APIGen: Automated Pipeline for Generating Verifiable and Diverse Function-Calling Datasets — Liu et al. (2024) (arXiv:2406.18518, 2024)

What this evaluates

Evaluates an LLM's ability to generate correct function calls from natural language prompts, covering single, multiple, parallel, and parallel-multiple API invocations across different programming languages.

Datasets

• Berkeley Function-Calling Benchmark (BFCL) — total 2000; splits: test (2000)

Metrics

• Overall Accuracy (primary) — range: percent
  • Weighted average of AST Evaluation and Executable Function Evaluation scores across four complexity levels (Simple, Multiple, Parallel, Parallel Multiple), combined with Relevance Detection.
• AST Evaluation — range: percent
  • Measures syntactic accuracy of generated function calls against predefined documentation, checking function names, required parameters, and data types.
• Executable Function Evaluation — range: percent
  • Runs the generated function calls to verify operational correctness and expected results.

Input / output format

Input: Natural language queries requiring API calls, covering single, multiple, parallel, and parallel-multiple function invocations across Java, JavaScript, and Python APIs.

Output: JSON-format function calls specifying the function name and parameters.

Scoring recipe

def score_bfcl(predictions, gold):
    ast_scores, exec_scores = [], []
    for pred, gold_item in zip(predictions, gold):
        ast_scores.append(check_ast_structure(pred, gold_item.schema))
        exec_scores.append(run_and_compare(pred, gold_item.expected_output))
    # Weighted average across complexity levels (Simple, Multiple, Parallel, Parallel Multiple)
    overall = weighted_average(ast_scores, exec_scores, gold.complexity_levels)
    return overall

Common pitfalls

• AST accuracy does not guarantee runtime correctness; models may output syntactically valid JSON that fails execution due to type mismatches or missing dependencies.
• Parallel and multiple function-calling scenarios are significantly harder and underrepresented in public datasets, leading to performance drops if models are only trained on simple single-call data.
• Relevance detection (identifying whether a function should be called at all) is evaluated separately from function call accuracy, and poor relevance detection can skew overall scores.

Evidence (verbatim from paper)

Benchmark. We evaluate the trained models' performance on the Berkeley Function-Calling Benchmark (BFCL) [9], which provides a comprehensive evaluation framework for assessing the function-calling capabilities of LLMs across various programming languages and application domains. ... The Berkeley Function-Calling Leaderboard (BFCL) evaluates LLMs using two main categories: Abstract Syntax Tree (AST) Evaluation and Executable Function Evaluation. ... The rank is based on the overall accuracy, which is a weighted average of different evaluation categories.

Citation

@misc{liu2024apigen,
  title={APIGen: Automated Pipeline for Generating Verifiable and Diverse Function-Calling Datasets},
  author={Liu et al. (2024)},
  year={2024},
  note={arXiv:2406.18518}
}

• arXiv: 2406.18518