Gorilla API Eval

gorilla-api-eval

Gorilla: Large Language Model Connected with Massive APIs — Patil et al. (2023) (arXiv:2305.15334, 2023)

What this evaluates

Evaluates an LLM's ability to generate correct API invocation code from natural language prompts, with or without retrieved documentation. It measures how well the model selects the appropriate API, avoids hallucinating non-existent APIs, and respects functional constraints like accuracy thresholds.

Datasets

• APIBench — total ?; splits: train (-1), test (-1)

Metrics

• AST accuracy (primary) — range: percent
  • Percentage of generated API calls whose Abstract Syntax Tree (AST) exactly matches the ground truth API call. Reported alongside hallucination error rate and wrong-API selection error rate.

Input / output format

Input: Natural language user prompt. Optionally concatenated with retrieved API documentation in the format: Use this API documentation for reference: .

Output: Python/API invocation code string (e.g., torchvision.models.resnet50()).

Scoring recipe

def score_api_call(generated_code, gold_code):
    gen_ast = parse_ast(generated_code)
    gold_ast = parse_ast(gold_code)
    if gen_ast == gold_ast:
        return 'correct'
    elif is_hallucination(generated_code):
        return 'hallucination'
    else:
        return 'wrong_api'
# Overall accuracy = count(correct) / total * 100

Common pitfalls

• Using a non-optimal retriever (e.g., BM25) at test time can significantly misguide the model and drop accuracy by 20-50% compared to zero-shot prompting.
• For HuggingFace, the dataset is not exhaustive, so evaluation for baseline models is reduced to a multiple-choice domain selection task, making direct comparison with TorchHub/TensorFlow Hub AST accuracy metrics difficult.
• Hallucination metrics are highly sensitive to RLHF tuning; GPT-3.5 sometimes exhibits fewer hallucinations than GPT-4 in zero-shot settings, contrary to expectations.

Evidence (verbatim from paper)

We report the overall accuracy, the error by hallucination and the error by selecting wrong API call. Note that for TorchHub and TensorHub, we evaluate all the models using AST tree accuracy score.

Citation

@misc{patil2023gorilla,
  title={Gorilla: Large Language Model Connected with Massive APIs},
  author={Patil et al. (2023)},
  year={2023},
  note={arXiv:2305.15334}
}

• arXiv: 2305.15334