Openapi Completion Eval

openapi-completion-eval

Optimizing Large Language Models for OpenAPI Code Completion — Petryshyn et al. (2024) (arXiv:2405.15729, 2024)

What this evaluates

Evaluates an LLM's ability to perform code infilling for OpenAPI specifications by predicting masked sections of API definitions. It probes semantic understanding of API structure, syntax correctness, and the model's robustness to varying context sizes and prompt formats.

Datasets

• masked OpenAPI definitions — total ?; splits: (unstated)

Metrics

• correctness (primary) — range: percent
  • Percentage of OpenAPI definitions that are correctly completed to match the expected ground truth.
• validity — range: percent
  • Percentage of generated completions that parse as syntactically valid OpenAPI specifications.

Input / output format

Input: A partial OpenAPI definition with a <<MASK>> marker indicating the location of the missing code, provided with a prefix and suffix context.

Output: The completed OpenAPI definition text filling the masked region.

Scoring recipe

correct_count = 0
valid_count = 0
for pred, gold in zip(predictions, golds):
    if pred == gold:
        correct_count += 1
    if is_valid_openapi(pred):
        valid_count += 1
correctness = (correct_count / len(predictions)) * 100
validity = (valid_count / len(predictions)) * 100

Common pitfalls

• Manual evaluation of GitHub Copilot introduces human latency and error, making automated comparison difficult and inflating generation time measurements.
• Fine-tuned models exhibit strong context-size bias, significantly underperforming when inference context is smaller than the training context.
• SPM infilling format often requires token healing at inference time, which can artificially lower validity scores if not handled.

Evidence (verbatim from paper)

The correctness rate, which is the primary target metric in this research, demonstrates a much smoother trend with a significant increase up to the context size of 4096 tokens. Following the completion procedure described, GitHub Copilot correctly completed 29% of the OpenAPI definitions. 68% of the completed definitions were valid.

Citation

@misc{petryshyn2024optimizing,
  title={Optimizing Large Language Models for OpenAPI Code Completion},
  author={Petryshyn et al. (2024)},
  year={2024},
  note={arXiv:2405.15729}
}

• arXiv: 2405.15729