Gaia Eval

gaia-eval

GAIA: a benchmark for General AI Assistants — Mialon et al. (2023) (arXiv:2311.12983, 2023)

What this evaluates

GAIA probes the ability of AI assistants to perform real-world, conceptually simple tasks that require multi-step reasoning, tool use, and multi-modal processing. It measures robustness in practical everyday reasoning and factual validation on questions explicitly designed to be outside the model's training data.

Datasets

• GAIA — total 466; splits: test (466)

Metrics

• score (primary) — range: [0, 1]
  • The fraction of questions answered correctly out of the total benchmark set. Answers are automatically validated against ground truth to ensure unambiguous evaluation.

Input / output format

Input: A prefix prompt specifying the required answer format, followed by a real-world question that may require web search, file reading, computation, or multi-modal processing.

Output: A structured answer adhering to the format specified in the prefix prompt (exact format details are in Figure 2 of the paper).

Scoring recipe

correct = 0
for question, gold_answer in dataset:
    prediction = model.generate(prefix_prompt + question)
    if prediction == gold_answer:
        correct += 1
accuracy = correct / len(dataset)

Common pitfalls

• GPT-4 with plugins score is an 'oracle' estimate because plugins are manually selected per task and are unstable or disappear, making it non-reproducible.
• Non-tool models can achieve non-zero scores on tool/modality tasks due to memorization of intermediate steps or alternative solving paths not intended by annotators.
• AutoGPT results are heavily influenced by its specific API prompt/generation parameters rather than just the underlying LLM capability.

Evidence (verbatim from paper)

Figure 4 Scores and time to answer per method and level. As stated in the main text, GPT4 + plugins score should be seen as an oracle since the plugins were chosen manually depending on the question.

Citation

@misc{mialon2023gaia,
  title={GAIA: a benchmark for General AI Assistants},
  author={Mialon et al. (2023)},
  year={2023},
  note={arXiv:2311.12983}
}

• arXiv: 2311.12983