Gptaraeval Eval

gptaraeval-eval

GPTAraEval: A Comprehensive Evaluation of ChatGPT on Arabic NLP — Khondaker et al. (2023) (arXiv:2305.14976, 2023)

What this evaluates

Evaluates large language models on Arabic natural language understanding and generation across 44 tasks and over 60 datasets, covering both Modern Standard Arabic and dialectal varieties.

Datasets

• GPTAraEval Benchmark Suite — total ?; splits: test (200), train (-1)

Metrics

• macro-F1 (primary) — range: [0, 1]
  • Macro-averaged F1 score computed across all classes for NLU tasks. NLG tasks use task-appropriate metrics (e.g., exact match, ROUGE, BLEU) as detailed in the paper's Table 2.

Input / output format

Input: Prompt containing the task instruction, optionally augmented with k few-shot examples (k ∈ {0, 3, 5, 10}) sampled from the training set, followed by the test instance.

Output: Free-form text response generated by the model, post-processed by stripping leading and trailing whitespace.

Scoring recipe

# For each dataset:
test_subset = random.sample(full_test_set, 200)
predictions = []
for instance in test_subset:
    prompt = build_prompt(instance, few_shot_examples)
    response = model.generate(prompt, temperature=0.0)
    predictions.append(response.strip())
if task_type == "NLU":
    metric_value = macro_f1(gold_labels=test_subset.labels, preds=predictions)
else:
    metric_value = task_specific_metric(gold=test_subset.text, preds=predictions)

Common pitfalls

• Evaluation uses a fixed 200-example subset from each test set rather than the full test set, which may limit statistical power and comparability with full-benchmark scores.
• Few-shot examples are nested (k-shot includes all examples from n < k shots), which differs from standard independent sampling and may artificially inflate performance trends.
• Post-processing only removes leading/trailing whitespace, potentially leaving formatting artifacts that negatively impact exact-match or string-based NLG metrics.

Evidence (verbatim from paper)

For evaluation, we randomly sample a set of 200 examples from the test set of each dataset, to keep the cost manageable. We evaluate ChatGPT (gpt-3.5-turbo), which is an optimized version of GPT-3.5 series. We set the temperature to 0.0 while generating responses from ChatGPT. We compare this model with BLOOMZ (7.1B parameters)... For NLU tasks, we use macro-F1 scores and for NLG tasks we use the appropriate metric suited to each task (Table 2).

Citation

@misc{khondaker2023gptaraeval,
  title={GPTAraEval: A Comprehensive Evaluation of ChatGPT on Arabic NLP},
  author={Khondaker et al. (2023)},
  year={2023},
  note={arXiv:2305.14976}
}

• arXiv: 2305.14976