Vllm Eval

vllm-eval

ViLLM-Eval: A Comprehensive Evaluation Suite for Vietnamese Large Language Models — Nguyen et al. (2024) (arXiv:2404.11086, 2024)

What this evaluates

Evaluates Vietnamese large language models on contextual reasoning, academic knowledge, general trivia, and long-form reading comprehension. Probes both language modeling capability (perplexity) and factual/reasoning accuracy across culturally and linguistically specific tasks.

Datasets

• LAMBADA Vietnamese — total ?; splits: val (-1), test (-1)
• Exam Vietnamese — total ?; splits: val (-1), test (-1)
• General Knowledge — total ?; splits: val (-1), test (-1)
• Comprehension QA — total ?; splits: val (-1), test (-1)

Metrics

• perplexity — range: other
  • Computed as the exponential of the average negative log-likelihood of the target word across the dataset.
• accuracy (primary) — range: [0, 1]
  • The fraction of instances where the model's predicted word or answer choice exactly matches the ground-truth label.

Input / output format

Input: For LAMBADA: paragraphs with the last word of the final sentence masked. For multiple-choice datasets: questions with four answer options, presented in a five-shot setting using five exemplars from the development split.

Output: For LAMBADA: the exact last word. For multiple-choice datasets: the selected answer choice (letter or text), determined either by the highest probability among options or extracted via regular expression from generated text.

Scoring recipe

# LAMBADA
ppl = exp(-mean(log_prob(last_word)))
acc = sum(1 for p, g in zip(preds, golds) if p == g) / len(golds)

# MCQ (5-shot)
if model_provides_probs:
    pred = argmax(prob_distribution[choices])
else:
    pred = regex_extract(generated_text, pattern="answer_choice")
acc = sum(1 for p, g in zip(preds, golds) if p == g) / len(golds)

Common pitfalls

• Test set labels are kept private; evaluation must be run via the provided library or submissions go to a public leaderboard.
• Zero-shot prompting yields suboptimal results; a five-shot setting with development exemplars is required for reliable performance.
• Models lacking probability APIs require regex-based answer extraction, which may fail if model output formatting deviates from expected patterns.

Evidence (verbatim from paper)

This work aims to use perplexity and accuracy as the metric. While ground-truth labels of the validation splits are released, the labels of the test split are kept private. This is to ensure the fair use of the ViLLM-Eval, as the ViLLM-Eval data may unconsciously be included in pre-training data due to web crawling. Instead, users can utilize our provided library and run the evaluation directly, where the ground-truth labels are embedded within the library but encrypted. A public leaderboard is maintained to display the test accuracy. Users have an option to include their submission results in the live leaderboard, depending on their own preference. For the LAMBADA Vietnamese dataset. The evaluation of LLMs involves calculating the perplexity of the last word of the last sentence of each input paragraph. Additionally, accuracy is used to show that the model accurately predicts that last word. At the same time, accuracy also helps calculate average comparison results. For datasets in the form of multiple-choice questions, LLMs are evaluated in a five-shot setting on ViLLM-Eval, where the five exemplars are from the development split. Probability distributions are utilized to determi

Citation

@misc{nguyen2024vllmeval,
  title={ViLLM-Eval: A Comprehensive Evaluation Suite for Vietnamese Large Language Models},
  author={Nguyen et al. (2024)},
  year={2024},
  note={arXiv:2404.11086}
}

• arXiv: 2404.11086