Haerae Bench Eval

haerae-bench-eval

HAE-RAE Bench: Evaluation of Korean Knowledge in Language Models — Son et al. (2023) (arXiv:2309.02706, 2023)

What this evaluates

Evaluates language models' proficiency in Korean cultural knowledge and context. It probes capabilities across vocabulary (loan words, standard nomenclature, rare words), history, general knowledge, and reading comprehension, specifically highlighting the limitations of English-trained or non-Korean-tailored models.

Datasets

• HAE-RAE Bench — total 1500; splits: test (1500)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly answered multiple-choice questions. For open-weight models, computed via log-likelihood over five options. For proprietary models, computed via exact match of the generated option number.

Input / output format

Input: Multiple-choice questions with five options, presented in 0-shot, 5-shot, or 10-shot formats with exemplars.

Output: For open models: log probabilities for each of the five options. For proprietary models: the number of the selected option.

Scoring recipe

def compute_accuracy(predictions, golds):
    correct = 0
    for pred, gold in zip(predictions, golds):
        if pred == gold:
            correct += 1
    return (correct / len(golds)) * 100

Common pitfalls

• Directly comparing log-likelihood accuracy (open models) with generation-based accuracy (proprietary models) is not feasible due to different evaluation methods.
• Assuming in-context learning significantly improves performance; the benchmark shows ICL is insufficient for cultural alignment.
• Assuming model size strongly predicts performance; ANOVA shows only ~26% of variance is explained by parameter count.

Evidence (verbatim from paper)

Unlike openly available models for which we leveraged a log probability method to gauge accuracy, these models do not provide log probabilities for individual tokens. Accordingly, we prompted the models to generate the number of the options they deemed correct. Direct comparison between these evaluation methods is not feasible. However, the method used for proprietary models is more challenging than the log-likelihood method applied to open models. The former entails generating answers from the entire vocabulary, whereas the latter restricts choices to five options.

Citation

@misc{son2023haeraebench,
  title={HAE-RAE Bench: Evaluation of Korean Knowledge in Language Models},
  author={Son et al. (2023)},
  year={2023},
  note={arXiv:2309.02706}
}

• arXiv: 2309.02706