Macedonian Benchmarks Eval

macedonian-benchmarks-eval

Towards Open Foundation Language Model and Corpus for Macedonian: A Low-Resource Language — Krsteski et al. (2025) (arXiv:2506.09560, 2025)

What this evaluates

Evaluates a low-resource language model's capability on standard commonsense reasoning, reading comprehension, and factual knowledge tasks adapted to Macedonian. It measures how well continued pretraining and instruction tuning improve performance on these benchmarks compared to multilingual baselines.

Datasets

• Macedonian Benchmarks (ARC Easy, ARC Challenge, BoolQ, HellaSwag, OpenBookQA, PIQA, WinoGrande) — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Fraction of correctly predicted answers out of the total number of instances in the benchmark.

Input / output format

Input: Multiple-choice questions in Macedonian.

Output: Model generates a predicted answer choice or text response.

Scoring recipe

def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return correct / len(gold_labels)

Common pitfalls

• Benchmarks are standard English datasets adapted to Macedonian; performance may not directly transfer to English or other languages.
• Qualitative human evaluation uses only 10 prompts, which may not represent full model capability or generalizability.
• Average score across 7 benchmarks is used as a single headline metric, masking task-specific strengths and weaknesses.

Evidence (verbatim from paper)

Finally, our model represents a significant improvement compared to the previous best Macedonian model, MKLLM, achieving higher accuracy across six out of seven benchmarks.

Citation

@misc{krsteski2025towards,
  title={Towards Open Foundation Language Model and Corpus for Macedonian: A Low-Resource Language},
  author={Krsteski et al. (2025)},
  year={2025},
  note={arXiv:2506.09560}
}

• arXiv: 2506.09560