Greek LLM Benchmark Eval

greek-llm-benchmark-eval

Open or Closed LLM for Lesser-Resourced Languages? Lessons from Greek — Pavlopoulos et al. (2025) (arXiv:2501.12826, 2025)

What this evaluates

Evaluates open-source (Llama-70b) and closed-source (GPT-4o mini) LLMs across seven distinct NLP tasks in Modern Greek. It probes capabilities in classification, sequence labeling, text generation, and machine translation to assess model performance in a lesser-resourced language setting.

Datasets

• SemEval-2020 Task 12 (OffensEval-2020 Greek) — total 175; splits: test (175); repo https://sites.google.com/site/offensevalsharedtask/offenseval-2020
• Greek Native Corpus (GNC) — total 227; splits: test (227)
• Global Voices Greek MT Corpus — total 175; splits: test (175); repo https://globalvoices.org/
• Areios Pagos Legal Summarization Corpus — total 175; splits: test (175); repo https://www.areiospagos.gr/
• University Help Desk Intent Classification Dataset — total 175; splits: test (175)
• Greek NER Annotated Dataset — total 5447; splits: test (5447); repo https://github.com/eellak/gsoc2018-spacy
• Greek Treebank (POS) — total 456; splits: test (456)

Metrics

• macro-F1 (primary) — range: [0, 1]
  • Unweighted mean of per-class F1 scores. Computed as the average of recall or precision for each class regardless of class imbalance.
• weighted-F1 — range: [0, 1]
  • Mean of per-class F1 scores weighted by the number of true instances for each class.
• WER — range: percent
  • Word Error Rate: (Substitutions + Deletions + Insertions) / Total Words in reference.
• CER — range: percent
  • Character Error Rate: (Substitutions + Deletions + Insertions) / Total Characters in reference.
• BERTScore F1 (primary) — range: [0, 1]
  • F1 score computed using contextual embeddings from BERT to match predictions to references based on cosine similarity.
• ROUGE-1/2/L F1 — range: [0, 1]
  • F1 scores for unigram (1), bigram (2), and longest common subsequence (L) n-gram overlaps between prediction and reference.

Input / output format

Input: Zero-shot prompts containing task instructions and, for POS tagging and NER, class/tag definitions with explanations. Input is raw Greek text (sentences, documents, or queries).

Output: Task-specific predictions: class labels (Toxicity, Intent), corrected text (GEC), translated text (MT), generated summary (Summarization), or token-level entity/POS tags (NER, POS).

Scoring recipe

def compute_metrics(predictions, references, task):
    if task in ['Toxicity', 'Intent', 'NER', 'POS']:
        return macro_f1(predictions, references), weighted_f1(predictions, references)
    elif task == 'GEC':
        return wer(references, predictions), cer(references, predictions)
    elif task == 'MT':
        return wer(references, predictions), cer(references, predictions), bertscore_f1(references, predictions)
    elif task == 'Summarization':
        return bertscore_f1(references, predictions), rouge1_f1(references, predictions), rouge2_f1(references, predictions), rougeL_f1(references, predictions)

Common pitfalls

• Only 175 instances were randomly sampled per test set (except NER/POS) primarily to limit API costs, which may not fully represent dataset distribution.
• GEC performance for GPT-4o mini may be artificially inflated if the model was exposed to the Korre et al. (2021) training data during pre-training.
• Low macro-F1 scores in NER and POS tagging stem from severe class imbalance and rare tags (e.g., MISC, X), masking high weighted-F1 performance on dominant classes.

Evidence (verbatim from paper)

Using character (CER) and word (WER) error rate, we find that gpt performs significantly better than llama in correcting grammatical errors. It is halved in WER compared to llama and only 1.74 in CER.

Citation

@misc{pavlopoulos2025openclosed,
  title={Open or Closed LLM for Lesser-Resourced Languages? Lessons from Greek},
  author={Pavlopoulos et al. (2025)},
  year={2025},
  note={arXiv:2501.12826}
}

• arXiv: 2501.12826