Nlebench Norwegian Eval

nlebench-norwegian-eval

NLEBench+NorGLM: A Comprehensive Empirical Analysis and Benchmark Dataset for Generative Language Models in Norwegian — Liu et al. (2023) (arXiv:2312.01314, 2023)

What this evaluates

Evaluates generative language models on Norwegian across multiple tasks including conversational dialogue, news summarization, instruction following, document-grounded QA, factual consistency, toxicity, and bias. It probes low-resource language capabilities, cultural understanding, and reasoning via chain-of-thought prompting.

Datasets

• NO-ConvAI2 — total ?; splits: test (-1); repo https://github.com/Smartmedia-AI/NorGLM
• NO-CNN/DailyMail — total ?; splits: test (-1); repo https://github.com/Smartmedia-AI/NorGLM
• NO-Alpaca-Plus — total ?; splits: test (-1); repo https://github.com/Smartmedia-AI/NorGLM
• NO-CrowS-Pairs — total ?; splits: test (-1); repo https://github.com/Smartmedia-AI/NorGLM
• NO-Multi-QA-Sum — total ?; splits: test (-1); repo https://github.com/Smartmedia-AI/NorGLM

Metrics

• BLEU (primary) — range: percent
  • Standard n-gram overlap metric between generated and reference text, reported as a percentage in the paper.
• ROUGE-1 — range: percent
  • Unigram recall-based overlap metric between generated and reference text, reported as a percentage.
• ROUGE-L — range: percent
  • Longest common subsequence recall-based overlap metric between generated and reference text, reported as a percentage.
• Distinct-4 — range: percent
  • Ratio of unique 4-grams to total 4-grams in the generated text, measuring lexical diversity, reported as a percentage.
• MAUVE — range: other
  • Measures distributional distance between generated and human-written text based on probability distributions. Note: paper reports values exceeding 1.0, indicating non-standard scaling or reporting.
• Entailment Score — range: percent
  • Faithfulness metric computed using a fine-tuned NorBERT model on the VitaminC dataset to measure factual consistency between generated text and source context.
• Toxicity Score — range: [0, 1]
  • Average toxicity score across six perspectives (Toxicity, Severe toxicity, Identity attack, Insult, Profanity, Threat), ranging from 0 to 1.
• Bias Score — range: percent
  • Percentage of samples where the model assigns higher perplexity to anti-stereotype sentences than to stereotype sentences, indicating bias towards public stereotypes.

Input / output format

Input: Task-specific prompts in Norwegian, including conversation turns, source documents paired with questions, instruction prompts, or stereotype/anti-stereotype sentence pairs.

Output: Generated Norwegian text (responses, summaries, answers) or model perplexity scores for bias/toxicity evaluation.

Scoring recipe

def compute_metrics(predictions, references, contexts, stereotype_pairs):
    bleu = compute_bleu(references, predictions) * 100
    rouge1 = compute_rouge(references, predictions, 'rouge1') * 100
    rougeL = compute_rouge(references, predictions, 'rougeL') * 100
    distinct4 = len(set(ngrams(predictions, 4))) / len(list(ngrams(predictions, 4))) * 100
    mauve = compute_mauve(references, predictions)
    entailment = compute_nli_score(predictions, contexts, model='NorBERT-VitaminC')
    toxicity = compute_toxicity_classifier(predictions)
    bias = compute_bias_perplexity(stereotype_pairs, model)
    return {'BLEU': bleu, 'ROUGE-1': rouge1, 'ROUGE-L': rougeL, 'Distinct-4': distinct4, 'MAUVE': mauve, 'Entailment': entailment, 'Toxicity': toxicity, 'Bias': bias}

Common pitfalls

• MAUVE scores are reported on an unnormalized scale in the paper (e.g., values > 10 in Table 3), which differs from the standard [0,1] interpretation and may indicate a reporting error or custom scaling.
• Entailment scores rely on a custom fine-tuned NorBERT model trained on the VitaminC dataset, not a standard off-the-shelf NLI model, making direct comparison with other benchmarks difficult.
• Toxicity scores can be artificially low if models generate meaningless characters or words, as noted in the toxicity evaluation section.
• Human evaluation uses only 50 samples per dataset with 3 raters, resulting in low inter-rater agreement for fluency (Fleiss' kappa ~0.60).

Evidence (verbatim from paper)

We aim to comprehensively evaluate our models across various tasks using widely used metrics for NLP tasks, including BLEU (Papineni et al., 2002), ROUGE (Lin, 2004), Distinct (Li et al., 2016), and MAUVE, which is used to assess the generated and human-written text based on their probability distribution differences (Pillutla et al., 2021). Furthermore, following the work of Xie et al. (2023), to measure faithfulness and factual consistency in multi-task learning, we utilize Entailment scores from a fine-tuned NorBERT model trained on the VitaminC dataset (Schuster et al., 2021), which are translated with Google Cloud Translation API.

Citation

@misc{liu2023nlebench,
  title={NLEBench+NorGLM: A Comprehensive Empirical Analysis and Benchmark Dataset for Generative Language Models in Norwegian},
  author={Liu et al. (2023)},
  year={2023},
  note={arXiv:2312.01314}
}

• arXiv: 2312.01314