Frenchbench Eval

frenchbench-eval

CroissantLLM: A Truly Bilingual French-English Language Model — Faysse et al. (2024) (arXiv:2402.00786, 2024)

What this evaluates

Evaluates bilingual French-English language understanding, cultural knowledge, and generation capabilities of LLMs across classification and open-ended tasks. Probes the model's ability to perform few-shot reasoning, factual recall, and text generation in both languages.

Datasets

• FrenchBench — total ?; splits: test (-1); repo https://github.com/CoderPat/croissant-llm-training
• English Benchmarks — total ?; splits: test (-1); repo https://github.com/CoderPat/croissant-llm-training

Metrics

• accuracy (primary) — range: [0, 1]
  • Fraction of correctly predicted labels over total instances. Computed as the mean accuracy across all classification tasks in the benchmark.
• ROUGE-1 — range: [0, 1]
  • Unigram overlap F1 score between the generated text and the reference text. Calculated as 2 * (precision * recall) / (precision + recall).

Input / output format

Input: 5-shot prompt containing a task description, five example question-answer pairs, and the target query. For classification tasks, options are provided; for generation tasks, open-ended questions are given.

Output: Model generates a sequence of tokens. For classification, the predicted option label or text. For generation, free-form text response.

Scoring recipe

def compute_metrics(predictions, golds):
    acc_scores = []
    rouge_scores = []
    for pred, gold in zip(predictions, golds):
        if gold in ['A', 'B', 'C', 'D']:
            acc_scores.append(1.0 if pred.strip() == gold else 0.0)
        else:
            rouge_scores.append(rouge1_score(gold, pred))
    return {'accuracy': sum(acc_scores)/len(acc_scores) if acc_scores else 0,
            'rouge1': sum(rouge_scores)/len(rouge_scores) if rouge_scores else 0}

Common pitfalls

• Evaluating base models in few-shot (5-shot) settings rather than zero-shot, which inflates performance on instruction-following and reasoning tasks.
• Using greedy decoding for translation and generation benchmarks instead of beam search or temperature sampling, which may underperform on open-ended generation.
• Assuming ROUGE-1 is sufficient for all generation tasks; the paper notes heuristic metrics may be insufficient and uses GPT-4 as a judge for some tasks.

Evidence (verbatim from paper)

We score results obtained in 5-shot settings with ROUGE-1. As heuristic-based metrics are often insufficient to capture the diversity of possible answers (Faysse et al., [2023]), we also score predictions using GPT4 as a judge, and confirm the ROUGE1 metric is well suited for this task given the closed and short nature of the answers.

Citation

@misc{faysse2024croissantllm,
  title={CroissantLLM: A Truly Bilingual French-English Language Model},
  author={Faysse et al. (2024)},
  year={2024},
  note={arXiv:2402.00786}
}

• arXiv: 2402.00786