Flue Eval

flue-eval

BARThez: a Skilled Pretrained French Sequence-to-Sequence Model — Moussa Kamal Eddine et al. (2020) (arXiv:2010.12321, 2020)

What this evaluates

Probes French sequence classification capabilities across sentiment analysis, paraphrase identification, and natural language inference.

Datasets

• CLS — total 6000; splits: train (4800), test (2000)
• PAWSX — total 53378; splits: train (49401), dev (1992), test (1985)
• XNLI — total 400202; splits: train (392702), dev (2490), test (5010)

Metrics

• Accuracy (primary) — range: [0, 1]
  • Proportion of correctly classified instances over the total number of instances.

Input / output format

Input: Single sentence or sentence pair (separated by a special token), fed to both encoder and decoder; last token representation used for classification.

Output: Class label (e.g., positive/negative, equivalent/not, entailment/contradiction/neutral)

Scoring recipe

correct = 0; total = 0
for doc in dataset:
    seq = format_pair(doc.s1, doc.s2)
    logits = model(seq)
    pred = argmax(logits)
    if pred == doc.label: correct += 1
    total += 1
accuracy = correct / total

Common pitfalls

• BART-based models require feeding input to both encoder and decoder and using the last token's representation, unlike standard BERT classification.
• Results are averaged over 3 runs with different seeds; reporting single runs may misrepresent performance variance.
• CLS dataset is split into 3 product categories; evaluation should be averaged or reported per category.

Evidence (verbatim from paper)

We evaluate the different models on five discriminative tasks from the FLUE benchmark (Le et al., 2019), the French equivalent of GLUE (Wang et al., 2018). ... Table 10 reports the test set accuracies.

Citation

@misc{moussa2020barthez,
  title={BARThez: a Skilled Pretrained French Sequence-to-Sequence Model},
  author={Moussa Kamal Eddine et al. (2020)},
  year={2020},
  note={arXiv:2010.12321}
}

• arXiv: 2010.12321