Multilingual Csr Eval

multilingual-csr-eval

Common Sense Beyond English: Evaluating and Improving Multilingual Language Models for Commonsense Reasoning — Lin et al. (2021) (arXiv:2106.06937, 2021)

What this evaluates

Evaluates multilingual language models on cross-lingual commonsense reasoning and plausibility. It probes whether models can rank correct assertions and answer multiple-choice questions across 11+ languages.

Datasets

• MickeyProbe — total 561000; splits: test (-1)
• X-CODAH — total ?; splits: test (-1)
• X-CSQA — total ?; splits: test (-1)

Metrics

• Hit@1 Accuracy — range: percent
  • Percentage of instances where the correct assertion is ranked exactly first.
• Hit@2 Accuracy — range: percent
  • Percentage of instances where the correct assertion is ranked within the top 2.
• Accuracy (primary) — range: percent
  • Percentage of correctly answered multiple-choice questions or ranked assertions.

Input / output format

Input: For MickeyProbe: a context sentence and a set of candidate assertions in a target language. For X-CODAH/X-CSQA: a question/context and multiple choice options in a target language.

Output: For MickeyProbe: a ranked list of assertions. For X-CODAH/X-CSQA: a single selected option label.

Scoring recipe

def compute_hit_at_k(predictions, gold, k):
    return int(gold in predictions[:k])

def compute_accuracy(predictions, gold):
    return int(predictions == gold)

Common pitfalls

• Hit@2 is explicitly noted as more flexible than Hit@1, so reporting only Hit@1 underestimates performance.
• X-CODAH results are broken down by linguistic categories (Idioms, Neg., Poly., Ref., Quant., Others), which vary significantly in difficulty across languages.
• Fine-tuning hyperparameters (learning rate, epochs, batch size) differ significantly between models and datasets, requiring careful reproduction from Table 7.

Evidence (verbatim from paper)

Table 5 shows the Hit@2 Accuracy of the five MLLMs for the MickeyProbe. Hit@2 Accuracy evaluates whether the models can rank the correct assertion within top 2. Unlike Hit@1 which only accepts best predictions, Hit@2 is more flexible. Thus, the performances in Hit@2 increase compared to the ones in Hit@1. We can see that the discrepancies across languages still exist.

Citation

@misc{lin2021commonsensebeyondenglish,
  title={Common Sense Beyond English: Evaluating and Improving Multilingual Language Models for Commonsense Reasoning},
  author={Lin et al. (2021)},
  year={2021},
  note={arXiv:2106.06937}
}

• arXiv: 2106.06937