Miners Eval

miners-eval

MINERS: Multilingual Language Models as Semantic Retrievers — Winata et al. (2024) (arXiv:2406.07424, 2024)

What this evaluates

Evaluates multilingual language models as semantic retrievers across 200+ languages without fine-tuning. It probes bitext mining, retrieval-augmented classification, and in-context learning classification to assess cross-lingual and code-switching capabilities.

Datasets

• MINERS (includes NusaX) — total ?; splits: test (-1); repo https://github.com/gentaiscool/miners

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly predicted class labels. In retrieval-based classification, final predictions are aggregated via majority voting over the retrieved context examples.
• precision — range: percent
  • Proportion of correctly predicted positive instances among all instances predicted as positive in the classification tasks.

Input / output format

Input: Bitext: sentence pairs in different languages. Retrieval-based classification: a query sentence and a candidate document pool. ICL classification: a query sentence plus a few-shot context of retrieved examples with ground-truth labels.

Output: Bitext: ranked list of retrieved sentence pairs. Classification: predicted class label (determined by majority voting over retrieved examples or direct generation by the LM).

Scoring recipe

def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return correct / len(gold_labels) * 100

def majority_vote(retrieved_labels):
    from collections import Counter
    return Counter(retrieved_labels).most_common(1)[0][0]

Common pitfalls

• Performance is highly sensitive to the number of retrieved documents (k), with larger k consistently boosting accuracy via majority voting.
• Code-switching (CS) and cross-lingual code-switching (XL CS) settings are significantly more challenging than monolingual or standard cross-lingual settings.
• Commercial API models may benefit from prior exposure to benchmark datasets, potentially inflating their reported performance.

Evidence (verbatim from paper)

The inclusion of few-shot context significantly improves the generative LM’s precision in predicting class labels, leading to enhancements. In retrieval-based classification, a larger $k$ offers more contextual examples, leading to more precise label predictions through majority voting.

Citation

@misc{winata2024miners,
  title={MINERS: Multilingual Language Models as Semantic Retrievers},
  author={Winata et al. (2024)},
  year={2024},
  note={arXiv:2406.07424}
}

• arXiv: 2406.07424