Sib200 Xlt Eval

sib200-xlt-eval

Cross-Prompt Encoder for Low-Performing Languages — Mikaberidze et al. (2025) (arXiv:2508.10352, 2025)

What this evaluates

This evaluation probes a model's ability to perform zero-shot and fully-supervised cross-lingual text classification across typologically diverse languages. It specifically measures how well parameter-efficient soft prompt tuning methods transfer knowledge from high-resource source languages to low-performing or unseen target languages without language-specific fine-tuning.

Datasets

• SIB-200 — total ?; splits: train (-1), test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • The proportion of correctly predicted class labels out of the total number of instances. Calculated as (number of correct predictions) / (total number of predictions).

Input / output format

Input: Raw text samples in one of ~200 target languages, processed by a frozen XLM-R large encoder with a lightweight classification head.

Output: A single predicted topic class label per text instance.

Scoring recipe

def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return correct / len(gold_labels)

Common pitfalls

• Confusing zero-shot cross-lingual transfer (ZS-XLT), which involves training on source languages and evaluating on targets, with pure zero-shot prompting of LLMs that receive no task-specific tuning.
• Using shared or language-specific verbalizers instead of a lightweight classification head, which introduces cross-lingual bias or fails for languages outside the tokenizer's vocabulary.
• Aggregating performance across language groups (Seen, Unseen, Low-Performing) without accounting for their strict subset relationships, which can mask transfer dynamics.

Evidence (verbatim from paper)

We evaluate on the SIB-200 benchmark, a multilingual topic classification dataset covering 200 typologically diverse languages. ... Table 1: ZS-XLT performance (accuracy) across different target groups.

Citation

@misc{mikaberidze2025crossprompt,
  title={Cross-Prompt Encoder for Low-Performing Languages},
  author={Mikaberidze et al. (2025)},
  year={2025},
  note={arXiv:2508.10352}
}

• arXiv: 2508.10352