Indicxtreme Eval

indicxtreme-eval

Towards Leaving No Indic Language Behind: Building Monolingual Corpora, Benchmark and Models for Indic Languages — Doddapaneni et al. (2022) (arXiv:2212.05409, 2022)

What this evaluates

Evaluates zero-shot cross-lingual transfer capabilities of language models on low-resource Indic languages by testing performance on nine natural language understanding tasks after training exclusively on English data.

Datasets

• IndicXTREME — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• task-specific accuracy/F1 (primary) — range: [0, 1]
  • Metric varies by task: accuracy for classification, F1 for sequence labeling, exact match for QA. All normalized to [0, 1].

Input / output format

Input: Indic language text instances from the test set, processed by a model fine-tuned on the corresponding English training and development sets.

Output: Task-specific prediction (e.g., class label, entity span, or generated answer).

Scoring recipe

def score(predictions, golds, task_type):
    if task_type == 'classification':
        return sum(p == g for p, g in zip(predictions, golds)) / len(golds)
    elif task_type == 'ner':
        return f1_score(golds, predictions, average='macro')
    elif task_type == 'qa':
        return sum(p == g for p, g in zip(predictions, golds)) / len(golds)
    else:
        return similarity_score(predictions, golds)

Common pitfalls

• Zero-shot evaluation strictly uses English for fine-tuning; using Indic language data for training violates the protocol.
• Hyperparameters are tuned independently per task for all baselines to ensure fairness, rather than using a single fixed set across tasks.
• Retrieval task bypasses fine-tuning entirely and uses mean-pooled sentence embeddings directly from the pretrained model.

Evidence (verbatim from paper)

We perform zero-shot evaluation by fine-tuning the model on English and testing on the available Indic test sets. The best configuration of the model is chosen based on its performance on the English development set. ... Models are fine-tuned for every task except for the retrieval task, where we directly use the mean pooled sentence representation from the last layer of the pretrained models.

Citation

@misc{doddapaneni2022indicxtreme,
  title={Towards Leaving No Indic Language Behind: Building Monolingual Corpora, Benchmark and Models for Indic Languages},
  author={Doddapaneni et al. (2022)},
  year={2022},
  note={arXiv:2212.05409}
}

• arXiv: 2212.05409