Indicgenbench Eval

indicgenbench-eval

IndicGenBench: A Multilingual Benchmark to Evaluate Generation Capabilities of LLMs on Indic Languages — Singh et al. (2024) (arXiv:2404.16816, 2024)

What this evaluates

Evaluates the multilingual and cross-lingual generation capabilities of LLMs across 29 Indic languages, covering summarization, machine translation, and question answering. It probes how model performance scales with language resourcedness, in-context learning, and fine-tuning.

Datasets

• CrossSum-In — total ?; splits: test (-1)
• Flores-In — total ?; splits: test (-1)
• XQuAD-In — total ?; splits: test (-1)
• XorQA-In — total ?; splits: test (-1)

Metrics

• Character-F1 (ChrF) (primary) — range: [0, 1]
  • Computes n-gram overlap between reference and hypothesis at the character level, combining precision and recall into an F1 score. Used for summarization and translation tasks.
• SQuAD-style Token-F1 (primary) — range: [0, 1]
  • Computes token-level precision and recall between the predicted answer span and the gold answer, then returns their harmonic mean. Used for QA tasks.

Input / output format

Input: Task-specific prompts in a few-shot or zero-shot setting. For summarization/translation: source text in one language, target in another. For QA: question and context passage in the target language or English.

Output: Generated text: a summary, a translated sentence, or a short answer span.

Scoring recipe

def compute_chrF(hypothesis, reference):
    # Character n-gram overlap (typically n=6)
    return chrF_score(hypothesis, reference)

def compute_token_f1(pred_span, gold_span):
    pred_tokens = set(pred_span.split())
    gold_tokens = set(gold_span.split())
    if not pred_tokens or not gold_tokens:
        return 0.0
    precision = len(pred_tokens & gold_tokens) / len(pred_tokens)
    recall = len(pred_tokens & gold_tokens) / len(gold_tokens)
    return 2 * precision * recall / (precision + recall)

Common pitfalls

• Token-level metrics like ROUGE and BLEU are unreliable for low-resource Indic languages; Character-F1 must be used instead.
• Translation performance differs significantly by direction (en→xx vs xx→en), with generation often lagging behind understanding.
• High token fertility in low-resource languages reduces the number of in-context examples that fit in the context window, artificially lowering few-shot performance.

Evidence (verbatim from paper)

For the cross-lingual summarization and translation tasks, CrossSum-In and Flores-In, we report Character-F1 (ChrF) metric since token-level metrics like ROUGE and BLEU are not reliable for low-resource languages. To stay consistent with existing literature on QA tasks, we report SQuAD-style Token-F1 on our XQuAD-In and XorQA-In QA tasks.

Citation

@misc{singh2024indicgenbench,
  title={IndicGenBench: A Multilingual Benchmark to Evaluate Generation Capabilities of LLMs on Indic Languages},
  author={Singh et al. (2024)},
  year={2024},
  note={arXiv:2404.16816}
}

• arXiv: 2404.16816