Milu Eval

milu-eval

MILU: A Multi-task Indic Language Understanding Benchmark — Verma et al. (2024) (arXiv:2411.02538, 2024)

What this evaluates

Evaluates large language models on multi-task understanding across 11 Indic languages, covering 8 domains and 41 subjects. It probes cultural knowledge, region-specific exam data, and multilingual reasoning capabilities.

Datasets

• MILU — total ?; splits: validation (9000); repo https://github.com/AI4Bharat/MILU

Metrics

• accuracy (primary) — range: [0, 1]
  • Proportion of correctly predicted answers. For multiple-choice questions, the answer with the highest conditional log-likelihood is selected. For generative API models, the parsed JSON response is compared against the gold answer.

Input / output format

Input: Question text, optionally preceded by few-shot examples from a validation set. For API models, prompts request structured JSON output.

Output: For non-API models: the selected answer string (multiple choice). For API models: a structured JSON object containing the predicted response.

Scoring recipe

def compute_accuracy(predictions, golds):
    correct = 0
    for pred, gold in zip(predictions, golds):
        # pred is either the argmax log-likelihood answer or parsed JSON string
        if pred == gold:
            correct += 1
    return correct / len(golds)

Common pitfalls

• Log-likelihood scoring for non-API models differs from generative scoring for API models, making direct comparison sensitive to the evaluation setup.
• Few-shot examples are drawn from a separate ~9,000 question validation set, not the test set itself.
• API models are only evaluated in zero-shot due to cost, limiting few-shot analysis for proprietary models.

Evidence (verbatim from paper)

For multiple choice questions, given $k$ possible answer strings, we select the answer string $(a_{i})$ with the highest conditional log probability, i.e., $argmax(logP(a_{1}|x),...,logP(a_{k}|x))$. The API-based models are evaluated using the generative approach... explicitly prompt these models to generate the correct response in a structured JSON format. It reveals that open multilingual models outperform language-specific ones, with GPT-4o achieving 74% average accuracy.

Citation

@misc{verma2024milu,
  title={MILU: A Multi-task Indic Language Understanding Benchmark},
  author={Verma et al. (2024)},
  year={2024},
  note={arXiv:2411.02538}
}

• arXiv: 2411.02538