Hintel Alignbench Eval

hintel-alignbench-eval

HinTel-AlignBench: A Framework and Benchmark for Hindi-Telugu with English-Aligned Samples — Chigrupaatii et al. (2025) (arXiv:2511.15183, 2025)

What this evaluates

Evaluates multimodal large language models on vision-language tasks in Hindi and Telugu, measuring performance regression when transitioning from English to these Indian languages. It probes native-language visual question answering, mathematical reasoning, and multiple-choice comprehension across STEM and cultural domains.

Datasets

• HinTel-AlignBench — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly answered questions. For VQAv2/CLEVR-Math, uses a hybrid strategy: exact match against gold answers first; if it fails, a GPT-4.1 model judges semantic equivalence. For JEE subsets, uses regex-based answer extraction followed by rule-based scoring. For MCQ subsets, direct match against the correct option.

Input / output format

Input: Image(s) paired with a question or instruction in Hindi, Telugu, or English.

Output: Text response containing the answer (single word, short phrase, multiple-choice option, or numeric value).

Scoring recipe

def compute_accuracy(predictions, golds, dataset_type):
    correct = 0
    for pred, gold in zip(predictions, golds):
        if dataset_type in ["VQAv2", "CLEVR-Math"]:
            if pred.strip().lower() == gold.strip().lower():
                correct += 1
            elif gpt41_judges_semantic_equivalence(pred, gold):
                correct += 1
        elif dataset_type == "JEE":
            pred_ans = regex_extract(pred)
            if rule_based_score(pred_ans, gold) == 1:
                correct += 1
        else: # MCQ
            if pred.strip().lower() == gold.strip().lower():
                correct += 1
    return (correct / len(golds)) * 100

Common pitfalls

• Exact match evaluation penalizes semantically correct answers that differ in surface form (e.g., 'yes' vs 'yes, it is').
• Regex-based parsing for JEE-style numeric/MCQ answers may fail if models output verbose reasoning instead of direct answers.
• Evaluating models without explicit support for Hindi/Telugu inflates or deflates results and misrepresents true multilingual capability.

Evidence (verbatim from paper)

We use accuracy as the evaluation metric for all of these sets. We extract the answers using regex-based parsing, and report the overall accuracy across all the questions. For VQAv2 and CLEVR-Math subsets, the answers are either a single word or short phrases. We adopt a hybrid evaluation strategy. We first evaluate a sample using exact match. Our exact match evaluation is built using the official VQA evaluation script (Goyal et al., 2017a), with the functionalities also extended to Hindi and Telugu. While exact match is strict and interpretable, it may penalize correct answers with minor surface-level variations (e.g., "yes" and "yes, it is", synonyms, etc.). If exact match fails for a sample, we evaluate that sample using "gpt-4.1-2025-04-14" (OpenAI, 2025) as desc in (?). This two-step approach enables both high precision and flexibility, especially in cases in which answers may vary in form but not meaning.

Citation

@misc{chigrupaatii2025hintelalignbench,
  title={HinTel-AlignBench: A Framework and Benchmark for Hindi-Telugu with English-Aligned Samples},
  author={Chigrupaatii et al. (2025)},
  year={2025},
  note={arXiv:2511.15183}
}

• arXiv: 2511.15183