Banglaberse Eval

banglaberse-eval

Many Dialects, Many Languages, One Cultural Lens: Evaluating Multilingual VLMs for Bengali Culture Understanding Across Historically Linked Languages and Regional Dialects — Sayeedi et al. (2026) (arXiv:2603.21165, 2026)

What this evaluates

Probes multilingual vision-language models' ability to understand and reason about Bengali cultural concepts across regional dialects and historically linked languages. It measures how well models maintain cultural grounding when faced with linguistic variation, testing both visual captioning and structured question-answering capabilities.

Datasets

• BanglaVerse — total 1152; splits: test (1152)

Metrics

• accuracy (%) (primary) — range: percent
  • Percentage of VQA instances where the model's selected answer option exactly matches the ground truth label.
• BERTScore-F1 — range: [0, 1]
  • F1 score computed using contextual embeddings from a BERT model to measure semantic similarity between generated and reference captions.
• LLM-as-a-Judge — range: [0, 1]
  • Holistic quality score from 0 to 1 generated by Gemini-2.5-Flash, evaluating captions across Relevance, Clarity, Conciseness, and Creativity.

Input / output format

Input: A single image paired with a text prompt (either a visual question or a captioning instruction) provided in one of four languages (Bangla, English, Hindi, Urdu) and one of five Bangla dialects.

Output: For captioning: a natural language description of the image. For VQA: a single selected answer option from the provided choices.

Scoring recipe

def compute_metrics(predictions, golds, references):
    acc = sum(1 for p, g in zip(predictions, golds) if p == g) / len(golds)
    bert_f1 = bertscore.score(predictions, references, lang='bn')[1].mean()
    judge_scores = [gemini_2_5_flash.evaluate(p, r, dims=['Relevance','Clarity','Conciseness','Creativity']) for p, r in zip(predictions, references)]
    llm_score = sum(judge_scores) / len(judge_scores)
    return {'accuracy': acc, 'bertscore_f1': bert_f1, 'llm_judge': llm_score}

Common pitfalls

• Assuming standard Bangla performance generalizes to regional dialects; the benchmark explicitly shows significant drops under dialectal variation.
• Relying solely on lexical overlap metrics for captioning; the protocol requires semantic similarity (BERTScore-F1) and holistic LLM-as-a-Judge scoring to capture cultural nuance.
• Ignoring decoding temperature settings; the protocol fixes temperature at 0.1 for deterministic outputs, which affects reproducibility.

Evidence (verbatim from paper)

For image captioning, we report BERTScore-F1 (Zhang et al., 2020) to measure semantic similarity between generated and reference captions, and LLM-as-a-Judge (Gu et al., 2024) scores obtained with Gemini-2.5-Flash to capture overall caption quality beyond surface-level lexical overlap. Specifically, the judge is instructed to evaluate the captions across four dimensions: Relevance, Clarity, Conciseness, and Creativity, to compute a final holistic score ranging from 0 to 1. For visual question answering, we use accuracy (%), defined as the percentage of questions for which the model selects the correct answer option.

Citation

@misc{sayeedi2026banglaberse,
  title={Many Dialects, Many Languages, One Cultural Lens: Evaluating Multilingual VLMs for Bengali Culture Understanding Across Historically Linked Languages and Regional Dialects},
  author={Sayeedi et al. (2026)},
  year={2026},
  note={arXiv:2603.21165}
}

• arXiv: 2603.21165