Indotabvqa Eval

indotabvqa-eval

INDOTABVQA: A Benchmark for Cross-Lingual Table Understanding in Bahasa Indonesia Documents — Gautam et al. (2026) (arXiv:2604.11970, 2026)

What this evaluates

Probes cross-lingual visual question answering on document images containing tables. It tests a model's ability to perform factual lookup, numerical comparison, aggregation, and structural reasoning across Bahasa Indonesia, English, Hindi, and Arabic.

Datasets

• IndoTabVQA — total 1593; splits: train (500), test (1043)

Metrics

• In-Match Accuracy (primary) — range: percent
  • A prediction is correct if the normalized ground truth answer appears as a substring within the normalized predicted answer. Normalization includes lowercasing, removing punctuation, collapsing whitespace, and handling number formatting variations.
• STS Accuracy — range: [0, 1]
  • Cosine similarity between dense vector representations of the predicted and ground truth answers, computed using the paraphrase-multilingual-MiniLM-L12-v2 sentence encoder.

Input / output format

Input: Document image I (PNG/JPEG) containing one or more tables, paired with a natural language question Q in Bahasa Indonesia, English, Hindi, or Arabic.

Output: Short free-form text or numeric value A in the same language as the question.

Scoring recipe

def normalize(text):
    return text.lower().replace('.', '').replace(',', '').replace(' ', ' ').strip()

def score_in_match(pred, gold):
    return 1 if normalize(gold) in normalize(pred) else 0

def score_sts(pred, gold, encoder):
    return cosine_similarity(encoder.encode(pred), encoder.encode(gold))

Common pitfalls

• Strict exact-match evaluation fails to capture correct answers that include conversational filler or extra context, which is why relaxed In-Match is used.
• Models without explicit table bounding boxes struggle significantly with multi-table documents and borderless layouts due to attention dilution.
• Cross-lingual evaluation requires strict language alignment; answers must be generated in the exact query language, not translated.

Evidence (verbatim from paper)

We use a relaxed matching criterion where a prediction is correct if the normalized ground truth answer appears as a substring within the predicted answer. Normalization involves converting text to lowercase, removing punctuation, collapsing whitespace, and handling number formatting variations. This relaxed matching accounts for VLMs that often generate answers with additional context (e.g., ‘if the ground truth is ‘5 tables’, a prediction of ‘There are 5 tables in the document’) would be considered correct. In-Match captures correct answers embedded in longer responses.

Citation

@misc{gautam2026indotabvqa,
  title={INDOTABVQA: A Benchmark for Cross-Lingual Table Understanding in Bahasa Indonesia Documents},
  author={Gautam et al. (2026)},
  year={2026},
  note={arXiv:2604.11970}
}

• arXiv: 2604.11970