Sea Vision Eval

sea-vision-eval

SEA-Vision: A Multilingual Benchmark for Comprehensive Document and Scene Text Understanding in Southeast Asia — Yue et al. (2026) (arXiv:2603.15409, 2026)

What this evaluates

Evaluates multimodal language models on document parsing and text-centric visual question answering across 11 Southeast Asian languages. Probes the models' ability to extract structured information from complex documents and answer questions based on visual-textual alignment in low-resource scripts.

Datasets

• SEA-Vision — total 22730; splits: test (-1)

Metrics

• answer accuracy (primary) — range: percent
  • Percentage of questions where the model's raw textual output exactly matches the ground truth answer after standard normalization. Higher values indicate better performance.
• NED — range: [0, 1]
  • Normalized Edit Distance measuring the structural and textual alignment between predicted and ground-truth document parsing outputs. Lower values indicate better performance.

Input / output format

Input: An image of a document or scene text page, optionally accompanied by a natural language question for the TEC-VQA task.

Output: For document parsing: a structured representation of OCR, tables, formulas, and reading order. For TEC-VQA: a raw textual answer string.

Scoring recipe

def compute_accuracy(predictions, golds):
    correct = sum(1 for p, g in zip(predictions, golds) if normalize(p) == normalize(g))
    return (correct / len(golds)) * 100

def compute_ned(pred_structure, gold_structure):
    dist = edit_distance(pred_structure, gold_structure)
    max_len = max(len(pred_structure), len(gold_structure))
    return dist / max_len if max_len > 0 else 0.0

Common pitfalls

• Models are evaluated strictly in a zero-shot setting using a unified prompt template; comparing against fine-tuned or task-specific baselines violates the evaluation protocol.
• Raw textual outputs are used directly for scoring without post-processing or parsing, which may unfairly penalize models that output structured formats (e.g., JSON, Markdown) differently than the ground truth.
• NED is a structural metric where lower is better, contrary to standard accuracy metrics where higher is better, which can lead to misinterpretation of results.

Evidence (verbatim from paper)

For document parsing, we report NED↓ where lower is better. For TEC-VQA (Text-centric Visual Question Answering), we report answer accuracy↑ where higher is better. To ensure a fair comparison, all models are assessed in a unified, zero-shot setting using the same prompt template and the answer accuracy is computed directly from their raw textual outputs without any task-specific fine-tuning.

Citation

@misc{yue2026seavision,
  title={SEA-Vision: A Multilingual Benchmark for Comprehensive Document and Scene Text Understanding in Southeast Asia},
  author={Yue et al. (2026)},
  year={2026},
  note={arXiv:2603.15409}
}

• arXiv: 2603.15409