Thaiocrbench Eval

thaiocrbench-eval

ThaiOCRBench: A Task-Diverse Benchmark for Vision-Language Understanding in Thai — Nonesung et al. (2025) (arXiv:2511.04479, 2025)

What this evaluates

Evaluates vision-language models on Thai-language text-rich visual tasks, including document parsing, table/chart recognition, handwritten content extraction, and visual question answering. It probes fine-grained text recognition, structural layout understanding, and semantic reasoning in a low-resource, script-complex language setting.

Datasets

• ThaiOCRBench — total 2808; splits: test (2808); repo https://github.com/scb-10x/ThaiOCRBench

Metrics

• Tree Edit Distance (TED) — range: [0, 1]
  • Minimum number of edit operations (insert, delete, substitute nodes) required to transform the predicted hierarchical tree into the reference tree.
• BMFL (primary) — range: [0, 1]
  • Composite metric averaging BLEU, METEOR, F1-score, and Normalized Levenshtein Similarity (NLS) for text generation and recognition tasks.
• F1-score — range: [0, 1]
  • Harmonic mean of precision and recall for entity-level prediction in key information extraction and mapping tasks.
• Average Normalized Levenshtein Similarity (ANLS) — range: [0, 1]
  • 1 - (Normalized Edit Distance), where edit distance is divided by the length of the longer string, used for VQA and text understanding tasks.
• Average score — range: [0, 1]
  • Mean of all task-specific metric scores reported across the 13 categories to serve as an overall performance indicator.

Input / output format

Input: Multimodal input consisting of an image (document, chart, table, or handwritten note) paired with a text prompt/question in Thai. All evaluations are conducted in a zero-shot setting.

Output: Text response in Thai (or code-switched) containing transcribed text, parsed structure (e.g., hierarchical layout for tables), extracted entities, or answers to VQA prompts.

Scoring recipe

def score(prediction, reference, task_type):
    if task_type in ['table_parsing', 'chart_parsing', 'doc_parsing']:
        return tree_edit_distance(prediction, reference)
    elif task_type in ['fine_grained_ocr', 'full_page_ocr', 'handwritten']:
        bleu = compute_bleu(prediction, reference)
        meteor = compute_meteor(prediction, reference)
        f1 = compute_f1(prediction, reference)
        nls = normalized_levenshtein(prediction, reference)
        return (bleu + meteor + f1 + nls) / 4
    elif task_type in ['key_info_extraction', 'key_info_mapping']:
        return f1_score(prediction, reference)
    elif task_type in ['vqa', 'classification', 'text_recognition']:
        return 1 - (levenshtein_distance(prediction, reference) / max(len(prediction), len(reference)))
    return 0

Common pitfalls

• Models frequently exhibit language bias or code-switching (mixing Thai and English) due to training data composition, which penalizes pure Thai evaluation.
• Structural mismatch occurs when models correctly recognize text tokens but fail to preserve hierarchical layout or table boundaries, heavily impacting TED scores.
• Hallucinated content is common in open-source models for fine-grained text and handwritten tasks, leading to misleading token-level metric scores if not manually verified.

Evidence (verbatim from paper)

For tasks involving the reconstruction of document layout and hierarchical content such as Table parsing, Chart parsing, and Document parsing, we employ the Tree Edit Distance (TED) metric, which quantifies structural similarity between predicted and reference outputs. TED is particularly suited to evaluating nested or hierarchical formats where layout consistency is critical. Text Generation and Recognition Tasks. For tasks requiring the transcription or generation of text such as Fine-grained text recognition, Full-page OCR, and Handwritten content extraction, we report multiple complementary metrics to assess both character-level accuracy and linguistic fidelity. These include BLEU, METEOR, F1-score, and Normalized Levenshtein Similarity (NLS). We average these into a single composite metric, referred to as BMFL. Structured Prediction Tasks. For Key information extraction and Key information mapping, we use the F1-score to evaluate precision and recall in entity-level prediction. This metric is appropriate for scenarios where exact field alignment is required and partial matches are penalized. Textual Understanding and Question Answering Tasks. For tasks involving semantic under

Citation

@misc{nonesung2025thaiocrbench,
  title={ThaiOCRBench: A Task-Diverse Benchmark for Vision-Language Understanding in Thai},
  author={Nonesung et al. (2025)},
  year={2025},
  note={arXiv:2511.04479}
}

• arXiv: 2511.04479