Merit Token Classification Eval

merit-token-classification-eval

The MERIT Dataset: Modelling and Efficiently Rendering Interpretable Transcripts — de Rodrigo et al. (2024) (arXiv:2409.00447, 2024)

What this evaluates

Evaluates document understanding models on token classification tasks using structured school transcripts. It probes the model's ability to accurately label tokens based on layout and text features across English and Spanish documents with varying templates and layouts.

Datasets

• MERIT — total ?; splits: train/val (-1), test (-1)

Metrics

• Token Classification (primary) — range: [0, 1]
  • Standard token-level classification accuracy or F1 score. Exact variant (micro vs macro) is not specified in the provided text.

Input / output format

Input: Document images with original words and bounding boxes provided under a perfect OCR hypothesis (no OCR errors).

Output: Token-level classification labels for each word/bounding box in the document.

Scoring recipe

def compute_token_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    total = len(gold)
    if total == 0: return 0.0
    return correct / total

Common pitfalls

• Perfect OCR assumption: models receive ground-truth words and bounding boxes, ignoring real-world OCR errors that typically degrade performance.
• Layout restriction: test set only includes samples with 'Model A' and 'Model B' layouts, excluding other template variations.
• Language split: English models are trained/evaluated on the English subset and Spanish models on the Spanish subset, limiting cross-lingual evaluation claims.

Evidence (verbatim from paper)

We train LayoutLMv2 [34], LayoutLMv3 [16], and LayoutXLM [5] on the Token Classification task, which is the primary niche of the MERIT Dataset. The samples used to train LayoutLMv2 and v3 are in English, while those for training LayoutXLM are in Spanish. ... we use samples from 5 schools to train and validate the model, reserving the remaining samples from the other two schools for testing. We decide to include only samples with Model A and Model B layouts as testing subsets. ... we work under the perfect OCR hypothesis, i.e., when testing, the model receives the original words and bounding boxes from the dataset, so no OCR induces downstream errors. We present the benchmark results in Table 8.

Citation

@misc{derodriguez2024merit,
  title={The MERIT Dataset: Modelling and Efficiently Rendering Interpretable Transcripts},
  author={de Rodrigo et al. (2024)},
  year={2024},
  note={arXiv:2409.00447}
}

• arXiv: 2409.00447