Tablex Eval

tablex-eval

TabLeX: A Benchmark Dataset for Structure and Content Information Extraction from Scientific Tables — Desai et al. (2021) (arXiv:2105.06400, 2021)

What this evaluates

Evaluates deep learning models on table structure recognition (TSR) and table content recognition (TCR) by predicting LaTeX token sequences from tabular images. It probes the model's ability to accurately reconstruct table layouts and textual content under varying aspect ratios and sequence lengths.

Datasets

• TabLeX — total ?; splits: TCD-250 (-1), TCD-500 (-1), TSD-250 (-1), TSD-500 (-1)

Metrics

• EMA (primary) — range: percent
  • Exact Match Accuracy (EMA) percentage. Measures the proportion of instances where the predicted LaTeX token sequence exactly matches the ground truth sequence.
• BLEU — range: percent
  • Standard corpus-level BLEU score. Measures n-gram precision between predicted and ground truth token sequences, typically with a geometric mean of 1- to 4-gram overlaps.
• WER — range: percent
  • Word Error Rate (WER) percentage. Computes the minimum number of insertions, deletions, and substitutions of tokens required to transform the prediction into the ground truth, normalized by the ground truth length.

Input / output format

Input: Image of a scientific table (with either conserved or fixed aspect ratio).

Output: A sequence of LaTeX tokens representing the table structure and/or content.

Scoring recipe

def compute_metrics(preds, golds):
    ema = sum(1 for p, g in zip(preds, golds) if p == g) / len(preds) * 100
    bleu = corpus_bleu([[g.split()] for g in golds], [p.split() for p in preds]) * 100
    wer = sum(edit_distance(p, g) / len(g) for p, g in zip(preds, golds)) / len(preds) * 100
    return {'EMA': ema, 'BLEU': bleu, 'WER': wer}

Common pitfalls

• Failing to strip curly braces ('{', '}') and dollar signs ('$') from both predictions and ground truth before computing EMA, which artificially deflates scores for content extraction (TCR).
• Ignoring aspect ratio variations; models perform significantly worse on 'conserved' aspect ratio images compared to 'fixed' aspect ratio images.
• Not accounting for sequence length degradation; longer sequences (e.g., TCD-500 vs TCD-250) cause a sharp drop in EMA scores.

Evidence (verbatim from paper)

Table 3 illustrates that higher sequence length significantly degrades the EMA score for both the TSR and TCR tasks. For TCD-250 and TCD-500, high BLEU scores ( >90 ) suggest that the model can predict a large chunk of IATEX content information correctly. However, the errors are higher for images with a conserved aspect ratio than with a fixed aspect ratio, which is confirmed by comparing their BLEU score and WER metrics. Similarly, for TSD-250 and TSD-500, high EMA and lower WER suggest that the model can correctly identify structure information for most of the tables. TCD yields a lower EMA score than TSD. We attribute this to several reasons. One of the reasons is that the TCR model fails to predict some of the curly braces ('{'} and '}') and dollar ('$') tokens in the predictions. After removing curly braces and dollar tokens from the ground truth and predictions, EMA scores for conserved and fixed aspect ratio images in TCD-250 increased to 68.78% and 75.33%, respectively.

Citation

@misc{desai2021tablex,
  title={TabLeX: A Benchmark Dataset for Structure and Content Information Extraction from Scientific Tables},
  author={Desai et al. (2021)},
  year={2021},
  note={arXiv:2105.06400}
}

• arXiv: 2105.06400