Table Structure Recognition Eval

table-structure-recognition-eval

Optimized Table Tokenization for Table Structure Recognition — Lysak et al. (2023) (arXiv:2305.03393, 2023)

What this evaluates

Evaluates table structure recognition (TSR) and cell detection capabilities by comparing two tokenization schemes (OTSL vs. HTML) on transformer-based image-to-sequence models. It measures how well the model predicts table layouts and cell bounding boxes across diverse document types.

Datasets

• PubTabNet — total 395000; splits: test (-1)
• FinTabNet — total 113000; splits: test (-1)
• PubTables-1M — total 1000000; splits: test (-1)

Metrics

• Tree Edit Distance score (TEDs) (primary) — range: [0, 1]
  • Measures the minimum number of edit operations (insert, delete, substitute) required to transform the predicted table structure tree into the ground truth tree. Reported separately for simple tables, complex tables (with cell spans), and all tables combined.
• Mean Average Precision (mAP) with 0.75 IOU threshold — range: [0, 1]
  • Computes the average precision of detected cell bounding boxes where the Intersection Over Union (IoU) with the ground truth bounding box is at least 0.75.

Input / output format

Input: Image of a table (document page crop).

Output: OTSL or HTML token sequence representing the table structure, plus bounding box coordinates for each detected table cell.

Scoring recipe

def compute_metrics(pred_otsl, pred_boxes, gt_html, gt_boxes):
    # Convert predicted OTSL sequence back to HTML for structure comparison
    pred_html = convert_otsl_to_html(pred_otsl)
    # Compute Tree Edit Distance score
    ted_score = tree_edit_distance_score(pred_html, gt_html)
    # Compute mAP@0.75 for bounding boxes
    mAP = mean_average_precision(pred_boxes, gt_boxes, iou_threshold=0.75)
    return ted_score, mAP

Common pitfalls

• TEDs must be computed after converting the predicted OTSL sequence back to HTML format; it cannot be computed directly on OTSL tokens.
• TEDs is reported separately for simple tables, complex tables (with cell spans), and all tables combined, so results are not a single scalar.
• Inference time measurements are hardware-specific (single-core AMD EPYC 7763 @2.45 GHz) and not directly comparable across different setups.

Evidence (verbatim from paper)

We rely on standard metrics such as Tree Edit Distance score (TEDs) for table structure prediction, and Mean Average Precision (mAP) with 0.75 Intersection Over Union (IOU) threshold for the bounding-box predictions of table cells. The predicted OTSL structures were converted back to HTML format in order to compute the TED score.

Citation

@misc{lysak2023optimized,
  title={Optimized Table Tokenization for Table Structure Recognition},
  author={Lysak et al. (2023)},
  year={2023},
  note={arXiv:2305.03393}
}

• arXiv: 2305.03393