Pubtables V2 Eval

pubtables-v2-eval

PubTables-v2: A new large-scale dataset for full-page and multi-page table extraction — Smock et al. (2025) (arXiv:2512.10888, 2025)

What this evaluates

Evaluates vision-language and specialized models on page-level and document-level table structure recognition, requiring them to extract hierarchical table structures from full pages or multi-page documents. It also probes cross-page table continuation prediction by testing whether models can identify when a table spans two contiguous pages.

Datasets

• PubTables-v2 — total 548414; splits: train (-1), test (-1)

Metrics

• GriTS_Top (primary) — range: [0, 1]
  • Pseudo F1-score for cells that enforces global consistency across all rows and columns, independent of output format. Evaluated on top-level table structure.
• GriTS_Con — range: [0, 1]
  • Same as GriTS_Top but evaluated on cell content/continuation consistency.
• Acc_Top — range: [0, 1]
  • Percentage of tables where predicted top-level structure exactly matches ground truth.
• Acc_Con — range: [0, 1]
  • Percentage of tables where predicted cell content exactly matches ground truth.
• F1 — range: [0, 1]
  • Harmonic mean of precision and recall for binary cross-page table continuation classification.

Input / output format

Input: Single page image, concatenated pair of contiguous page images, or full document (multiple pages).

Output: Table structure representation (e.g., HTML/graph) for each detected table, or binary label (continues/does not continue) for cross-page task.

Scoring recipe

def evaluate_page_level(pred_tables, gt_tables):
    # Match predictions to ground truth using Hungarian algorithm to maximize total GriTS
    matched_pairs = hungarian_match(pred_tables, gt_tables, score_fn=gripts_score)
    gripts_top = sum(gripts_score(p, g) for p, g in matched_pairs) / len(gt_tables)
    exact_match = sum(1 for p, g in matched_pairs if p == g) / len(gt_tables)
    return gripts_top, exact_match

def evaluate_cross_page(pred_labels, gt_labels):
    tp = sum(1 for p, g in zip(pred_labels, gt_labels) if p == 1 and g == 1)
    fp = sum(1 for p, g in zip(pred_labels, gt_labels) if p == 1 and g == 0)
    fn = sum(1 for p, g in zip(pred_labels, gt_labels) if p == 0 and g == 1)
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    return f1

Common pitfalls

• GriTS evaluates global table structure consistency rather than isolated cell overlaps, making it sensitive to row/column alignment errors.
• Page-level evaluation automatically matches predicted and ground-truth tables via the Hungarian algorithm, so models do not need explicit table correspondence labels.
• Cross-page negative samples are strictly drawn from contiguous pages within the same document to prevent models from exploiting stylistic shortcuts instead of learning visual continuation cues.

Evidence (verbatim from paper)

To measure TSR performance, we use GriTS metrics [smock2023grits] and exact match accuracy. GriTS can be interpreted as a pseudo F1-score for cells. One advantage of GriTS is that it enforces global consistency on the result across all rows and columns and is not tied to a particular format for representing the table. Exact match accuracy (Acc) is the percentage of tables for which the predictions and ground truth match exactly.

Citation

@misc{smock2025pubtablesv2,
  title={PubTables-v2: A new large-scale dataset for full-page and multi-page table extraction},
  author={Smock et al. (2025)},
  year={2025},
  note={arXiv:2512.10888}
}

• arXiv: 2512.10888