Pharmaship Eval

pharmaship-eval

PharmaShip: An Entity-Centric, Reading-Order-Supervised Benchmark for Chinese Pharmaceutical Shipping Documents — Xie et al. (2025) (arXiv:2512.23714, 2025)

What this evaluates

Evaluates layout-aware document understanding models on Chinese pharmaceutical shipping documents. It probes semantic entity recognition, entity linking, and reading order prediction, specifically testing robustness to dense tabular layouts and long-range semantic dependencies.

Datasets

• PharmaShip — total ?; splits: test (-1); repo https://github.com/KevinYuLei/PharmaShip

Metrics

• F1 (primary) — range: percent
  • Harmonic mean of Precision and Recall: 2 * (Precision * Recall) / (Precision + Recall). Reported separately for SER, EL, and ROP tasks.

Input / output format

Input: Chinese pharmaceutical shipping documents (images/PDFs) with text segments, layout coordinates, and geometric/pixel information.

Output: Predicted semantic entity boundaries, linked entity identifiers, and ordered sequences or graphs for word-level and segment-level reading order.

Scoring recipe

def compute_f1(precision, recall):
    if precision + recall == 0:
        return 0.0
    return 2 * (precision * recall) / (precision + recall)

# Applied per task (SER, EL, ROP)
# Precision = TP / (TP + FP)
# Recall = TP / (TP + FN)

Common pitfalls

• Models overfit to layout heuristics and block-level 'false coupling' present in conventional benchmarks like FUNSD, failing on entity-centric annotations.
• Long-range medical relations and dense tabular layouts cause precision-recall trade-offs; geometry-only models are precision-leaning while pixel-aware models are recall-leaning.
• Segment-level reading order prediction suffers from boundary ambiguity and long-range crossings, making it significantly harder than word-level ordering.

Evidence (verbatim from paper)

Table III reports Performance comparison of different models on SER, EL, and ROP tasks of PharmaShip. Improvements $(\uparrow)$ denote F1 gains of RORE-enhanced variants.

Citation

@misc{xie2025pharmaship,
  title={PharmaShip: An Entity-Centric, Reading-Order-Supervised Benchmark for Chinese Pharmaceutical Shipping Documents},
  author={Xie et al. (2025)},
  year={2025},
  note={arXiv:2512.23714}
}

• arXiv: 2512.23714