Few Shot Ner Doc Eval

few-shot-ner-doc-eval

Towards Few-shot Entity Recognition in Document Images: A Graph Neural Network Approach Robust to Image Manipulation — Krishnan et al. (2023) (arXiv:2305.14828, 2023)

What this evaluates

Evaluates few-shot named entity recognition on document images, measuring how well models identify entity spans with limited training examples. It also probes model robustness to geometric image manipulations like rotation, scaling, and shifting during inference.

Datasets

• FUNSD — total 199; splits: train (-1), test (-1)
• CORD — total 1000; splits: train (-1), test (-1)

Metrics

• word-level F-1 score (primary) — range: percent
  • Word-level precision, recall, and F-1 score computed after converting predictions and ground truth into IOBES tagging format. F-1 is the harmonic mean of precision and recall.

Input / output format

Input: Document images with word-level bounding boxes and associated textual content, provided in few-shot settings with 1–10 labeled training examples.

Output: IOBES tagging sequence for each word token in the document image.

Scoring recipe

pred_iobes = convert_to_iobes(predictions)
gold_iobes = convert_to_iobes(gold_labels)
p = seqeval.metrics.precision_score(gold_iobes, pred_iobes, average='macro')
r = seqeval.metrics.recall_score(gold_iobes, pred_iobes, average='macro')
f1 = 2 * p * r / (p + r) if (p + r) > 0 else 0.0
return f1

Common pitfalls

• Few-shot results vary significantly with seed; must average over 6 random seeds per shot size to match reported numbers.
• Predictions must be explicitly converted to IOBES format before scoring; raw BIO or direct token matching will yield incorrect F-1.
• Robustness evaluation requires applying specific geometric transformations (shift=20, scale=2x, rotate=8°) to test images during inference.

Evidence (verbatim from paper)

For model evaluation, the results are first converted into IOBES tagging style and we then compute the word-level precision, recall and F-1 score using the seqeval APIs (Nakayama, 2018).

Citation

@misc{krishnan2023towards,
  title={Towards Few-shot Entity Recognition in Document Images: A Graph Neural Network Approach Robust to Image Manipulation},
  author={Krishnan et al. (2023)},
  year={2023},
  note={arXiv:2305.14828}
}

• arXiv: 2305.14828