Few Shot Entity Recognition Eval

few-shot-entity-recognition-eval

Towards Few-shot Entity Recognition in Document Images: A Label-aware Sequence-to-Sequence Framework — Wang et al. (2022) (arXiv:2204.05819, 2022)

What this evaluates

Evaluates few-shot entity recognition in document images by measuring how well a model identifies and classifies entity spans using minimal labeled examples. It probes the model's ability to jointly leverage textual semantics and spatial layout information under low-data regimes.

Datasets

• FUNSD — total ?; splits: train (149), test (50)
• CORD-Lv1 — total ?; splits: train (800), test (100)

Metrics

• word-level F-1 score (primary) — range: percent
  • Standard word-level precision, recall, and F1 computed after converting predictions to IOBES tagging style. Evaluated using the Nakayama (2018) API for sequence labeling.

Input / output format

Input: Document images with associated word-level bounding boxes and textual content. The model processes text tokens alongside spatial coordinates/embeddings.

Output: IOBES tagging sequence corresponding to each word in the document.

Scoring recipe

pred_iobes = model.generate_iobes(image, words, boxes)
gold_iobes = get_gold_iobes(image)
p, r, f1 = compute_word_level_metrics(gold_iobes, pred_iobes)
# Average over 6 random few-shot seeds
return np.mean([f1 for _ in range(6)])

Common pitfalls

• Few-shot splits are generated by randomly sampling training pages; results are highly seed-dependent, so reporting mean ± standard deviation across 6 seeds is mandatory.
• Evaluation is strictly word-level IOBES tagging; span-based or character-level metrics are not used, and comparisons must use the same IOBES conversion.
• Dataset sizes are reported in pages, not word/entity counts, which can mislead expectations about the actual number of training instances.

Evidence (verbatim from paper)

To evaluate our model, we first convert our results into IOBES tagging style and compute the word-level precision, recall, and F-1 score using the APIs from Nakayama (2018) so that all comparisons with sequence labeling methods are under the same metrics.

Citation

@misc{wang2022laser,
  title={Towards Few-shot Entity Recognition in Document Images: A Label-aware Sequence-to-Sequence Framework},
  author={Wang et al. (2022)},
  year={2022},
  note={arXiv:2204.05819}
}

• arXiv: 2204.05819