Funsd Form Understanding Eval

funsd-form-understanding-eval

FUNSD: A Dataset for Form Understanding in Noisy Scanned Documents — Jaume et al. (2019) (arXiv:1905.13538, 2019)

What this evaluates

Evaluates end-to-end form understanding on noisy scanned documents, covering text detection, optical character recognition, word grouping, semantic entity labeling, and entity linking.

Datasets

• FUNSD — total 199; splits: train (-1), test (-1)

Metrics

• F1-score (primary) — range: [0, 1]
  • Computed from precision and recall at a fixed Intersection over Union (IoU) threshold of 0.5 for bounding boxes. F1 = 2 * (Precision * Recall) / (Precision + Recall).
• Levenshtein similarity — range: [0, 1]
  • S(w_p, w_gt) = 1 - L(w_p, w_gt) / max(|w_p|, |w_gt|), where L is the Levenshtein distance and |.| is character count. Case-sensitive and includes checkbox recognition.
• Adjusted Rand Index (ARI) — range: [0, 1]
  • Measures clustering agreement between predicted word groups and ground-truth semantic entities, adjusted for chance. Assumes the optimal number of clusters equals the number of ground-truth entities.
• F1-score (entity linking) — range: [0, 1]
  • Binary classification metric evaluating predicted directed relations between correctly identified and labeled semantic entities. Computed from precision and recall of link existence.

Input / output format

Input: Scanned document images with ground-truth bounding boxes for words, semantic entities, and directed relations between entities.

Output: Predicted bounding boxes for words/entities, OCR text strings, cluster assignments for word grouping, entity labels (question/answer/header/other), and predicted directed links between entities.

Scoring recipe

det_precision, det_recall = compute_detection_metrics(pred_boxes, gt_boxes, iou_thresh=0.5)
det_f1 = 2 * det_precision * det_recall / (det_precision + det_recall)

ocr_sims = []
for pred_word, gt_word in zip(correctly_detected_preds, gt_words):
    dist = levenshtein_distance(pred_word, gt_word)
    sim = 1 - dist / max(len(pred_word), len(gt_word))
    ocr_sims.append(sim)
avg_ocr_sim = sum(ocr_sims) / len(gt_words)

ari = adjusted_rand_index(pred_word_clusters, gt_entity_clusters)

link_precision, link_recall = compute_link_metrics(pred_links, gt_links)
link_f1 = 2 * link_precision * link_recall / (link_precision + link_recall)

Common pitfalls

• IoU threshold for detection is fixed at 0.5, not the standard 0.5/0.75 used in object detection.
• OCR similarity is normalized by the maximum length of predicted vs ground truth, not just ground truth length.
• Word grouping is evaluated as clustering with ARI, assuming the number of clusters exactly matches ground-truth entities.
• Entity linking F1 is computed only on entities that are already correctly detected and labeled, ignoring error cascading.

Evidence (verbatim from paper)

We test text detection at the word level... compute the precision, recall, and F1 score of the FUNSD test set at IoU=0.5. We evaluate the relevance of the OCR output by computing the Levenshtein similarity between the predicted word and the ground-truth word: S(w_p, w_gt)=1-L(w_p, w_gt)/max(|w_p|,|w_gt|). We propose using the adjusted rand index (ARI) as a metric. The metric we used verifies whether the predicted links exist among all the semantic entities correctly identified and labeled. We can then compute the precision, recall, and F1-score.

Citation

@misc{jaume2019funsd,
  title={FUNSD: A Dataset for Form Understanding in Noisy Scanned Documents},
  author={Jaume et al. (2019)},
  year={2019},
  note={arXiv:1905.13538}
}

• arXiv: 1905.13538