Notes Bank Eval

notes-bank-eval

NoTeS-Bank: Benchmarking Neural Transcription and Search for Scientific Notes Understanding — Pal et al. (2025) (arXiv:2504.09249, 2025)

What this evaluates

Evaluates vision-language models on evidence-based visual question answering over unstructured, handwritten scientific notes. The benchmark probes a model's ability to localize relevant visual evidence via bounding boxes, classify content types, and generate natural language answers explicitly grounded in the visual input.

Datasets

• NoTeS-Bank — total ?; splits: test (-1)

Metrics

• NDCG@5 (primary) — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank 5. It measures the quality of the retrieved evidence documents by summing the relevance scores of each item, discounted logarithmically by its position in the ranked list, and normalizing by the ideal DCG.
• MRR — range: [0, 1]
  • Mean Reciprocal Rank. Calculated as the average of the reciprocal of the rank of the first relevant document retrieved across all queries.
• Recall@K — range: [0, 1]
  • The proportion of relevant evidence items successfully retrieved within the top K results, measuring the system's ability to find all necessary supporting documents.
• IoU — range: [0, 1]
  • Intersection over Union between the predicted bounding box and the ground-truth bounding box for evidence localization. It quantifies spatial overlap accuracy.
• ANLS — range: [0, 1]
  • Average Normalized Levenshtein Similarity. It measures character-level similarity between the predicted answer and the ground-truth answer, providing robustness to OCR errors and handwriting variations.

Input / output format

Input: A single image I of a handwritten scientific note (1–3 pages) containing unstructured text, symbols, equations, and diagrams, paired with a natural language question Q.

Output: A natural language answer A and a set of evidence E = {(B_i, L_i, G_i)}_{i=1}^p, where B_i is a bounding box, L_i is a content-type label, and G_i is grounding information for each relevant visual portion.

Scoring recipe

def evaluate(predictions, gold):
    iou_scores = [compute_iou(pred_box, gold_box) for pred_box, gold_box in zip(predictions['boxes'], gold['boxes'])]
    ndcg = compute_ndcg(predictions['retrieved_docs'], gold['relevant_docs'], k=5)
    mrr = compute_mrr(predictions['retrieved_docs'], gold['relevant_docs'])
    recall = compute_recall(predictions['retrieved_docs'], gold['relevant_docs'], k=5)
    anls = compute_levenshtein_similarity(predictions['answer'], gold['answer'])
    return {'IoU': iou_scores, 'NDCG@5': ndcg, 'MRR': mrr, 'Recall@5': recall, 'ANLS': anls}

Common pitfalls

• Models frequently generate plausible text answers without correctly localizing the supporting visual evidence via bounding boxes.
• Handwriting variability and non-linear layouts cause severe OCR degradation, leading to mismatched evidence retrieval.
• Content-type classification (e.g., equations vs. diagrams) is often overlooked, breaking the explicit grounding requirement.

Evidence (verbatim from paper)

The benchmark exposes critical gaps in current VLMs and retrieval systems, particularly in handling messy handwriting, non-linear layouts, and visual-semantic grounding, establishing new evaluation metrics (NDCG@5, MRR, Recall@K, IoU, ANLS) for visual document understanding.

Citation

@misc{pal2025notesbank,
  title={NoTeS-Bank: Benchmarking Neural Transcription and Search for Scientific Notes Understanding},
  author={Pal et al. (2025)},
  year={2025},
  note={arXiv:2504.09249}
}

• arXiv: 2504.09249