Doc Img Class Retrieval Eval

doc-img-class-retrieval-eval

Evaluation of Deep Convolutional Nets for Document Image Classification and Retrieval — Harley et al. (2015) (arXiv:1502.07058, 2015)

What this evaluates

Evaluates the ability of deep convolutional neural networks and hand-crafted feature methods to classify scanned document images into predefined categories and retrieve semantically similar documents from a large corpus.

Datasets

• SmallTobacco — total 3482; splits: train (800), val (200), test (2482)
• BigTobacco — total 400000; splits: train (320000), val (40000), test (40000)

Metrics

• classification accuracy (primary) — range: [0, 1]
  • The fraction of test images correctly assigned to their true category label.
• mAP@10 — range: [0, 1]
  • Mean Average Precision at rank 10. AP = sum_{k=1}^{10} (P(k) * rel(k)) / number of relevant documents. mAP is the average AP across all test queries.

Input / output format

Input: Scanned document images, typically resized to 227x227 pixels for CNN processing, or converted to bag-of-words feature vectors.

Output: Predicted class label (classification) or a ranked list of training set images (retrieval).

Scoring recipe

# Classification Accuracy
correct = sum(1 for pred, gold in zip(predictions, gold_labels) if pred == gold)
accuracy = correct / len(gold_labels)

# mAP@10
aps = []
for query, gold_label in zip(test_queries, test_labels):
    ranked_docs = get_ranked_docs(query)[:10]
    rel_count = 0
    ap = 0.0
    for k, doc in enumerate(ranked_docs, 1):
        if doc.label == gold_label:
            rel_count += 1
            ap += rel_count / k
    ap /= max(rel_count, 1)
    aps.append(ap)
mAP10 = sum(aps) / len(aps)

Common pitfalls

• SmallTobacco results report the median performance over 10 random train/val/test splits, not the mean.
• Retrieval relevance is strictly defined by matching the document's class label, ignoring other visual or semantic similarities.
• Feature vectors larger than 128 dimensions are compressed via PCA and L2-normalized before retrieval, which can alter distance-based rankings.

Evidence (verbatim from paper)

The SmallTobacco dataset was split as in the related work: 800 images were used for training, 200 for validation, and the remainder for testing. Since this is a small dataset, 10 random splits in those proportions were created; results reflect the median performance from those splits. In the case of retrieval, the median was selected based on mean average precision at the 10th retrieval (mAP@10). ... Retrieval was measured using mean average precision (mAP). Average precision computes the average value of precision as a function of recall on some interval. Formally, the discrete version of this metric is given by AP = sum_{k=1}^n (P(k) * rel(k)) / number of relevant documents... Mean average precision is simply the average precision summed over all queries, divided by the number of queries.

Citation

@misc{harley2015docimg,
  title={Evaluation of Deep Convolutional Nets for Document Image Classification and Retrieval},
  author={Harley et al. (2015)},
  year={2015},
  note={arXiv:1502.07058}
}

• arXiv: 1502.07058