Emnist Classification Eval

emnist-classification-eval

EMNIST: an extension of MNIST to handwritten letters — Cohen et al. (2017) (arXiv:1702.05373, 2017)

What this evaluates

Evaluates a model's ability to recognize and classify handwritten characters (digits and letters) from standardized 28x28 grayscale images. It probes robustness to case variations, class overlap, and imbalanced distributions across multiple dataset configurations.

Datasets

• EMNIST — total 814255; splits: train (-1), test (-1)

Metrics

• classification accuracy (primary) — range: percent
  • Percentage of correctly classified characters in the test set. Calculated as (correct predictions / total test samples) × 100%. Reported as mean ± standard deviation over 10 or 20 independent trials with different random weight initializations.

Input / output format

Input: 28×28 grayscale image of a single handwritten character.

Output: Discrete class label corresponding to one of the 10 digits, 26 lowercase letters, 26 uppercase letters, or merged letter classes, depending on the dataset variant.

Scoring recipe

correct = 0
for img, label in test_set:
    pred = model.predict(img)
    if pred == label:
        correct += 1
accuracy = (correct / len(test_set)) * 100

Common pitfalls

• Case sensitivity causes high misclassification rates between visually similar letters (e.g., I/L, G/Q) and digits (e.g., 1/I).
• Overall accuracy in By_Merge/By_Class is heavily skewed by the much larger number of digit samples compared to letter samples.
• Performance variance across trials is driven by random weight initialization rather than dataset changes, requiring multiple runs for stable reporting.

Evidence (verbatim from paper)

The accuracy of the classification task increased with each hidden layer size, reaching a maximum mean accuracy of $78.02%\pm 0.09%$ over twenty trials of networks containing 10,000 hidden layer neurons. The graph shows the accuracy for both datasets in terms of the percentage of characters correctly identified and shows that the By_Merge dataset outperforms the By_Class dataset at every hidden layer size.

Citation

@misc{cohen2017emnist,
  title={EMNIST: an extension of MNIST to handwritten letters},
  author={Cohen et al. (2017)},
  year={2017},
  note={arXiv:1702.05373}
}

• arXiv: 1702.05373