Gpt4v Ocr Eval

gpt4v-ocr-eval

Exploring OCR Capabilities of GPT-4V(ision) : A Quantitative and In-depth Evaluation — Shi et al. (2023) (arXiv:2310.16809, 2023)

What this evaluates

Evaluates the optical character recognition and document understanding capabilities of GPT-4V across multiple tasks including scene text recognition, handwritten text recognition, mathematical expression recognition, and table structure recognition. It probes the model's robustness to different languages, handwriting styles, complex layouts, and input resolutions.

Datasets

• CUTE80 — total 80; splits: test (50); repo https://github.com/Yuliang-Liu/MultimodalOCR
• SCUT-CTW1500 — total 1500; splits: test (50); repo https://github.com/Yuliang-Liu/MultimodalOCR
• Total-Text — total 1555; splits: test (50); repo https://github.com/Yuliang-Liu/MultimodalOCR
• WordArt — total 6316; splits: test (50); repo https://github.com/Yuliang-Liu/MultimodalOCR
• ReCTS — total 25000; splits: test (50); repo https://github.com/Yuliang-Liu/MultimodalOCR
• MLT19 — total 20000; splits: train (200); repo https://github.com/Yuliang-Liu/MultimodalOCR
• IAM — total 1539; splits: test (50)
• CASIA-HWDB — total 5090; splits: test (50)
• CROHME2014 — total 9820; splits: test (50)
• HME100K — total 100000; splits: test (50)
• SciTSR — total 15000; splits: test (50)
• WTW — total 14581; splits: test (50)

Metrics

• WAICS (primary) — range: percent
  • Word Accuracy Ignoring Case and Symbols. Calculated as the ratio of correctly recognized words (case-insensitive, ignoring symbols) to the total number of ground truth words.
• Precision/Recall/F1 — range: percent
  • For end-to-end text spotting. Precision is correctly identified words divided by generated words. Recall is correctly identified words divided by ground truth words. F1 = 2 * precision * recall / (precision + recall).
• WER/CER — range: percent
  • Word Error Rate and Character Error Rate for handwritten English text. Measures the percentage of words/characters that differ from the ground truth.
• TEDS-S — range: [0, 1]
  • Tree-Edit-Distance-Based Similarity score that disregards cell text content and only evaluates the accuracy of the predicted table structure (HTML tree).

Input / output format

Input: Image of text/document + a task-specific text prompt (e.g., 'What is the scene text in the image?') uploaded via GPT-4V's web interface. Each image is processed in a separate dialogue window to prevent context interference.

Output: Text response generated by GPT-4V containing the recognized text, LaTeX formula, or HTML table structure, depending on the task prompt.

Scoring recipe

def evaluate(predictions, golds, metric):
    if metric == 'WAICS':
        return sum(1 for p, g in zip(predictions, golds) if p.lower() == g.lower()) / len(golds)
    elif metric in ['Precision', 'Recall', 'F1']:
        pred_words = [w for p in predictions for w in p.split()]
        gold_words = [w for g in golds for w in g.split()]
        tp = sum(1 for w in pred_words if w in gold_words)
        prec = tp / len(pred_words) if pred_words else 0
        rec = tp / len(gold_words) if gold_words else 0
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
        return {'precision': prec, 'recall': rec, 'f1': f1}
    elif metric == 'TEDS-S':
        return compute_teds_score(predictions, golds, ignore_text=True)
    elif metric in ['WER', 'CER']:
        return edit_distance_rate(predictions, golds, granularity=metric)

Common pitfalls

• API rate limits (50 conversations per 3 hours) force sampling only 50 images per dataset, which may not represent full dataset performance.
• Using isolated dialogue windows per image prevents context leakage but significantly increases API costs and prevents batch processing.
• Space-splitting for spotting metrics may fail on multi-word expressions, punctuation, or non-space-delimited languages like Chinese.

Evidence (verbatim from paper)

For the evaluation of word-level recognition, we employ word accuracy ignoring case and symbols (WAICS) as metric. In the task of end-to-end text spotting, the predictions of GPT-4V and ground truths (GT) are split with spaces and then evaluated using precision and recall. Precision represents the ratio of correctly identified words to those generated by GPT-4V, while recall is the ratio of correctly identified words to the total number of GT words. We also compute the F1 score as follow. F1 = 2precisionrecall/(precision+recall)

Citation

@misc{shi2023exploring,
  title={Exploring OCR Capabilities of GPT-4V(ision) : A Quantitative and In-depth Evaluation},
  author={Shi et al. (2023)},
  year={2023},
  note={arXiv:2310.16809}
}

• arXiv: 2310.16809