Textatlas Eval

textatlas-eval

TextAtlas5M: A Large-scale Dataset for Dense Text Image Generation — Wang et al. (2025) (arXiv:2502.07870, 2025)

What this evaluates

Evaluates text-to-image generation models on their ability to render long, dense, and structurally complex text accurately within images. It probes both semantic alignment between the prompt and the generated image, and precise character/word-level OCR fidelity across diverse layouts, styles, and real-world scenes.

Datasets

• TextAtlasEval — total 4000; splits: test (4000)

Metrics

• CLIP Score (CS) — range: [-1, 1]
  • Cosine similarity between the CLIP-ViT-B/32 embeddings of the generated image and the ground truth text prompt. Higher values indicate stronger semantic alignment.
• OCR Accuracy (Acc.) (primary) — range: percent
  • Word-level exact match rate between OCR-extracted text and ground truth, allowing up to 80% character mismatch tolerance per word.
• F1 Score (F1.) — range: percent
  • Harmonic mean of word-level precision and recall computed on OCR-extracted vs. ground truth text, with an 80% mismatch tolerance threshold.
• Character Error Rate (CER) — range: [0, 1]
  • Normalized Levenshtein edit distance between the OCR-extracted text and the ground truth text, measuring character-level fidelity.

Input / output format

Input: Natural language prompt specifying text content, layout structure, and visual style.

Output: A single generated RGB image (typically 512x512 or 1024x1024 pixels).

Scoring recipe

def evaluate(dataset):
    results = {'CS': [], 'Acc': [], 'F1': [], 'CER': []}
    for prompt, gen_img, gt_text in dataset:
        img_emb = clip_vit_b_32.encode_image(gen_img)
        txt_emb = clip_vit_b_32.encode_text(gt_text)
        results['CS'].append(cosine_similarity(img_emb, txt_emb))
        ocr_text = paddle_ocr.extract_text(gen_img)
        acc, f1, cer = compute_word_metrics(ocr_text.split(), gt_text.split(), tolerance=0.8)
        results['Acc'].append(acc)
        results['F1'].append(f1)
        results['CER'].append(cer)
    return {k: mean(v) for k, v in results.items()}

Common pitfalls

• OCR extraction accuracy is highly sensitive to image resolution and font size; small or stylized text often yields lower extraction rates independent of generation quality.
• CLIP Score measures semantic alignment, not text fidelity; a model can achieve high CLIP scores while failing completely on OCR metrics.
• The 80% mismatch tolerance for word-level metrics differs from standard exact-match or full Levenshtein evaluations, inflating accuracy/F1 scores compared to conventional benchmarks.

Evidence (verbatim from paper)

For evaluation, we compute image-text similarity using CLIP-ViT-B/32, where higher scores indicate stronger alignment. For OCR-based metrics, we use PaddleOCR to extract generated text and compare it to ground truth. We report word-level accuracy, F1 score, and character error rate (CER), allowing up to 80% mismatch tolerance in word-level evaluation.

Citation

@misc{wang2025textatlas5m,
  title={TextAtlas5M: A Large-scale Dataset for Dense Text Image Generation},
  author={Wang et al. (2025)},
  year={2025},
  note={arXiv:2502.07870}
}

• arXiv: 2502.07870