Text Rendering Eval

text-rendering-eval

TextDiffuser: Diffusion Models as Text Painters — Chen et al. (2023) (arXiv:2305.10855, 2023)

What this evaluates

Evaluates a model's ability to generate images with accurate, legible, and layout-controlled text based on text prompts or masked regions. It probes text coherence, character-level rendering fidelity, and alignment between generated text and background imagery.

Datasets

• MARIO-10M — total ?; splits: train (-1), test (-1)
• DrawBenchText — total ?; splits: test (-1)

Metrics

• FID — range: [0, ∞)
  • Fréchet Inception Distance measuring the statistical distance between feature distributions of real and generated images. Lower is better.
• CLIPScore — range: [0, 1]
  • Cosine similarity between image and text embeddings extracted from a pre-trained CLIP model. Higher is better.
• OCR(F-measure) (primary) — range: [0, 1]
  • Harmonic mean of OCR precision and recall, computed after extracting text via Microsoft Read API and comparing against ground truth keywords. Higher is better.
• OCR(Accuracy) — range: [0, 1]
  • Proportion of generated images where detected words exactly match the prompt keywords. Higher is better.

Input / output format

Input: Text prompts (for whole-image generation) or source images with masked text bounding boxes (for part-image generation).

Output: 512×512 RGB images.

Scoring recipe

def evaluate(predictions, gold):
    ocr_results = [microsoft_read_api(img) for img in predictions]
    acc = sum(1 for d in ocr_results if d == gold) / len(gold)
    prec, rec, f1 = compute_precision_recall_f1(ocr_results, gold)
    clip_scores = [clip_similarity(img, prompt) for img, prompt in zip(predictions, prompts)]
    fid = frechet_inception_distance(real_images, predictions)
    return {'acc': acc, 'f1': f1, 'clip': np.mean(clip_scores), 'fid': fid}

Common pitfalls

• FID scores are not directly comparable across different output resolutions (e.g., 512x512 vs 1024x1024).
• OCR metrics heavily depend on the specific OCR engine/API used; results may vary significantly with different text recognition models.
• CLIPScore measures semantic alignment but does not guarantee character-level spelling accuracy or legibility.

Evidence (verbatim from paper)

We utilize Microsoft Read API to detect and recognize the texts in generated images. We use Accuracy (Acc) as the metric to justify whether the detected words exactly match the keywords. ... According to Table 4, we demonstrate the quantitative results of the text-to-image task compared with existing methods. Our TextDiffuser obtains the best CLIPScore while achieving comparable performance in terms of FID. Besides, TextDiffuser achieves the best performance regarding four OCR-related metrics.

Citation

@misc{chen2023textdiffuser,
  title={TextDiffuser: Diffusion Models as Text Painters},
  author={Chen et al. (2023)},
  year={2023},
  note={arXiv:2305.10855}
}

• arXiv: 2305.10855