Lex Bench Eval

lex-bench-eval

LeX-Art: Rethinking Text Generation via Scalable High-Quality Data Synthesis — Zhao et al. (2025) (arXiv:2503.21749, 2025)

What this evaluates

Evaluates text-to-image generation models on their ability to accurately render specified text within images, control visual attributes (color, position, font), and maintain aesthetic quality. It measures OCR fidelity, attribute controllability, and human-perceived aesthetics.

Datasets

• LeX-Bench — total ?; splits: easy (-1), medium (-1), hard (-1)
• SimpleBench — total ?; splits: (unstated)
• CreateBench — total ?; splits: (unstated)
• AnyText-Benchmark — total ?; splits: (unstated)

Metrics

• PNED (primary) — range: [0, 1]
  • Pairwise Normalized Edit Distance between the ground-truth text prompt and the OCR-extracted text from the generated image. Lower values indicate higher text fidelity.
• Recall — range: [0, 1]
  • Binary metric indicating whether the NED between prompt and OCR output is ≤ 0.3. Higher values indicate better text completeness.
• Aesthetic — range: score [1, 5]
  • Human-annotated average score for visual quality and composition.
• CLIPScore — range: [0, 1]
  • Standard cosine similarity between image and text embeddings in CLIP space.
• Accuracy — range: [0, 1]
  • Sentence accuracy (Sen. Acc) from AnyText-Benchmark, measuring exact or near-exact text match.

Input / output format

Input: Text prompt (optionally enhanced by LeX-Enhancer)

Output: Generated image (1024×1024 resolution)

Scoring recipe

def evaluate(predictions, gold_prompts):
    ocr_texts = [run_ocr(img) for img in predictions]
    pned = [normalized_edit_distance(g, o) for g, o in zip(gold_prompts, ocr_texts)]
    recall = [1.0 if ned <= 0.3 else 0.0 for ned in pned]
    aesthetic = [human_rate(img) for img in predictions]
    return {
        'PNED': mean(pned),
        'Recall': mean(recall),
        'Aesthetic': mean(aesthetic)
    }

Common pitfalls

• PNED is a pairwise normalized edit distance where lower values are better, contrary to standard accuracy metrics.
• Recall is strictly thresholded at NED ≤ 0.3, not based on exact character matching or standard F1 scores.
• Aesthetic scores are human preference averages (likely 1–5 scale), not automated computational metrics.

Evidence (verbatim from paper)

These benchmarks provide diverse text prompts and assess models using OCR-based metrics such as sentence accuracy (Sen. Acc), normalized edit distance (NED), and OCR-F1. In addition, they report CLIP Score*[[14]]* and Frechet Inception Distance (FID)[[15]] to evaluate visual-textual alignment and image quality. Note that recall is computed based on NED thresholding at 0.3 to mitigate the impact of minor character errors.

Citation

@misc{zhao2025lexart,
  title={LeX-Art: Rethinking Text Generation via Scalable High-Quality Data Synthesis},
  author={Zhao et al. (2025)},
  year={2025},
  note={arXiv:2503.21749}
}

• arXiv: 2503.21749