Sketchduo Eval

sketchduo-eval

StableSketcher: Enhancing Diffusion Model for Pixel-based Sketch Generation via Visual Question Answering Feedback — Park et al. (2025) (arXiv:2510.20093, 2025)

What this evaluates

This evaluation probes a diffusion model's ability to generate pixel-level sketches that align with detailed text prompts while maintaining stylistic abstraction and human-like drawing characteristics. It measures both perceptual image quality and fine-grained text-to-image semantic alignment across multiple complementary metrics.

Datasets

• SketchDUO — total 3000; splits: train (1800), test (1200)

Metrics

• FID — range: [0, inf)
  • Fréchet Inception Distance measures the Wasserstein-2 distance between the distributions of real and generated sketch images in the Inception feature space. Lower values indicate better image quality and distribution matching.
• LPIPS — range: [0, 1]
  • Learned Perceptual Image Patch Similarity computes perceptual distance using deep network features. Lower values indicate higher perceptual similarity to reference sketches.
• CLIPScore — range: [0, 1]
  • Cosine similarity between image and text embeddings from the CLIP model. Higher values indicate stronger text-image alignment.
• BERTScore — range: [0, 1]
  • Precision, recall, and F1 computed over token-level embeddings from BERT, matching predictions to gold text via cosine similarity. Higher values indicate better semantic alignment.
• TIFAScore (primary) — range: [0, 1]
  • Text-Image Fine-grained Alignment Score evaluates element-level fidelity by parsing prompts into specific visual elements and measuring their presence/accuracy in generated images. Higher values indicate better prompt fidelity.

Input / output format

Input: Text prompts describing sketch content, style, and background conditions (e.g., 'A black line drawing of a teddy bear with a friendly smile on a white background.').

Output: Pixel-based sketch images (binary/grayscale line drawings) generated by the diffusion model.

Scoring recipe

def evaluate_sketches(predictions, gold_prompts, real_sketches):
    results = {}
    results['FID'] = frechet_inception_distance(real_sketches, predictions)
    results['LPIPS'] = perceptual_l2_similarity(real_sketches, predictions)
    results['CLIPScore'] = clip_similarity(predictions, gold_prompts)
    results['BERTScore'] = bert_semantic_match(predictions, gold_prompts)
    results['TIFAScore'] = tifa_element_fidelity(predictions, gold_prompts)
    return results

Common pitfalls

• Metric directionality confusion: FID and LPIPS are lower-is-better, whereas CLIPScore, BERTScore, and TIFAScore are higher-is-better.
• Over-reliance on BERTScore: BERTScore captures global semantic similarity but misses fine-grained instance details; TIFAScore is required for accurate prompt fidelity assessment in sketch generation.
• User study ranking interpretation: Lower mean rank indicates stronger preference, which contradicts standard accuracy benchmarks and can cause misinterpretation if not explicitly noted.

Evidence (verbatim from paper)

To evaluate the quality of the generated images, we adopt five metrics that encompass both image quality and text-image alignment. For image quality assessment, we employ Fréchet Inception Distance (FID) and LPIPS. For text-image alignment, we leverage CLIPScore, BERTScore, and TIFAScore.

Citation

@misc{park2025stablesketcher,
  title={StableSketcher: Enhancing Diffusion Model for Pixel-based Sketch Generation via Visual Question Answering Feedback},
  author={Park et al. (2025)},
  year={2025},
  note={arXiv:2510.20093}
}

• arXiv: 2510.20093