# Sketchduo Eval

> This evaluation probes a diffusion model's ability to generate pixel-level sketches that align with detailed text prompts while maintaining stylistic abstraction and human-like drawing characteristics. It measures both perceptual image quality and fine-grained text-to-image semantic alignment across multiple complementary metrics. Use when the user wants to benchmark on SketchDUO, or asks about evaluating this task. Reports TIFAScore.

- Skill: `qhjqhj00/sketchduo-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/sketchduo-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/sketchduo-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/sketchduo-eval

---


# sketchduo-eval

> StableSketcher: Enhancing Diffusion Model for Pixel-based Sketch Generation via Visual Question Answering Feedback — Park et al. (2025) (arXiv:2510.20093, 2025)

## What this evaluates

This evaluation probes a diffusion model's ability to generate pixel-level sketches that align with detailed text prompts while maintaining stylistic abstraction and human-like drawing characteristics. It measures both perceptual image quality and fine-grained text-to-image semantic alignment across multiple complementary metrics.

## Datasets

- **SketchDUO** — total 3000; splits: train (1800), test (1200)

## Metrics

- `FID` — range: [0, inf)
  - Fréchet Inception Distance measures the Wasserstein-2 distance between the distributions of real and generated sketch images in the Inception feature space. Lower values indicate better image quality and distribution matching.
- `LPIPS` — range: [0, 1]
  - Learned Perceptual Image Patch Similarity computes perceptual distance using deep network features. Lower values indicate higher perceptual similarity to reference sketches.
- `CLIPScore` — range: [0, 1]
  - Cosine similarity between image and text embeddings from the CLIP model. Higher values indicate stronger text-image alignment.
- `BERTScore` — range: [0, 1]
  - Precision, recall, and F1 computed over token-level embeddings from BERT, matching predictions to gold text via cosine similarity. Higher values indicate better semantic alignment.
- `TIFAScore` **(primary)** — range: [0, 1]
  - Text-Image Fine-grained Alignment Score evaluates element-level fidelity by parsing prompts into specific visual elements and measuring their presence/accuracy in generated images. Higher values indicate better prompt fidelity.

## Input / output format

**Input**: Text prompts describing sketch content, style, and background conditions (e.g., 'A black line drawing of a teddy bear with a friendly smile on a white background.').

**Output**: Pixel-based sketch images (binary/grayscale line drawings) generated by the diffusion model.

## Scoring recipe

```python
def evaluate_sketches(predictions, gold_prompts, real_sketches):
    results = {}
    results['FID'] = frechet_inception_distance(real_sketches, predictions)
    results['LPIPS'] = perceptual_l2_similarity(real_sketches, predictions)
    results['CLIPScore'] = clip_similarity(predictions, gold_prompts)
    results['BERTScore'] = bert_semantic_match(predictions, gold_prompts)
    results['TIFAScore'] = tifa_element_fidelity(predictions, gold_prompts)
    return results
```

## Common pitfalls

- Metric directionality confusion: FID and LPIPS are lower-is-better, whereas CLIPScore, BERTScore, and TIFAScore are higher-is-better.
- Over-reliance on BERTScore: BERTScore captures global semantic similarity but misses fine-grained instance details; TIFAScore is required for accurate prompt fidelity assessment in sketch generation.
- User study ranking interpretation: Lower mean rank indicates stronger preference, which contradicts standard accuracy benchmarks and can cause misinterpretation if not explicitly noted.

## Evidence (verbatim from paper)

> To evaluate the quality of the generated images, we adopt five metrics that encompass both image quality and text-image alignment. For image quality assessment, we employ Fréchet Inception Distance (FID) and LPIPS. For text-image alignment, we leverage CLIPScore, BERTScore, and TIFAScore.

## Citation

```bibtex
@misc{park2025stablesketcher,
  title={StableSketcher: Enhancing Diffusion Model for Pixel-based Sketch Generation via Visual Question Answering Feedback},
  author={Park et al. (2025)},
  year={2025},
  note={arXiv:2510.20093}
}
```

- arXiv: 2510.20093