Lunara Aesthetic Eval

lunara-aesthetic-eval

Moonworks Lunara Aesthetic Dataset — Wang et al. (2026) (arXiv:2601.07941, 2026)

What this evaluates

This evaluation protocol assesses the quality of the Lunara dataset by measuring visual aesthetic appeal, semantic alignment between images and prompts, cross-modal retrieval accuracy, and perceptual diversity across images. It provides a structured framework to verify that the dataset prioritizes high-quality, stylistically diverse, and semantically grounded image-text pairs over noisy web-scraped alternatives.

Datasets

• Lunara Aesthetic Dataset — total 2000; splits: full (2000)

Metrics

• LAION Aesthetics v2 score (primary) — range: [0, 10]
  • Predicted aesthetic score from a CLIP-based model trained to approximate human judgments of visual appeal. Reported as mean, median, percentiles, and percentage exceeding a threshold of 6.5.
• CLIP cosine similarity — range: [-1, 1]
  • Mean cosine similarity between image and text embeddings extracted by a pretrained CLIP model (ViT-B/32 or ViT-L/14).
• Recall@K — range: [0, 100] percent
  • Percentage of correct image-text pairs retrieved within the top K candidates during bidirectional cross-modal retrieval.
• LPIPS — range: [0, 1]
  • Learned Perceptual Image Patch Similarity computed over randomly sampled image pairs to quantify intra- and inter-category visual diversity.

Input / output format

Input: Image-prompt pairs from the dataset; evaluation models receive either an image, a text prompt, or both depending on the metric.

Output: Numerical scores: aesthetic score (float), cosine similarity (float), recall percentages (float), and perceptual similarity values (float).

Scoring recipe

# 1. Aesthetic Score
scores = [aesthetics_v2_model(img) for img in images]
mean_aesthetic = sum(scores) / len(scores)
pct_high = sum(1 for s in scores if s >= 6.5) / len(scores)

# 2. CLIP Alignment & Retrieval
img_emb = clip_model.encode(images)
txt_emb = clip_model.encode(prompts)
clip_sim = cosine_similarity(img_emb, txt_emb).mean()

retrieval_scores = txt_emb @ img_emb.T
recalls = [top_k_accuracy(retrieval_scores, k) for k in [1, 5, 10]]

# 3. Diversity
lpips_vals = [lpips_model(i, j) for i, j in sampled_pairs]
intra_div = mean(lpips_vals[:2000])
inter_div = mean(lpips_vals[2000:])

Common pitfalls

• CLIP similarity values are backbone-dependent and not directly comparable across different models or datasets; always specify the backbone used.
• Cross-modal retrieval performance can be suppressed by high visual similarity within categories (e.g., portraits), which inflates difficulty and lowers Recall@1 despite accurate captions.
• The LAION Aesthetics v2 threshold of 6.5 is arbitrary; comparing percentages above this threshold requires consistent model calibration and dataset size.

Evidence (verbatim from paper)

We evaluate image aesthetics using the LAION Aesthetics v2 predictor, a CLIP-based model trained to approximate aggregate human judgments of visual appeal. We compare our dataset against several widely used vision–language datasets: Conceptual Captions (CC3M) (Sharma et al. (2018)), a random subset of LAION-2B-Aesthetic (Schuhmann et al. (2022)), and the Wikipedia-based Image–Text dataset (WIT) (Srinivasan et al. (2021)). Table 2 reports full distributional statistics of predicted aesthetic scores. Our dataset (Lunara) achieves a substantially higher mean aesthetic score (6.32) than all baselines, exceeding CC3M by +1.54, LAION-2B-Aesthetic by +1.07, and WIT by +1.24.

Citation

@misc{wang2026lunara,
  title={Moonworks Lunara Aesthetic Dataset},
  author={Wang et al. (2026)},
  year={2026},
  note={arXiv:2601.07941}
}

• arXiv: 2601.07941