Pamela Eval

pamela-eval

Personalizing Text-to-Image Generation to Individual Taste — Maerten et al. (2026) (arXiv:2604.07427, 2026)

What this evaluates

Probes a model's ability to predict individual user aesthetic preferences for AI-generated images based on prompt, image, and user demographics. It specifically tests both interpolation for known users and zero-shot few-shot generalization to novel users.

Datasets

• PAM∃LA — total 70000; splits: train (50222), val_seen (6551), test_seen (9735), val_unseen (926), test_unseen (2470); repo https://github.com/PAMELA-bench/PAMELA_Predictor

Metrics

• SROCC (primary) — range: [-1, 1]
  • Spearman Rank Correlation Coefficient; measures the rank-order correlation between predicted and actual preference ratings.
• PLCC — range: [-1, 1]
  • Pearson Linear Correlation Coefficient; measures the linear correlation between predicted and actual preference ratings.
• pairwise accuracy — range: [0, 1]
  • The proportion of correctly ordered image pairs based on predicted versus actual ratings.

Input / output format

Input: A text prompt, a generated image, and user demographic metadata (age, gender, education, art experience) or a few-shot context of k image-rating pairs for unseen users.

Output: A continuous scalar aesthetic preference rating score.

Scoring recipe

import scipy.stats as stats
def score(preds, golds):
    srocc = stats.spearmanr(preds, golds).correlation
    plcc = stats.pearsonr(preds, golds).correlation
    correct, total = 0, 0
    for i in range(len(preds)):
        for j in range(i+1, len(preds)):
            if golds[i] != golds[j]:
                total += 1
                if (preds[i] > preds[j]) == (golds[i] > golds[j]):
                    correct += 1
    return {'SROCC': srocc, 'PLCC': plcc, 'pairwise_accuracy': correct / total if total else 0}

Common pitfalls

• Confusing 'seen users' (interpolation) with 'unseen users' (zero-shot few-shot generalization), which require fundamentally different evaluation protocols.
• Averaging ratings across users before evaluation, which defeats the purpose of personalized preference prediction and masks individual taste variations.
• Relying solely on population-level metrics (AvgSROCC) instead of user-level metrics (UserSROCC), which fails to capture the model's ability to capture idiosyncratic preferences.

Evidence (verbatim from paper)

Our model outperforms all baselines across both evaluation regimes (user level vs population average) and all three metrics (SROCC, PLCC, pairwise accuracy).

Citation

@misc{maerten2026pamela,
  title={Personalizing Text-to-Image Generation to Individual Taste},
  author={Maerten et al. (2026)},
  year={2026},
  note={arXiv:2604.07427}
}

• arXiv: 2604.07427