Yochameleon Personalized Eval

yochameleon-personalized-eval

YoChameleon: Personalized Vision and Language Generation — Nguyen et al. (2025) (arXiv:2504.20998, 2025)

What this evaluates

Evaluates a multimodal model's ability to personalize to a few-shot visual concept for both understanding (recognition and QA) and pixel-level image generation. It probes whether learnable soft prompts can capture subject-specific details without catastrophic forgetting, while measuring token efficiency compared to standard prompting.

Datasets

• Yo’LLaVA dataset — total 40; splits: test (-1)

Metrics

• Recognition Accuracy (primary) — range: [0, 1]
  • Weighted accuracy balancing positive and negative classes to mitigate class imbalance on the 333 positive vs 13,000 negative test set.
• CLIP Image Similarity Score — range: [0, 1]
  • Cosine similarity between CLIP image embeddings of the generated output and the reference positive example, averaged over 100 generations per concept.
• Question Answering Accuracy — range: [0, 1]
  • Percentage of correct answers on 500 multiple-choice questions (A/B) covering visual and text-based attributes.
• Facial Similarity Score — range: [0, 1]
  • ArcFace similarity metric between generated and real images for 10 human concepts.

Input / output format

Input: Recognition: single image + prompt 'Is in this photo?'. QA: multiple-choice question (A/B) with visual or textual context. Generation: prompt 'A photo of '.

Output: Recognition: 'Yes' or 'No'. QA: 'A' or 'B'. Generation: RGB image.

Scoring recipe

def compute_metrics(predictions, golds, generated_imgs=None, ref_imgs=None):
    # Recognition: weighted accuracy
    pos_acc = sum(1 for p, g in zip(predictions, golds) if p == g == 'Yes') / sum(1 for g in golds if g == 'Yes')
    neg_acc = sum(1 for p, g in zip(predictions, golds) if p == g == 'No') / sum(1 for g in golds if g == 'No')
    rec_acc = (pos_acc + neg_acc) / 2
    # Generation: CLIP Image Similarity
    clip_sim = sum(clip_cosine_similarity(g, r) for g, r in zip(generated_imgs, ref_imgs)) / len(generated_imgs)
    return {'recognition_accuracy': rec_acc, 'clip_i': clip_sim}

Common pitfalls

• Using standard accuracy instead of the specified weighted accuracy, which would heavily penalize the model on the imbalanced 333 positive vs 13,000 negative test set.
• Confusing the ~1,100 training negative images (used for soft-positive augmentation) with the 13,000 test negative images.
• Comparing token counts directly without accounting for the base model's architecture, as Yo'Chameleon uses 32 latent tokens while baselines use ~64 to ~4k visible tokens.

Evidence (verbatim from paper)

In total, there are 333 positive and 13,000 negative images for recognition. During testing, we present a photo and ask the model “Is in this photo?” The ground-truth answer is either “Yes” or “No”. We use a weighted accuracy metric to balance the positive and negative classes, following the protocol in[[19]]. For question-answering, we provide multiple-choice questions (A or B) with 100 visual and 400 text-based questions. For image generation, we produce 100 images per concept using the prompt “A photo of ” and compute the CLIP Image Similarity Score between the generated images and positive examples.

Citation

@misc{nguyen2025yochameleon,
  title={YoChameleon: Personalized Vision and Language Generation},
  author={Nguyen et al. (2025)},
  year={2025},
  note={arXiv:2504.20998}
}

• arXiv: 2504.20998