P Bench Eval

p-bench-eval

Personalized Visual Instruction Tuning — Pi et al. (2024) (arXiv:2410.07113, 2024)

What this evaluates

Evaluates multimodal large language models' ability to recognize and respond to specific individuals in images using in-context learning. It probes robustness to complex scenes (multiple people, augmentations) and the capability to correctly reject unanswerable queries.

Datasets

• P-Bench — total ?; splits: test (-1); repo https://github.com/sterzhang/PVIT

Metrics

• accuracy (primary) — range: percent
  • Percentage of correct responses across multiple-choice questions and description tasks. For unanswerable queries, it measures the rejection rate (correctly identifying that the question cannot be answered).

Input / output format

Input: Image(s) of a target individual and/or scene, paired with text prompts containing multiple-choice questions or open-ended description requests.

Output: Model-generated text response, either a selected multiple-choice option or a descriptive sentence.

Scoring recipe

correct = 0
total = 0
for pred, gold in zip(predictions, gold_labels):
    if gold == 'unanswerable':
        correct += 1 if pred == 'unanswerable' else 0
    else:
        correct += 1 if pred == gold else 0
    total += 1
return (correct / total) * 100

Common pitfalls

• Models frequently hallucinate answers for unanswerable questions instead of correctly rejecting them, which drastically lowers robustness scores.
• Performance degrades significantly when scene images contain multiple individuals or use augmentations, requiring careful stratified evaluation across complexity levels.

Evidence (verbatim from paper)

In the figure on the right, we illustrate the evaluation accuracy after training with various amounts of data. Specifically, the horizontal axis indicates the number of data units, and each unit contains 8000 samples. We observe clear performance boost when scaling up the training dataset.

Citation

@misc{pi2024personalized,
  title={Personalized Visual Instruction Tuning},
  author={Pi et al. (2024)},
  year={2024},
  note={arXiv:2410.07113}
}

• arXiv: 2410.07113