Papi Personality Alignment Eval

papi-personality-alignment-eval

Personality Alignment of Large Language Models — Zhu et al. (2024) (arXiv:2408.11779, 2024)

What this evaluates

Evaluates how well a language model's generated responses align with a specific individual's personality traits across the Big Five and Dark Triad dimensions. It measures the distance between the model's predicted personality profile and the target profile, with lower scores indicating better alignment.

Datasets

• PAPI — total 320000; splits: test (-1); repo https://github.com/zhu-minjun/PAlign

Metrics

• Aligned Score (primary) — range: other
  • Distance between the model's predicted personality scores and the target personality scores across 8 dimensions (Big Five + Dark Triad), calculated per Eq. 2 in the paper. Lower scores indicate better alignment.

Input / output format

Input: A prompt containing a statement from the IPIP-NEO-300 inventory, along with the target individual's personality trait scores to align to.

Output: Model-generated text response to the prompt.

Scoring recipe

# For each test instance:
# 1. Generate model response to the IPIP-NEO-300 statement.
# 2. Use a personality assessment model to predict the 8 trait scores from the response.
# 3. Compute distance (e.g., MSE) between predicted scores and target scores for each trait.
# 4. Average or sum distances across the 8 traits to get the Aligned Score.
# Lower Aligned Score = better alignment.

Common pitfalls

• The Aligned Score is a distance metric where lower values indicate better performance, contrary to typical accuracy metrics.
• Personality scores are predicted by an external assessor rather than being self-reported by the model.
• The evaluation covers both multiple-choice and open-ended generation tasks, requiring different scoring pipelines.

Evidence (verbatim from paper)

Comparison of alignment methods on the PAPI dataset using the Aligned Score of Big-Five personality traits and Dark Triad traits. The scores in the table represent the distance of alignment calculated by Eq. [2], with lower scores indicating better alignment.

Citation

@misc{zhu2024personalityalignment,
  title={Personality Alignment of Large Language Models},
  author={Zhu et al. (2024)},
  year={2024},
  note={arXiv:2408.11779}
}

• arXiv: 2408.11779