Personamem Eval

personamem-eval

Know Me, Respond to Me: Benchmarking LLMs for Dynamic User Profiling and Personalized Responses at Scale — Jiang et al. (2025) (arXiv:2504.14225, 2025)

What this evaluates

Evaluates large language models' ability to track dynamic user profile evolution over time and generate personalized responses to in-situ queries. It probes long-context memory, preference tracking, and contextual alignment across interleaved multi-session conversations.

Datasets

• PersonaMem — total 6000; splits: test (6000); repo https://github.com/bowen-upenn/PersonaMem

Metrics

• multiple-choice selection (primary) — range: [0, 1]
  • Calculated as the number of correctly identified responses divided by the total number of in-situ queries. Models must pick the single correct chatbot response from four options.

Input / output format

Input: A user persona profile containing demographic information and dynamic characteristics, interleaved multi-turn conversation histories across multiple task sessions, and a first-person in-situ user query asking for help or suggestions based on the current state of the profile.

Output: A multiple-choice selection identifying the single correct chatbot response from four provided options.

Scoring recipe

def compute_metric(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return correct / len(gold)

Common pitfalls

• Models often fail to distinguish between outdated and irrelevant information, defaulting to static facts from early conversation turns.
• The interleaved temporal order of sessions requires strict chronological tracking, which many models lose when context windows exceed 32k tokens.

Evidence (verbatim from paper)

To understand how well LLM chatbots can track the evolution in a user’s profile from the conversation histories, we evaluate LLMs by whether they can provide the most suitable response to in-situ user queries, where the user issues the query to LLM in a new conversation session from the first-person perspective. Depending on the time of the in-situ query, the expected response from the model will differ. We cast the problem as a multiple-choice selection, where LLM needs to identify the correct response out of four choices, where the incorrect choices are based on either outdated or irrelevant information with respect to the current state of the user’s profile.

Citation

@misc{jiang2025knowme,
  title={Know Me, Respond to Me: Benchmarking LLMs for Dynamic User Profiling and Personalized Responses at Scale},
  author={Jiang et al. (2025)},
  year={2025},
  note={arXiv:2504.14225}
}

• arXiv: 2504.14225