Prefeval Eval

prefeval-eval

Galaxy: A Cognition-Centered Framework for Proactive, Privacy-Preserving, and Self-Evolving LLM Agents — Bao et al. (2025) (arXiv:2508.03991, 2025)

What this evaluates

Measures an agent's capability to retain and adhere to user preferences during long, multi-turn conversations. It tests whether the model can maintain consistency without external reminders or with explicit preference cues.

Datasets

• PrefEval — total ?; splits: test (-1)

Metrics

• preference retention accuracy (primary) — range: percent
  • Accuracy of whether the agent correctly maintains user preferences across multi-round conversations, measured under Zero-Shot (no reminders) and Reminder conditions.

Input / output format

Input: Long multi-turn conversational prompts containing explicit and implicit user preferences.

Output: Agent responses that must align with previously stated user preferences.

Scoring recipe

correct = 0
total = len(preference_checks)
for check in preference_checks:
  if agent_response_matches_preference(check):
    correct += 1
return (correct / total) * 100

Common pitfalls

• Failing to distinguish between Zero-Shot and Reminder evaluation conditions.
• Evaluating preference retention on short conversations instead of long multi-turn contexts.

Evidence (verbatim from paper)

PrefEval focuses on whether agents can maintain user preferences in long conversations. It has two ways of measuring the accuracy of preference retention in multi-round conversations: without reminding users of their preferences (Zero-Shot) and by reminding users of their preferences (Reminder).

Citation

@misc{bao2025galaxy,
  title={Galaxy: A Cognition-Centered Framework for Proactive, Privacy-Preserving, and Self-Evolving LLM Agents},
  author={Bao et al. (2025)},
  year={2025},
  note={arXiv:2508.03991}
}

• arXiv: 2508.03991