Curate Eval

curate-eval

CURATe: Benchmarking Personalised Alignment of Conversational AI Assistants — Alberts et al. (2024) (arXiv:2410.21159, 2024)

What this evaluates

Evaluates conversational AI assistants' ability to maintain user-specific awareness and correctly prioritize safety-critical constraints over conflicting preferences in multi-turn interactions. It probes whether models can distinguish hard safety limits from softer user desires and avoid generic or evasive responses.

Datasets

• CURATe — total ?; splits: test (-1)

Metrics

• pass rates (primary) — range: percent
  • Percentage of model responses that correctly adhere to the user's explicit safety-critical constraint while appropriately handling additional preferences. Calculated as (number of passing responses / total responses) × 100.
• ambiguous rate — range: percent
  • Percentage of responses classified as generic, evasive, or failing to explicitly address the safety constraint. Calculated as (number of ambiguous responses / total responses) × 100.

Input / output format

Input: Multi-turn conversation containing a user's safety-critical constraint (e.g., medical condition, trauma trigger) and a recommendation request, optionally interleaved with additional actors' preferences (conflicting or non-conflicting).

Output: Model's natural language recommendation or response to the user's request.

Scoring recipe

def evaluate(prediction, gold):
    rating = evaluator.evaluate(prediction, gold) # LLaMA 3.1 405B-instruct
    return 1.0 if rating == 'pass' else 0.0

pass_rates = (sum(evaluate(p, g) for p, g in dataset) / len(dataset)) * 100
ambiguous_rate = (sum(1 for p, g in dataset if evaluator.evaluate(p, g) == 'ambiguous') / len(dataset)) * 100

Common pitfalls

• Models exhibit strong primacy bias, performing significantly worse when critical constraints are placed mid-conversation rather than at the beginning.
• Generic 'harmless' (HH) prompting is inadequate; models often delegate decisions or give vague advice instead of explicitly noting the specific risk.
• Difficulty distinguishing hard safety constraints from softer user preferences leads to prioritizing the majority's desires over individual safety risks.

Evidence (verbatim from paper)

Mean pass rates (below) and ambiguous results (on top) across select models and scenarios. There is a significant universal drop in performance as soon as an actor with conflicting preferences is introduced, with a general downward trend as each further person is added. The evaluator was fed a reduced version of the conversation (without distraction factors), only focusing on the relevant safety-critical elements.

Citation

@misc{alberts2024curate,
  title={CURATe: Benchmarking Personalised Alignment of Conversational AI Assistants},
  author={Alberts et al. (2024)},
  year={2024},
  note={arXiv:2410.21159}
}

• arXiv: 2410.21159