Openassistant Human Eval Eval

openassistant-human-eval-eval

KTO: Model Alignment as Prospect Theoretic Optimization — Ethayarajh et al. (2024) (arXiv:2402.01306, 2024)

What this evaluates

Evaluates the quality of aligned language models on multi-turn conversational prompts by measuring how often their generated responses are preferred over supervised fine-tuning targets by human annotators.

Datasets

• OpenAssistant test set — total 256; splits: test (256)

Metrics

• winrate (primary) — range: percent
  • Winrate = (number of times the aligned model's output is preferred over the SFT target) / (total valid comparisons). Reported with 90% binomial confidence intervals.

Input / output format

Input: Multi-turn conversation where the LLM plays the assistant role, with the final turn being the user prompt.

Output: Generated response from the aligned model (compared against the ground-truth SFT target response).

Scoring recipe

wins = 0
total = 0
for prompt in sampled_prompts:
    model_out = generate(prompt)
    sft_out = get_sft_target(prompt)
    if not requires_domain_expertise(prompt):
        total += 1
        if human_prefers(model_out, sft_out):
            wins += 1
winrate = (wins / total) * 100
ci = binomial_confidence_interval(wins, total, alpha=0.10)

Common pitfalls

• Domain-specific prompts (e.g., coding) were filtered out, reducing the effective sample size from 256 to 214.
• Confidence intervals are reported at 90%, not the standard 95%, which affects statistical significance thresholds.
• Human evaluation is compared against GPT-4-as-a-judge, but agreement rates differ (~66-69%), so results are not directly interchangeable.

Evidence (verbatim from paper)

For human evaluation, we randomly sampled 256 prompts from the OpenAssistant test set and generated outputs from Mistral 7B models aligned with DPO and KTO. All inputs were multi-turn conversations between a user and an assistant, where the LLM played the role of the assistant (see Table 4 for an example) and the last turn in the input was that of the user. These were sent to a third-party data annotation service where a pool of workers picked either the generated output or the SFT target (from the OpenAssistant dataset) as the more appropriate response by the assistant. Any question that required specific domain experience (e.g., coding) were skipped, leading to 214 comparisons for DPO and KTO each. The winrates of the aligned model over the SFT targets are 72.9%±5.3 for KTO and 62.1%±5.7 for DPO (where the intervals are 90% binomial confidence intervals).

Citation

@misc{ethayarajh2024kto,
  title={KTO: Model Alignment as Prospect Theoretic Optimization},
  author={Ethayarajh et al. (2024)},
  year={2024},
  note={arXiv:2402.01306}
}

• arXiv: 2402.01306