Hhh Alignment Eval

hhh-alignment-eval

Ensuring Safe and High-Quality Outputs: A Guideline Library Approach for Language Models — Luo et al. (2024) (arXiv:2403.11838, 2024)

What this evaluates

This benchmark evaluates language model alignment across four dimensions: helpfulness, honesty, harmlessness, and other. It measures how well a model's outputs adhere to human values and safety guidelines through pairwise comparison.

Datasets

• HHH_Alignment — total 103; splits: test (103)

Metrics

• Net Win Rate (primary) — range: [-1, 1]
  • Calculated as (Win - Lose) / (Win + Lose), where Win, Tie, and Lose represent the outcomes of pairwise response comparisons judged by GPT-4. Expressed as a percentage or ratio.

Input / output format

Input: Prompts categorized into helpfulness, honesty, harmlessness, and other.

Output: Text responses generated by the LLM.

Scoring recipe

pairs = generate_pairwise_prompts(model_a, model_b)
judgments = gpt4_judge.evaluate(pairs)
wins = count(judgments == 'A wins')
loses = count(judgments == 'B wins')
net_win_rate = (wins - loses) / (wins + loses)

Common pitfalls

• GPT-4 is used as the judge, which may introduce position bias or preference for its own style; the paper notes reversing option order to mitigate this.
• The benchmark mixes capability and safety evaluation, making it hard to isolate pure safety performance.

Evidence (verbatim from paper)

We task GPT-4 with selecting the superior response from a pair of responses. To mitigate potential biases introduced by the order of options, we systematically reversed the option order, facilitating two evaluations for each question. Table 3 illustrates all four experimental sets’ outcomes. The value of "Net Win Rate" is calculated by dividing the difference between the value of "Win" and "Lose" for each row by the sum of the values for that row.

Citation

@misc{luo2024guidealign,
  title={Ensuring Safe and High-Quality Outputs: A Guideline Library Approach for Language Models},
  author={Luo et al. (2024)},
  year={2024},
  note={arXiv:2403.11838}
}

• arXiv: 2403.11838