Dpo Ppo Multi Bench Eval

dpo-ppo-multi-bench-eval

Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback — Ivison et al. (2024) (arXiv:2406.09279, 2024)

What this evaluates

This protocol evaluates language models across factual knowledge, mathematical reasoning, instruction following, code generation, truthfulness, and safety/refusal capabilities. It uses standardized benchmarks to measure how preference optimization methods and data quality impact model performance.

Datasets

• MMLU — total ?; splits: test (-1); repo https://github.com/hendrycks/test
• GSM8k — total ?; splits: test (-1)
• Big Bench Hard — total ?; splits: test (-1)
• TruthfulQA — total ?; splits: test (-1); repo https://github.com/sylinrl/TruthfulQA/
• AlpacaEval — total ?; splits: test (-1)
• IFEval — total ?; splits: test (-1); repo https://github.com/google-research/google-research/tree/master/instruction_following_eval
• HumanEval+ — total ?; splits: test (-1)
• MBPP+ — total ?; splits: test (-1)
• ToxiGen — total ?; splits: test (-1); HF tomh/toxigen_roberta
• XSTest — total ?; splits: test (-1); repo https://github.com/paul-rottger/exaggerated-safety

Metrics

• average accuracy (primary) — range: [0, 1]
  • Proportion of correctly answered questions after extracting the final answer (e.g., last number for GSM8k, first word after trigger phrase for BBH).
• Loose Accuracy — range: [0, 1]
  • Prompt-level accuracy where a response is correct only if all verifiable constraints are satisfied after normalization.
• pass@10 — range: [0, 1]
  • Probability that at least one of 10 sampled code generations passes all test cases, sampled at temperature 0.8.
• % Informative and Truthful — range: [0, 100]
  • Percentage of responses judged as both informative and truthful by GPT-based classifiers.
• F1 metric — range: [0, 1]
  • F1 score aggregating precision and recall of model refusals on safety prompts, detected via GPT-4.

Input / output format

Input: Task-specific prompts with few-shot examples (0 for MMLU, 8 for GSM8k, 3 for BBH), instruction templates for fine-tuned models, or safety/toxicity prompts. Models generate text, numbers, or code up to 8192 tokens.

Output: Model-generated responses containing final answers, code completions, or refusal statements. Answers are post-processed via regex/keyword extraction before scoring.

Scoring recipe

def score(predictions, gold):
    scores = []
    for pred, gold_ans in zip(predictions, gold):
        if dataset == 'GSM8k':
            extracted = extract_last_number(pred)
        elif dataset == 'BBH':
            extracted = extract_first_word_after(pred, 'So the answer is') or pred
        elif dataset == 'IFEval':
            extracted = normalize_constraints(pred)
        else:
            extracted = pred.strip()
        
        if dataset == 'IFEval':
            correct = all_constraints_satisfied(extracted, gold_constraints)
        elif dataset in ['HumanEval+', 'MBPP+']:
            correct = run_code_and_check_tests(extracted, gold_tests, k=10)
        else:
            correct = extracted == gold_ans
        scores.append(correct)
    return sum(scores) / len(scores)

Common pitfalls

• For GSM8k, extracting the first number instead of the last number in the response leads to incorrect accuracy scores.
• For BBH, the fallback rule extracts the entire response if the trigger phrase is missing, which often includes verbose reasoning and causes false negatives.
• For IFEval, Loose Accuracy requires ALL constraints to be met after normalization; partial matches are incorrectly counted as correct if not strictly enforced.
• For safety benchmarks (ToxiGen, XSTest), using heuristic rules instead of GPT-4 judges yields unreliable refusal detection due to varied model response formats.

Evidence (verbatim from paper)

Because all answers in GSM are numbers, we extract the last number in the model response as the final answer. We report average accuracy across test examples. For the CoT setup, we extract the first word after the phrase ‘So the answer is’, or the entire response if there is no such substring present. We report average accuracy over sub-tasks (all of which use accuracy as the primary metric).

Citation

@misc{ivison2024unpacking,
  title={Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback},
  author={Ivison et al. (2024)},
  year={2024},
  note={arXiv:2406.09279}
}

• arXiv: 2406.09279