Dph Alignment Eval

dph-alignment-eval

Would I Lie To You? Inference Time Alignment of Language Models using Direct Preference Heads — Asada Hadji-Kyriacou et al. (2024) (arXiv:2405.20053, 2024)

What this evaluates

Evaluates language models on natural language understanding, commonsense reasoning, and reading comprehension to measure alignment quality and reasoning preservation. It compares standard log-probability predictions against scores derived from a learned Direct Preference Head (DPH) reward model to assess self-evaluation capabilities.

Datasets

• GLUE — total ?; splits: test (-1)
• RACE — total ?; splits: test (-1)
• ARC — total ?; splits: test (-1)
• OpenBookQA — total ?; splits: test (-1)
• HellaSwag — total ?; splits: val (-1)
• WinoGrande — total ?; splits: val (-1)
• BoolQ — total ?; splits: val (-1)
• PIQA — total ?; splits: val (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Proportion of correctly predicted options or answers out of total instances. Calculated per task and averaged for suite-level scores.
• GLUE average score — range: [0, 1]
  • Macro-average of task-specific scores (accuracy, F1, or correlation) across GLUE subtasks, reported with and without WNLI.
• Spearman/Pearson correlation — range: [-1, 1]
  • Rank (Spearman) or linear (Pearson) correlation coefficient between predicted and gold scores for regression tasks like CoLA and STS-B.

Input / output format

Input: Text prompts containing a question or context followed by multiple-choice options or a sentence completion target.

Output: Model-generated text or selected option index. Predictions are obtained either from highest log-probability completions or by selecting the option with the highest DPH reward score.

Scoring recipe

def compute_accuracy(predictions, golds):
    correct = sum(1 for p, g in zip(predictions, golds) if p == g)
    return correct / len(golds)

def compute_glue_avg(task_scores):
    return sum(task_scores.values()) / len(task_scores)

def compute_correlation(preds, golds):
    # Use scipy.stats.spearmanr or pearsonr
    return correlation_coefficient(preds, golds)

Common pitfalls

• GLUE average score is reported both with and without WNLI; excluding WNLI changes the macro-average significantly.
• Different datasets use different splits (test vs. validation) as specified in the methodology, which must be respected to match reported scores.
• DPH predictions are derived from reward scores on hidden states, not directly from the language modeling head's log-probabilities; confusing the two leads to incorrect baseline comparisons.

Evidence (verbatim from paper)

With SFT alone we are able to attain comparable performance to TinyLlama using half as many parameters, and when applying DPH alignment we achieve a 7.2% increase over the TinyLlama average score and the highest accuracy in 5 of the 7 tasks.

Citation

@misc{asadahadji2024directpreferenceheads,
  title={Would I Lie To You? Inference Time Alignment of Language Models using Direct Preference Heads},
  author={Asada Hadji-Kyriacou et al. (2024)},
  year={2024},
  note={arXiv:2405.20053}
}

• arXiv: 2405.20053