Instruction Tuning Eval

instruction-tuning-eval

Instruction Tuning with GPT-4 — Peng et al. (2023) (arXiv:2304.03277, 2023)

What this evaluates

Evaluates the instruction-following capability and alignment (helpfulness, honesty, harmlessness) of instruction-tuned LLMs on unseen tasks across English and Chinese.

Datasets

• User-Oriented-Instructions-252 — total 252; splits: test (252)
• Vicuna-Instructions-80 — total 80; splits: test (80)
• Unnatural Instructions — total 68478; splits: test (68478)

Metrics

• Relative Score (GPT-4) (primary) — range: percent
  • Pairwise comparison rated 1-10 by GPT-4. Relative score = (sum of scores for model A) / (sum of scores for model A + sum of scores for model B) * 100%.
• HHH Human Vote % — range: percent
  • Human annotators on MTurk vote on three criteria: Helpfulness, Honesty, Harmlessness. Reported as percentage of votes for win, tie, or lose against a baseline.
• ROUGE-L — range: [0, 1]
  • Standard ROUGE-L score averaged over 9,000 samples from Unnatural Instructions, grouped by ground-truth response length.

Input / output format

Input: Instruction prompt (text) provided to the model.

Output: Text response generated by the model.

Scoring recipe

def compute_relative_score(preds_A, preds_B, prompts):
    scores_A, scores_B = [], []
    for p, a, b in zip(prompts, preds_A, preds_B):
        scores_A.append(gpt4_pairwise_score(a, b))
        scores_B.append(gpt4_pairwise_score(b, a))
    return sum(scores_A) / (sum(scores_A) + sum(scores_B)) * 100

Common pitfalls

• The GPT-4 automatic evaluation uses pairwise comparisons against a specific opponent (ChatGPT or GPT-4), so scores are relative, not absolute.
• Human evaluation only covers the 252-user-oriented instructions, not the full Vicuna or Unnatural Instructions sets.
• ROUGE-L is computed on a 9K subset of Unnatural Instructions, not the full 68K.

Evidence (verbatim from paper)

We compare LLaMA-GPT4 with GPT-4 and Alpaca unnatural instructions in Figure 6. In terms of the average ROUGE-L scores, Alpaca outperforms the other two models.

Citation

@misc{peng2023instructiontuning,
  title={Instruction Tuning with GPT-4},
  author={Peng et al. (2023)},
  year={2023},
  note={arXiv:2304.03277}
}

• arXiv: 2304.03277