Infobench Eval

infobench-eval

InFoBench: Evaluating Instruction Following Ability in Large Language Models — Qin et al. (2024) (arXiv:2401.03601, 2024)

What this evaluates

Evaluates large language models' ability to follow complex, multi-constraint instructions by decomposing them into granular criteria (Content, Linguistic, Style, Format, Number) and measuring adherence. It probes fine-grained instruction following rather than holistic response quality.

Datasets

• InFoBench — total 500; splits: Easy (25), Hard (25); repo https://github.com/qinyiwei/InfoBench

Metrics

• DRFR (primary) — range: [0, 1]
  • Decomposed Requirements Following Ratio. Counts the number of stratified criteria satisfied by the response and normalizes by the total number of requirements in the instruction.
• Pairwise Kappa Agreement — range: [0, 1]
  • Fleiss’ Kappa coefficient measuring inter-annotator agreement across three evaluators who categorize pairwise model responses into three classes: Model-A wins, Tie, or Model-B wins.
• WPLD — range: [0, 2]
  • Weighted Pairwise Label Distance. Computes the expected distance between annotation and ground truth: sum_{i=0}^{2} i * P(PLD=i), where PLD is 0 (correct), 1 (tie misclassified), or 2 (order reversed).

Input / output format

Input: Instruction prompt and the model's generated response.

Output: DRFR: A continuous ratio score between 0 and 1. Pairwise comparison: A categorical label (-1, 0, or 1) indicating relative model performance.

Scoring recipe

def compute_drfr(response, criteria):
    satisfied = sum(1 for c in criteria if response_meets(response, c))
    return satisfied / len(criteria) if criteria else 0.0

def compute_wpld(pred_labels, gold_labels):
    plds = [0 if p == g else (1 if abs(p - g) == 1 else 2) for p, g in zip(pred_labels, gold_labels)]
    return sum(i * p for i, p in enumerate(plds)) / len(plds)

Common pitfalls

• DRFR requires instructions to be explicitly decomposed into specific criteria before scoring; applying it to raw instructions yields meaningless results.
• Pairwise comparisons must be aggregated across multiple annotators (e.g., majority vote or Kappa) to mitigate individual bias, as single-annotator scores show low agreement.
• GPT-4 automatic evaluation uses sequential multi-turn prompts for decomposed questions rather than batched single-turn prompts, which significantly impacts token consumption and context window usage.

Evidence (verbatim from paper)

DRFR counts all the stratified requirements and normalized by the number of requirements in the instruction This enables a direct, side-by-side comparison of response pairs from two distinct models, referred to as Model-A and Model-B. For each instructional prompt, the paired responses are categorized into three distinct classifications: 1) Model-A outperforms Model-B, 2) Model-A and Model-B are equally effective, and 3) Model-B outperforms Model-A. Finally, a Fleiss’ Kappa Agreement is employed to measure the agreements among three evaluators concerning the pairwise categorization.

Citation

@misc{qin2024infobench,
  title={InFoBench: Evaluating Instruction Following Ability in Large Language Models},
  author={Qin et al. (2024)},
  year={2024},
  note={arXiv:2401.03601}
}

• arXiv: 2401.03601