Ifeval Eval

ifeval-eval

Instruction-Following Evaluation for Large Language Models — Zhou et al. (2023) (arXiv:2311.07911, 2023)

What this evaluates

Evaluates large language models' ability to follow explicit, verifiable instructions embedded in prompts, such as length constraints, keyword inclusion, formatting rules, and language requirements. It measures both strict and loose compliance across individual instructions and entire prompts to assess structural and syntactic robustness.

Datasets

• IFEval — total 541; splits: test (541); repo https://github.com/google-research/google-research

Metrics

• Prompt-level strict-accuracy — range: [0, 1]
  • Percentage of prompts where all verifiable instructions within that prompt are followed exactly.
• Inst-level strict-accuracy (primary) — range: [0, 1]
  • Percentage of individual verifiable instructions across all prompts that are followed exactly.
• Prompt-level loose-accuracy — range: [0, 1]
  • Prompt-level accuracy computed with a loose criterion that tolerates minor formatting variations (e.g., whitespace, capitalization) as defined in the paper's Section 2.2.
• Inst-level loose-accuracy — range: [0, 1]
  • Instruction-level accuracy computed with the same loose criterion, allowing minor variations while still checking core constraint satisfaction.

Input / output format

Input: A text prompt containing natural language instructions with one or more explicit, verifiable constraints (e.g., word count limits, required keywords, specific output formats, or language requirements).

Output: The model's generated response text, which is parsed deterministically to check compliance against the verifiable constraints specified in the prompt.

Scoring recipe

def compute_metrics(predictions, golds):
    prompt_strict_correct = 0
    inst_strict_correct = 0
    total_prompts = len(predictions)
    total_insts = 0
    for pred, gold in zip(predictions, golds):
        insts_followed = all(check_constraint(pred, c) for c in gold.constraints)
        if insts_followed:
            prompt_strict_correct += 1
        inst_strict_correct += sum(check_constraint(pred, c) for c in gold.constraints)
        total_insts += len(gold.constraints)
    return {
        'prompt_level_strict_accuracy': prompt_strict_correct / total_prompts,
        'inst_level_strict_accuracy': inst_strict_correct / total_insts
    }

Common pitfalls

• Confusing strict vs. loose accuracy criteria; strict requires exact match to constraints, while loose allows minor formatting variations.
• Evaluating only instruction-level accuracy without considering prompt-level strict accuracy, which penalizes models for failing any single constraint in a multi-instruction prompt.
• Using LLM-as-a-judge for verification instead of deterministic parsing, which introduces bias and reduces reproducibility.

Evidence (verbatim from paper)

For evaluating each model, we compute four accuracy scores: 1. Prompt-level strict-accuracy: The percentage of prompts that all verifiable instructions in each prompt are followed. 2. Inst-level strict-accuracy: The percentage of verifiable instructions that are followed. 3. Prompt-level loose-accuracy: Prompt-level accuracy computed with the loose criterion. See Section 2.2 for details. 4. Inst-level loose-accuracy: Instruction-level accuracy computed with a loose criterion. See Section 2.2 for details.

Citation

@misc{zhou2023ifeval,
  title={Instruction-Following Evaluation for Large Language Models},
  author={Zhou et al. (2023)},
  year={2023},
  note={arXiv:2311.07911}
}

• arXiv: 2311.07911