P3 Niv2 Eval

p3-niv2-eval

Instruction Matters: A Simple yet Effective Task Selection for Optimized Instruction Tuning of Specific Tasks — Changho Lee et al. (2024) (arXiv:2404.16418, 2024)

What this evaluates

This evaluation protocol assesses the zero-shot generalization capability of instruction-tuned language models across diverse NLP tasks. It measures how well a model trained on a selected subset of instruction-tuning datasets performs on held-out tasks from the same meta-datasets and external benchmarks, focusing on both classification accuracy and text generation quality.

Datasets

• P3 (Public Pool of Prompts) — total ?; splits: train (-1), test (-1)
• NIV2 (SuperNaturalInstructions V2) — total ?; splits: train (-1), test (-1)
• Big-Bench — total ?; splits: test (-1)
• Big-Bench Hard (BBH) — total ?; splits: test (-1)

Metrics

• ACC (primary) — range: percent
  • Standard classification accuracy calculated as the proportion of correctly predicted labels out of total instances. For P3, performance is averaged across all instructions within each task before computing the final score.
• ROUGE-L — range: percent
  • Recall-oriented understudy for gisting evaluation based on the longest common subsequence between predicted and reference text. Used primarily for NIV2 and reported as a percentage.

Input / output format

Input: Instruction prompt (Task Definition or paraphrased instruction) followed by task-specific input text. P3 instructions use standardized placeholders ({{text}}, {{candidate}}). NIV2 uses only the Task Definition.

Output: Model-generated prediction: a classification label for P3/BBH tasks, or a generated text sequence for NIV2 tasks. Greedy decoding is used with a maximum length of 256 tokens.

Scoring recipe

def score(predictions, gold, metric='ACC'):
    if metric == 'ACC':
        correct = sum(1 for p, g in zip(predictions, gold) if p == g)
        return (correct / len(gold)) * 100
    elif metric == 'ROUGE-L':
        # Standard ROUGE-L F1 calculation
        return rouge_l_score(predictions, gold) * 100
    # For P3: average task-level accuracy across all instructions per task

Common pitfalls

• P3 tasks contain multiple instructions; evaluation must average performance across all instructions per task, not just per task instance.
• NIV2 instruction tuning only uses the Task Definition; positive/negative examples and explanations are excluded from the selector training.
• Placeholders in P3 instructions (e.g., {{-}}) must be standardized to {{text}} or {{candidate}} to prevent the model from learning misleading shortcuts.
• The evaluation is strictly zero-shot; no samples from held-out evaluation tasks are used for model checkpoint selection or validation.

Evidence (verbatim from paper)

For P3 evaluation, following the evaluation method from Sanh et al. (2022), we apply rank classification and measure the model’s performance on every instruction of the target task. We then calculate the average performance for the task. Note that each target task in P3 has 10.09 instructions on average. For NIV2 evaluation, we follow the same evaluation protocol as in Wang et al. (2022) and report ROUGE-L score. We adopt greedy decoding with a maximum generation length of 256.

Citation

@misc{lee2024instructionmatters,
  title={Instruction Matters: A Simple yet Effective Task Selection for Optimized Instruction Tuning of Specific Tasks},
  author={Changho Lee et al. (2024)},
  year={2024},
  note={arXiv:2404.16418}
}

• arXiv: 2404.16418