Inducer Tuning Eval

inducer-tuning-eval

Inducer-tuning: Connecting Prefix-tuning and Adapter-tuning — Chen et al. (2022) (arXiv:2210.14469, 2022)

What this evaluates

Evaluates parameter-efficient fine-tuning methods on natural language understanding and generation tasks, measuring how well they approximate full fine-tuning performance while using significantly fewer trainable parameters.

Datasets

• MNLI — total ?; splits: test (-1)
• SST2 — total ?; splits: test (-1)
• WebNLG-challenge — total ?; splits: train (-1), test (-1)
• CoQA — total ?; splits: test (-1)

Metrics

• Accuracy (primary) — range: [0, 1]
  • Proportion of correctly predicted class labels out of total instances.
• BLEU — range: [0, 1]
  • N-gram overlap between generated and reference text, typically with a brevity penalty.
• MET — range: [0, 1]
  • METEOR metric measuring alignment-based precision, recall, and penalty for fragmentation.
• TER — range: [0, 1]
  • Translation Error Rate; ratio of edits (insertions, deletions, substitutions) to reference length. Lower is better.
• EM — range: [0, 1]
  • Exact match; 1 if generated answer exactly matches reference, 0 otherwise.
• F1 — range: [0, 1]
  • Harmonic mean of precision and recall for answer extraction.

Input / output format

Input: NLU: single sentence or premise-hypothesis pair. NLG: structured triples (WebNLG) or conversational context plus question (CoQA).

Output: NLU: discrete class label (e.g., entailment/contradiction/neither or positive/negative). NLG: generated text string or short answer.

Scoring recipe

def compute_metrics(predictions, references):
    acc = sum(p == r for p, r in zip(predictions, references)) / len(references)
    bleu = corpus_bleu([references], [predictions])
    ter = sum(edit_distance(p, r) for p, r in zip(predictions, references)) / len(references)
    em = sum(p == r for p, r in zip(predictions, references)) / len(references)
    prec = sum(1 for p, r in zip(predictions, references) if p in r) / len(predictions)
    rec = sum(1 for p, r in zip(predictions, references) if p in r) / len(references)
    f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
    return {'Accuracy': acc, 'BLEU': bleu, 'TER': ter, 'EM': em, 'F1': f1}

Common pitfalls

• WebNLG uses a SEEN/UNSEEN/ALL split where training only uses SEEN categories, but test performance is reported across all splits.
• TER metric is lower-is-better, unlike BLEU/EM/F1 which are higher-is-better.
• Prefix-tuning parameter counts differ between training and inference due to a re-parametrization trick, requiring careful accounting when comparing to adapters.

Evidence (verbatim from paper)

We test the performance of our methods on both NLU and NLG tasks. For NLU tasks, we follow (He et al., 2021a) to use RoBERTaBASE (Liu et al., 2019) on MNLI (Williams et al., 2018) and SST2 (Socher et al., 2013) from the GLUE benchmark (Wang et al., 2019); in SST2, the models predict the two-way sentiment (positive/negative) of a given sentence, and the MNLI task is to decide, given a premise and a hypothesis, whether there is entailment, contradiction, or neither. We use GPT-2SMALL (Radford et al., 2019) for NLG tasks: WebNLG-challenge (Gardent et al., 2017) focuses on table-to-text tasks... CoQA (Reddy et al., 2019) provides the data for conversational question answering... Table 2: Accuracy (%) on MNLI and SST2.

Citation

@misc{chen2022inducertuning,
  title={Inducer-tuning: Connecting Prefix-tuning and Adapter-tuning},
  author={Chen et al. (2022)},
  year={2022},
  note={arXiv:2210.14469}
}

• arXiv: 2210.14469