Lora Dropout Eval

lora-dropout-eval

LoRA Meets Dropout under a Unified Framework — Wang et al. (2024) (arXiv:2403.00812, 2024)

What this evaluates

Evaluates the effectiveness of transformer-specific dropout methods (e.g., HiddenKey, DropKey, HiddenCut) when combined with LoRA for parameter-efficient fine-tuning. It probes the model's ability to mitigate overfitting in LoRA settings across diverse natural language understanding and generation tasks.

Datasets

• GLUE — total ?; splits: train (-1), val (-1), test (-1)
• E2E — total ?; splits: train (-1), test (-1)
• WebNLG — total ?; splits: train (-1), test (-1)

Metrics

• Accuracy (primary) — range: percent
  • Percentage of correctly predicted class labels.
• Pearson correlation — range: [-1, 1]
  • Pearson correlation coefficient between predicted and reference scores.
• BLEU (primary) — range: percent
  • n-gram based precision metric for machine translation and text generation.
• NIST — range: percent
  • Information-content-weighted variant of BLEU.
• METEOR — range: percent
  • Metric that aligns machine translation with reference translations using synonymy and stemming.
• ROUGE_L — range: percent
  • Recall-Oriented Understudy for Gisting Evaluation based on longest common subsequence.
• CIDEr — range: percent
  • Consensus-based Image Description Evaluation metric using TF-IDF weighting.
• TER — range: percent
  • Translation Edit Rate measuring the number of edits needed to change hypothesis to reference.
• Matthews correlation — range: [-1, 1]
  • Correlation coefficient between predicted and actual binary classifications.

Input / output format

Input: Text sequences for NLU tasks; source sentences for NLG tasks.

Output: Class labels or scores for NLU; generated text sequences for NLG.

Scoring recipe

def score(predictions, golds, metric, task):
    if task == 'NLU':
        if metric == 'Accuracy':
            return sum(p == g for p, g in zip(predictions, golds)) / len(golds)
        elif metric == 'Matthews_corr':
            return matthews_corrcoef(golds, predictions)
        elif metric == 'Pearson_corr':
            return pearsonr(golds, predictions)[0]
    elif task == 'NLG':
        if metric == 'BLEU':
            return compute_bleu(golds, predictions)
        elif metric == 'ROUGE_L':
            return compute_rouge_l(golds, predictions)
        # ... other NLG metrics similarly

Common pitfalls

• LoRA baseline uses rank 8 and scalar 16, differing from the original LoRA paper's default rank 4.
• DropAttention's gradient stopping mechanism (NoGrad()) causes instability and poor performance at higher dropout rates.
• KL divergence regularization improves performance, while JS divergence shows no apparent impact in LoRA settings.

Evidence (verbatim from paper)

For NLU tasks, we utilize six datasets from GLUE benchmark*(Wang et al., [2018])*: SST-2 (Socher et al., [2013])... Pearson correlation is reported for STS-B, while accuracy is utilized for others.

Citation

@misc{wang2024lora,
  title={LoRA Meets Dropout under a Unified Framework},
  author={Wang et al. (2024)},
  year={2024},
  note={arXiv:2403.00812}
}

• arXiv: 2403.00812