Lora Land Eval

lora-land-eval

LoRA Land: 310 Fine-tuned LLMs that Rival GPT-4, A Technical Report — Justin Zhao et al. (2024) (arXiv:2405.00732, 2024)

What this evaluates

This evaluation probes the effectiveness of LoRA fine-tuning across 31 diverse NLP tasks by comparing base LLMs against their fine-tuned counterparts and proprietary models like GPT-4. It measures how much performance lift fine-tuning provides and whether smaller open-weight models can surpass larger closed-source models after adaptation.

Datasets

• magicoder — total ?; splits: test (-1)
• mmlu — total ?; splits: test (-1)
• glue_wnli — total ?; splits: test (-1)
• arc_combined — total ?; splits: test (-1)
• wikisql — total ?; splits: test (-1)
• boolq — total ?; splits: test (-1)
• customer_support — total ?; splits: test (-1)
• glue_cola — total ?; splits: test (-1)
• winogrande — total ?; splits: test (-1)
• glue_sst2 — total ?; splits: test (-1)
• dbpedia — total ?; splits: test (-1)
• hellaswag — total ?; splits: test (-1)
• glue_qnli — total ?; splits: test (-1)
• e2e_nlg — total ?; splits: test (-1)
• glue_qqp — total ?; splits: test (-1)
• bc5cdr — total ?; splits: test (-1)
• glue_mnli — total ?; splits: test (-1)
• webnlg — total ?; splits: test (-1)
• tldr_content_gen — total ?; splits: test (-1)
• glue_mrpc — total ?; splits: test (-1)
• jigsaw — total ?; splits: test (-1)
• hellaswag_processed — total ?; splits: test (-1)
• viggo — total ?; splits: test (-1)
• glue_stsb — total ?; splits: test (-1)
• gsm8k — total ?; splits: test (-1)
• conllpp — total ?; splits: test (-1)
• tldr_headline_gen — total ?; splits: test (-1)
• drop — total ?; splits: test (-1)
• legal — total ?; splits: test (-1)
• reuters — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Proportion of correct predictions out of total instances. Calculated as (number of correct predictions) / (total number of predictions).
• rouge — range: [0, 1]
  • ROUGE-N metric (typically ROUGE-L or ROUGE-1/2) measuring overlap between generated text and reference text using recall/precision/F1 of n-grams or longest common subsequence.
• mae — range: [0, inf)
  • Mean Absolute Error measuring the average absolute difference between predicted and actual continuous values.

Input / output format

Input: Task-specific prompts or instructions corresponding to each of the 31 NLP benchmarks (e.g., multiple-choice questions, text generation prompts, classification instructions).

Output: Task-specific model outputs (e.g., selected option, generated text, regression value) evaluated against ground-truth references using the task-appropriate metric.

Scoring recipe

def compute_metric(predictions, golds, metric_type):
    if metric_type == 'accuracy':
        return sum(1 for p, g in zip(predictions, golds) if p == g) / len(golds)
    elif metric_type == 'rouge':
        return rouge_score(golds, predictions, rouge_types=['rouge1', 'rouge2', 'rougeL'])
    elif metric_type == 'mae':
        return mean_absolute_error(golds, predictions)

Common pitfalls

• Confusing base model scores with fine-tuned model scores when calculating performance lift.
• Assuming a single evaluation metric applies across all 31 tasks; the benchmark uses accuracy, ROUGE, and MAE depending on the task type.
• Overlooking reported implementation bugs in specific model families (e.g., Gemma) that caused fine-tuning to underperform base models, skewing average lift calculations.

Evidence (verbatim from paper)

mmlu | accuracy | 0.506 | 0.589 | 0.774 | 0.083 | -0.185 glue_wnli | accuracy | 0.437 | 0.873 | 0.93 | 0.436 | -0.057 arc_combined | accuracy | 0.673 | 0.915 | 0.947 | 0.242 | -0.032 wikisql | rouge | 0.301 | 0.898 | 0.909 | 0.597 | -0.011

Citation

@misc{zhao2024loraland,
  title={LoRA Land: 310 Fine-tuned LLMs that Rival GPT-4, A Technical Report},
  author={Justin Zhao et al. (2024)},
  year={2024},
  note={arXiv:2405.00732}
}

• arXiv: 2405.00732