Corda Peft Eval

corda-peft-eval

CorDA: Context-Oriented Decomposition Adaptation of Large Language Models for Task-Aware Parameter-Efficient Fine-tuning — Yang et al. (2024) (arXiv:2406.05223, 2024)

What this evaluates

Evaluates parameter-efficient fine-tuning methods across mathematical reasoning, code generation, instruction following, and general language understanding tasks, while measuring their ability to retain pre-trained world knowledge.

Datasets

• MetaMathQA — total ?; splits: train (-1)
• GSM8k — total ?; splits: test (-1)
• Math — total ?; splits: validation (-1)
• CodeFeedback — total ?; splits: train (-1)
• HumanEval — total ?; splits: test (-1)
• MBPP — total ?; splits: test (-1)
• WizardLM-Evol-Instruct — total ?; splits: train (-1)
• MTBench — total ?; splits: test (-1)
• TriviaQA — total ?; splits: test (-1)
• NQ open — total ?; splits: test (-1)
• WebQS — total ?; splits: test (-1)
• GLUE — total ?; splits: test (-1)
• Wikitext-2 — total ?; splits: test (-1)
• Penn TreeBank (PTB) — total ?; splits: test (-1)

Metrics

• exact match scores (primary) — range: percent
  • Percentage of generated answers that exactly match the ground truth reference string.
• MTBench score — range: other
  • Average score assigned by an LLM-as-judge model across multi-turn instruction-following conversations, typically on a scale of 1-10.
• Perplexity — range: other
  • Exponential of the average negative log-likelihood of the reference text under the model distribution: exp(-1/N * sum(log P(x_i))). Lower is better.
• Matthew’s correlation / Pearson’s correlation / Accuracy — range: percent
  • Standard GLUE metrics: Matthew’s correlation for CoLA, Pearson’s correlation for STS-B, and accuracy for SST-2, MRPC, QNLI, and RTE.

Input / output format

Input: Text-based prompts, questions, or code instructions provided as strings.

Output: Generated text responses (answers, code snippets, or instructions).

Scoring recipe

def compute_metrics(predictions, golds, task_type):
    if task_type == 'exact_match':
        return sum(1 for p, g in zip(predictions, golds) if p.strip() == g.strip()) / len(golds) * 100
    elif task_type == 'perplexity':
        return math.exp(-sum(log_probs) / len(log_probs))
    elif task_type == 'glue':
        return compute_glue_metrics(predictions, golds)
    elif task_type == 'mtbench':
        return llm_judge_score(predictions, golds)

Common pitfalls

• Knowledge-preserved adaptation freezes high-singular-value components to retain world knowledge, which inherently trades off peak task performance compared to instruction-previewed mode.
• MTBench relies on LLM-as-judge evaluation, meaning scores can fluctuate based on the judge model version, temperature, and prompt formatting.
• Exact-match scoring is strict for math and code; execution-based or relaxed matching often yields higher scores but is not used in this paper's primary tables.

Evidence (verbatim from paper)

The world knowledge is evaluated by the exact match scores (%) on TriviaQA, NQ open, and WebQS. Following the settings in [[47]], the Math ability is trained on MetaMathQA and tested on GSM8k and Math validation sets.

Citation

@misc{yang2024corda,
  title={CorDA: Context-Oriented Decomposition Adaptation of Large Language Models for Task-Aware Parameter-Efficient Fine-tuning},
  author={Yang et al. (2024)},
  year={2024},
  note={arXiv:2406.05223}
}

• arXiv: 2406.05223