Federated LLM Peft Eval

federated-llm-peft-eval

FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning — Kuang et al. (2023) (arXiv:2309.00363, 2023)

What this evaluates

Evaluates the effectiveness and efficiency of federated fine-tuning large language models using parameter-efficient fine-tuning (PEFT) algorithms across code generation, general language, and mathematical reasoning tasks under different data heterogeneity and privacy constraints.

Datasets

• Fed-CodeAlpaca — total ?; splits: train (-1)
• Fed-Dolly — total ?; splits: train (-1)
• Fed-GSM8K-3 — total ?; splits: train (-1)
• HumanEval — total ?; splits: test (-1)
• HELM — total ?; splits: test (-1)
• GSM8K-test — total ?; splits: test (-1)

Metrics

• Evaluation Scores(%) (primary) — range: percent
  • Reported as mean percentage ± standard deviation over three runs with different random seeds. Computed by comparing model predictions against gold labels on the respective test sets.
• Pass@1 — range: percent
  • Probability that the first generated solution passes the test cases. Averaged across clients for personalized settings.
• GPU Usage (MB) — range: other
  • Peak GPU memory consumption during fine-tuning, excluding input tokens and optimizer state.
• Message Size (MB) — range: other
  • Number of bytes of serialized adapter parameters transmitted between server and client per communication round.
• Computation Time (Sec.) — range: other
  • Duration of one training step (batch size 1) from forward to backward propagation.

Input / output format

Input: Fine-tuning datasets (Fed-CodeAlpaca, Fed-Dolly, Fed-GSM8K-3) partitioned across clients; evaluation instances from HumanEval, HELM, and GSM8K-test.

Output: Model-generated text/code/math solutions; evaluated against gold labels to compute percentage scores or Pass@1.

Scoring recipe

def compute_protocol(predictions, gold, peft_algo, fl_algo, scenario):
    scores = []
    for seed in [1, 2, 3]:
        model = init_llm('LLaMA-7B')
        model = fine_tune(model, dataset, peft_algo, fl_algo, seed)
        preds = model.predict(evaluation_set)
        if peft_algo == 'LoRA' and task == 'code':
            score = pass_at_1(preds, gold)
        else:
            score = exact_match_or_accuracy(preds, gold)
        scores.append(score)
    return mean(scores), std(scores)

Common pitfalls

• Confusing the three training scenarios: Global (centralized), Fed (federated aggregation), and Local (independent client training).
• Half-precision training causes precision loss that disproportionately harms pFedMe compared to FedAvg.
• OPT-2.7B fails on HELM subtasks due to input length limits, requiring exclusion from final scores.

Evidence (verbatim from paper)

The averaged evaluation scores (Pass@1 scores) with their standard deviation are reported. We note that evaluation scores with pFedMe are obtained by benchmarking each personalized client individually and then computing their average scores.

Citation

@misc{kuang2023federatedscollm,
  title={FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning},
  author={Kuang et al. (2023)},
  year={2023},
  note={arXiv:2309.00363}
}

• arXiv: 2309.00363