Fed Plora Eval

fed-plora-eval

Heterogeneous Federated Fine-Tuning with Parallel One-Rank Adaptation — Zhang et al. (2026) (arXiv:2602.16936, 2026)

What this evaluates

Evaluates the performance of heterogeneous federated fine-tuning methods across multiple NLP domains (instruction following, NLU, finance) under both IID and non-IID data partitioning settings. It probes the ability of LoRA-based federated learning algorithms to maintain model accuracy while accommodating resource-constrained clients with varying rank allocations.

Datasets

• Natural Instructions — total ?; splits: train (-1), test (-1)
• GLUE benchmark — total ?; splits: train (-1), test (-1)
• FPB — total ?; splits: train (-1), test (-1)
• FIQA — total ?; splits: train (-1), test (-1)
• TFNS — total ?; splits: train (-1), test (-1)

Metrics

• Rouge-L (primary) — range: [0, 1]
  • Standard Rouge-L score measuring the longest common subsequence between generated text and reference, normalized by reference length.
• Accuracy (primary) — range: [0, 1]
  • Proportion of correctly predicted class labels out of total instances. For GLUE, task-specific accuracy or macro-F1 is averaged across all sub-tasks.

Input / output format

Input: Instruction-response pairs or classification prompts, partitioned across clients using IID splits or pathological/Dirichlet non-IID distributions.

Output: Fine-tuned model predictions (text generations or class labels) aggregated into a global model for evaluation.

Scoring recipe

def evaluate(predictions, golds, metric='accuracy'):
    if metric == 'rouge-l':
        return rouge_l_score(predictions, golds)
    elif metric == 'accuracy':
        correct = sum(1 for p, g in zip(predictions, golds) if p == g)
        return correct / len(golds)
    # For multi-task benchmarks like GLUE, compute per-task metric then average
    return sum(task_scores) / len(task_scores)

Common pitfalls

• Non-IID settings use pathological or Dirichlet data partitioning, not random splits.
• Heterogeneous baselines use uneven rank allocations (e.g., ranks 1, r_m, R) rather than uniform ranks.
• Results are averaged over three independent runs, with standard deviations reported as upper/lower bounds.

Evidence (verbatim from paper)

Table 1 reports averaged Rouge-L scores of the fine-tuned global model under both IID and non-IID settings... As shown in Table [2], our method, Fed-PLoRA, demonstrates substantial improvements over these baselines on IID GLUE benchmark... The main experimental results are reported as the average with upper and lower deviations over three repeat experiments.

Citation

@misc{zhang2026heterogeneous,
  title={Heterogeneous Federated Fine-Tuning with Parallel One-Rank Adaptation},
  author={Zhang et al. (2026)},
  year={2026},
  note={arXiv:2602.16936}
}

• arXiv: 2602.16936