# Fed Plora Eval

> Evaluates the performance of heterogeneous federated fine-tuning methods across multiple NLP domains (instruction following, NLU, finance) under both IID and non-IID data partitioning settings. It probes the ability of LoRA-based federated learning algorithms to maintain model accuracy while accommodating resource-constrained clients with varying rank allocations. Use when the user wants to benchmark on Natural Instructions, GLUE benchmark, FPB, FIQA, TFNS, or asks about evaluating this task. Reports Rouge-L, Accuracy.

- Skill: `qhjqhj00/fed-plora-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/fed-plora-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/fed-plora-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/fed-plora-eval

---


# fed-plora-eval

> Heterogeneous Federated Fine-Tuning with Parallel One-Rank Adaptation — Zhang et al. (2026) (arXiv:2602.16936, 2026)

## What this evaluates

Evaluates the performance of heterogeneous federated fine-tuning methods across multiple NLP domains (instruction following, NLU, finance) under both IID and non-IID data partitioning settings. It probes the ability of LoRA-based federated learning algorithms to maintain model accuracy while accommodating resource-constrained clients with varying rank allocations.

## Datasets

- **Natural Instructions** — total ?; splits: train (-1), test (-1)
- **GLUE benchmark** — total ?; splits: train (-1), test (-1)
- **FPB** — total ?; splits: train (-1), test (-1)
- **FIQA** — total ?; splits: train (-1), test (-1)
- **TFNS** — total ?; splits: train (-1), test (-1)

## Metrics

- `Rouge-L` **(primary)** — range: [0, 1]
  - Standard Rouge-L score measuring the longest common subsequence between generated text and reference, normalized by reference length.
- `Accuracy` **(primary)** — range: [0, 1]
  - Proportion of correctly predicted class labels out of total instances. For GLUE, task-specific accuracy or macro-F1 is averaged across all sub-tasks.

## Input / output format

**Input**: Instruction-response pairs or classification prompts, partitioned across clients using IID splits or pathological/Dirichlet non-IID distributions.

**Output**: Fine-tuned model predictions (text generations or class labels) aggregated into a global model for evaluation.

## Scoring recipe

```python
def evaluate(predictions, golds, metric='accuracy'):
    if metric == 'rouge-l':
        return rouge_l_score(predictions, golds)
    elif metric == 'accuracy':
        correct = sum(1 for p, g in zip(predictions, golds) if p == g)
        return correct / len(golds)
    # For multi-task benchmarks like GLUE, compute per-task metric then average
    return sum(task_scores) / len(task_scores)
```

## Common pitfalls

- Non-IID settings use pathological or Dirichlet data partitioning, not random splits.
- Heterogeneous baselines use uneven rank allocations (e.g., ranks 1, r_m, R) rather than uniform ranks.
- Results are averaged over three independent runs, with standard deviations reported as upper/lower bounds.

## Evidence (verbatim from paper)

> Table 1 reports averaged Rouge-L scores of the fine-tuned global model under both IID and non-IID settings... As shown in Table [2], our method, Fed-PLoRA, demonstrates substantial improvements over these baselines on IID GLUE benchmark... The main experimental results are reported as the average with upper and lower deviations over three repeat experiments.

## Citation

```bibtex
@misc{zhang2026heterogeneous,
  title={Heterogeneous Federated Fine-Tuning with Parallel One-Rank Adaptation},
  author={Zhang et al. (2026)},
  year={2026},
  note={arXiv:2602.16936}
}
```

- arXiv: 2602.16936