qhjqhj00 monkey-jump-eval
Monkey Jump : MoE-Style PEFT for Efficient Multi-Task Learning — Prottasha et al. (2026) (arXiv:2601.06356, 2026)
What this evaluates
Evaluates the performance and parameter/memory/throughput efficiency of a gradient-free MoE-style PEFT routing mechanism across text, image, and video benchmarks compared to standard and MoE-PEFT baselines.
Datasets
- 47 Benchmarks (Text/Image/Video) — total 153000; splits: test (-1)
Metrics
accuracy(primary) — range: percent- Percentage of correctly predicted labels or tokens compared to ground truth. Reported as mean ± standard deviation over 5 independent runs.
throughput— range: it/s- Number of iterations processed per second during training or inference.
peak_gpu_memory— range: GB- Maximum GPU memory allocated during forward/backward passes.
Input / output format
Input: Text prompts (Llama-3-8B-Instruct), images, or video frames (LLaVA-OneVision-Qwen2-7B) depending on modality.
Output: Class labels or generated text responses.
Scoring recipe
def compute_accuracy(predictions, gold_labels):
correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
return (correct / len(gold_labels)) * 100
def compute_efficiency(model, dataloader, device):
params = sum(p.numel() for p in model.parameters() if p.requires_grad)
peak_mem = torch.cuda.max_memory_allocated(device) / 1e9
start = time.time()
for batch in dataloader:
model(batch)
throughput = len(dataloader) / (time.time() - start)
return params, peak_mem, throughput
Common pitfalls
- Results are averaged over 5 runs with standard deviation reported; single-run scores may vary significantly.
- Efficiency metrics (memory, throughput) are hardware-specific (H100 GPU) and may not generalize to other architectures.
- MJ variants reuse existing adapters rather than adding new experts, so total model size remains nearly identical to baselines despite different routing.
Evidence (verbatim from paper)
On CS&QA, MJLoRAFA achieves the highest accuracy (66.16%), outperforming MoRE (66.09%) and MixLoRA (66.03%). ... MJ achieves 1.5–2× faster training. MJ-Propulsion reaches 5.94 it/s throughput and completes training in 5.0 minutes, compared to 3.02–3.83 it/s and 7.7–9.4 minutes for MoE-PEFT methods.
Citation
@misc{prottasha2026monkeyjump,
title={Monkey Jump : MoE-Style PEFT for Efficient Multi-Task Learning},
author={Prottasha et al. (2026)},
year={2026},
note={arXiv:2601.06356}
}
- arXiv: 2601.06356