Mllm Cl Eval

mllm-cl-eval

MLLM-CL: Continual Learning for Multimodal Large Language Models — Hongbo Zhao et al. (2025) (arXiv:2506.05453, 2025)

What this evaluates

Evaluates continual learning capabilities in multimodal large language models across two settings: Domain Continual Learning (DCL) for acquiring IID domain-specific knowledge (remote sensing, medical, autonomous driving, science, finance) and Ability Continual Learning (ACL) for non-IID skill acquisition (OCR, math & logic, visual perception, GUI agent). It measures the model's ability to learn new tasks sequentially while mitigating catastrophic forgetting of previously learned capabilities.

Datasets

• MLLM-CL — total ?; splits: test (-1)

Metrics

• last accuracy — range: percent
  • The accuracy across all previously seen tasks after the model has learned the final task in the sequence. Calculated as the mean of per-task final accuracies.
• MFT (primary) — range: percent
  • Mean Finetune Accuracy. The average of the model's accuracy on each task immediately after that specific task has been trained.
• MFN — range: percent
  • Mean Final Accuracy. The average of the model's final accuracy on each task after the entire continual learning sequence is complete.
• MAA — range: percent
  • Mean Average Accuracy. The average of the model's accuracy on each task across all training steps or checkpoints.
• BWT — range: percent
  • Backward Transfer. The difference between the final accuracy on a task and its accuracy immediately after training, averaged over all tasks except the last one. Negative values indicate catastrophic forgetting.

Input / output format

Input: Multimodal input consisting of an image and a text prompt/question tailored to the specific domain or ability task.

Output: Text response containing the model's answer or prediction for the given prompt.

Scoring recipe

def compute_metrics(predictions, golds, task_ids):
    task_scores = {}
    for pred, gold, task in zip(predictions, golds, task_ids):
        score = 1.0 if pred == gold else 0.0
        task_scores.setdefault(task, []).append(score)
    last_accuracy = sum(sum(accs)/len(accs) for accs in task_scores.values()) / len(task_scores)
    return {'last_accuracy': last_accuracy}

Common pitfalls

• Evaluating only on the most recently learned task instead of all previously seen tasks, which completely misses catastrophic forgetting.
• Confusing MFT (mean accuracy immediately after each task's training) with MFN (mean final accuracy across all tasks after the full sequence).
• Ignoring the router's selection accuracy, which can significantly impact overall performance if the routing mechanism fails to activate the correct expert module.

Evidence (verbatim from paper)

We report the last accuracy, which is the accuracy of all seen tasks after learning the last task, mean finetune accuracy (MFT), mean final accuracy (MFN), mean average accuracy (MAA), and backward transfer (BWT) following standard metrics in continual learning(Guo et al., 2025a; Chen et al., 2025).

Citation

@misc{zhao2025mllmcl,
  title={MLLM-CL: Continual Learning for Multimodal Large Language Models},
  author={Hongbo Zhao et al. (2025)},
  year={2025},
  note={arXiv:2506.05453}
}

• arXiv: 2506.05453