Mcitlib Eval

mcitlib-eval

MCITlib: Multimodal Continual Instruction Tuning Library and Benchmark — Guo et al. (2025) (arXiv:2508.07307, 2025)

What this evaluates

Evaluates multimodal continual instruction tuning (MCIT) methods on MLLMs, measuring their ability to learn new multimodal tasks sequentially while mitigating catastrophic forgetting and cross-modal conflict.

Datasets

• MLLM-DCL — total ?; splits: test (-1); repo https://github.com/Ghy0501/MCITlib
• UCIT — total ?; splits: test (-1); repo https://github.com/Ghy0501/MCITlib

Metrics

• MAA (primary) — range: percent
  • Mean Accuracy Across tasks; reflects the model’s average performance across all tasks and throughout the entire continual learning process.
• MFN (primary) — range: percent
  • Mean Final Network performance; reflects the model’s average performance across all tasks and throughout the entire continual learning process.
• MFT — range: percent
  • Measures performance on new tasks; higher values indicate better ability to learn new tasks effectively.
• BWT — range: percent
  • Backward Transfer; measures forgetting by comparing final performance on previous tasks to their performance immediately after learning. Negative values indicate forgetting.

Input / output format

Input: Multimodal instruction tuning instances (image-text pairs with prompts) organized sequentially across domains (Remote Sensing, Medical, Scientific, Financial, CLEVR-Math, etc.). Exact per-instance format not specified in the excerpt.

Output: Model predictions/responses to multimodal instructions. Evaluated via domain-specific accuracy or task-specific metrics aggregated into MFT, MFN, MAA, and BWT.

Scoring recipe

# Pseudo-code based on text description
# MAA/MFN: Average accuracy across all tasks and stages
maa = mean([accuracy(task_i, stage_j) for all tasks i, stages j])
# BWT: Backward Transfer (performance drop on previous tasks after learning new ones)
bwt = mean([accuracy(task_i, final_stage) - accuracy(task_i, stage_i)] for i in tasks)
# MFT: Forward/Task performance metric (higher is better)
mft = mean([accuracy(task_i, stage_i) for i in tasks])

Common pitfalls

• BWT values are typically negative in continual learning; a higher (less negative) BWT indicates less forgetting, not worse performance.
• Some methods achieve high BWT by severely limiting new task learning ('learn less and forget less'), which artificially inflates forgetting metrics but harms overall utility.
• MFN and MAA aggregate performance across all stages and tasks, so they can mask stage-specific degradation or catastrophic forgetting on individual domains.

Evidence (verbatim from paper)

MFN and MAA reflect the model’s average performance across all tasks and throughout the entire continual learning process, making them more representative. Among all methods, DISCO achieves the best results on both benchmarks... Although all methods show improvements in the BWT metric, which suggests some mitigation of forgetting, a closer examination of the MFT shows that certain methods, such as ModalPrompt and HiDe, achieve this by compromising their ability to learn new tasks effectively.

Citation

@misc{guo2025mcitlib,
  title={MCITlib: Multimodal Continual Instruction Tuning Library and Benchmark},
  author={Guo et al. (2025)},
  year={2025},
  note={arXiv:2508.07307}
}

• arXiv: 2508.07307