Continual Multimodal Eval

continual-multimodal-eval

Mitigating Intra- and Inter-modal Forgetting in Continual Learning of Unified Multimodal Models — Wei et al. (2025) (arXiv:2512.03125, 2025)

What this evaluates

This benchmark evaluates a model's ability to sequentially learn a mix of visual understanding and generation tasks without catastrophically forgetting previously acquired knowledge. It specifically probes intra-modal retention (maintaining performance on earlier tasks) and inter-modal stability (preventing updates for one modality from degrading the other).

Datasets

• ScienceQA — total ?; splits: test (-1)
• TextVQA — total ?; splits: test (-1)
• GQA — total ?; splits: test (-1)
• VizWiz — total ?; splits: test (-1)
• ImageNet — total ?; splits: test (-1)
• CustomConcept101 — total 101; splits: test (-1)

Metrics

• Average Accuracy (ACC) (primary) — range: [0, 1]
  • Mean accuracy across all tasks after the final round of training. Answer correctness is determined via exact match.
• Average Forgetting (Fgt) — range: [0, 1]
  • Mean reduction from the best observed accuracy to the final accuracy across all tasks. Lower values indicate better retention.
• Image alignment — range: [-1, 1]
  • Cosine similarity between CLIP embeddings of the generated image and a reference image depicting the target concept. Higher is better.
• Text alignment — range: [-1, 1]
  • Cosine similarity between the generated image embedding and the CLIP embedding of its text prompt. Higher indicates better prompt fidelity.
• Fréchet Inception Distance (FID) — range: [0, ∞)
  • Fréchet distance between feature distributions of generated and real images. Lower values mean synthetic images are closer to the real distribution.

Input / output format

Input: For understanding tasks: an image paired with a text question or classification prompt. For generation tasks: a text prompt describing a target concept or scene.

Output: For understanding tasks: a text string (answer or class label). For generation tasks: a synthetic image.

Scoring recipe

# Average Accuracy (ACC)
acc_scores = [1.0 if pred == gold else 0.0 for pred, gold in zip(predictions, golds)]
ACC = sum(acc_scores) / len(acc_scores)

# Average Forgetting (Fgt)
forgetting_scores = [best_acc - final_acc for best_acc, final_acc in zip(best_accuracies, final_accuracies)]
Fgt = sum(forgetting_scores) / len(forgetting_scores)

Common pitfalls

• Low average forgetting does not necessarily indicate effective knowledge retention; it can simply mean the model failed to learn the task in the first place (e.g., starting from a very low baseline accuracy).
• Metric directionality varies across the benchmark: ACC, Fgt, and alignment scores are higher-is-better, while FID is lower-is-better. Misinterpreting the optimization direction leads to incorrect baseline comparisons.
• Exact-match evaluation is used for VQA tasks, which may penalize semantically correct but syntactically different answers.

Evidence (verbatim from paper)

For continual instruction tuning, we follow the protocols from [69, 70] and report two metrics: (1) Average Accuracy (ACC), defined as the mean accuracy across all tasks after the final round of training; and (2) Average Forgetting (Fgt), which quantifies performance degradation on previous tasks and is computed as the mean reduction from the best observed accuracy to the final accuracy, for each task. We use exact matches to determine answers correctness.

Citation

@misc{wei2025mode,
  title={Mitigating Intra- and Inter-modal Forgetting in Continual Learning of Unified Multimodal Models},
  author={Wei et al. (2025)},
  year={2025},
  note={arXiv:2512.03125}
}

• arXiv: 2512.03125