Coin Eval

coin-eval

CoIN: A Benchmark of Continual Instruction tuNing for Multimodel Large Language Model — Cheng Chen et al. (2024) (arXiv:2403.08350, 2024)

What this evaluates

Evaluates continual instruction tuning in multimodal large language models by measuring how well they retain task-specific instruction alignment and underlying reasoning knowledge when trained sequentially on diverse datasets.

Datasets

• ScienceQA — total ?; splits: test (-1)
• TextVQA — total ?; splits: test (-1)
• ImageNet — total ?; splits: test (-1)
• GQA — total ?; splits: test (-1)
• VizWiz — total ?; splits: test (-1)
• Grounding — total ?; splits: test (-1)
• VQAv2 — total ?; splits: test (-1)
• OCR-VQA — total ?; splits: test (-1)

Metrics

• Truth Alignment (primary) — range: percent
  • Task-specific accuracy measuring whether the model's output exactly matches the ground truth or follows the required instruction format.
• Reasoning Capability — range: [0, 10]
  • LLM-judged score (0-10) assessing whether the model retains underlying reasoning knowledge despite failing to align with the task instruction.
• MAA — range: percent
  • Mean Accuracy Across tasks, calculated as the arithmetic average of per-task Truth Alignment accuracies.
• BWT — range: percent
  • Backward Transfer, measuring catastrophic forgetting as the average performance drop on previously learned tasks after sequential fine-tuning.

Input / output format

Input: Multimodal prompt consisting of an image and a text instruction/question (e.g., 'What is written on the front of that building? Answer the question using a single word or phrase.')

Output: Text response matching the task's expected format (e.g., single word/phrase, option letter, or OCR token).

Scoring recipe

def compute_metrics(predictions, golds, previous_accuracies, current_accuracies):
    truth_alignment = [1 if p == g else 0 for p, g in zip(predictions, golds)]
    truth_alignment_acc = sum(truth_alignment) / len(truth_alignment)
    reasoning_capability = llm_judge_score(predictions, golds) # 0-10 scale
    maa = sum(truth_alignment_acc) / num_tasks
    bwt = sum(current_acc - prev_acc for prev_acc, current_acc in zip(previous_accuracies, current_accuracies)) / num_tasks
    return truth_alignment_acc, reasoning_capability, maa, bwt

Common pitfalls

• Confusing Truth Alignment (instruction following) with Reasoning Capability (knowledge retention); the paper explicitly shows they degrade independently.
• Assuming multi-task training is always the upper bound; task gaps can make it perform worse than sequential fine-tuning on the first task.
• Ignoring task order diversity; random vs. alphabetical ordering significantly impacts BWT and overall MAA due to knowledge interference.

Evidence (verbatim from paper)

Quantitative results about the ability of Truth Alignment and Reasoning Capability are shown in Tab. 2 and Tab. 3, respectively. For the results of truth alignment of Tab. 2, we have the following observations: ... However, due to the absence of techniques to regulate learning, these models suffer from forgetting, resulting in -32.62 of LLaVA and -16.94 of Qwen-VL in terms of BWT.

Citation

@misc{chen2024coin,
  title={CoIN: A Benchmark of Continual Instruction tuNing for Multimodel Large Language Model},
  author={Cheng Chen et al. (2024)},
  year={2024},
  note={arXiv:2403.08350}
}

• arXiv: 2403.08350