Continual Instruction Tuning Eval

continual-instruction-tuning-eval

Continual Instruction Tuning for Large Multimodal Models — He et al. (2023) (arXiv:2311.16206, 2023)

What this evaluates

Evaluates the ability of large multimodal models to sequentially learn new instruction-following tasks without catastrophically forgetting previously acquired capabilities. It measures both retained performance on old tasks and the degree of forgetting across sequential training stages.

Datasets

• Flickr30k — total ?; splits: test (-1)
• TextCaps — total ?; splits: test (-1)
• VQA v2 — total ?; splits: test (-1)
• OCR-VQA — total ?; splits: test (-1)
• GQA — total ?; splits: test (-1)
• VizWiz — total ?; splits: test (-1)
• TextVQA — total ?; splits: test (-1)

Metrics

• Average performance ($A_t$) (primary) — range: percent
  • The mean evaluation score across all tasks seen up to the current training stage $t$. Computed as $A_t = \frac{1}{t} \sum_i A_{t,i}$, where $A_{t,i}$ is the score on task $i$ after training on task $t$.
• Average forgetting ($F_t$) — range: percent
  • The mean drop in performance on previously learned tasks relative to their peak historical scores. Computed as $F_t = \frac{1}{t-1} \sum_i \max_{j < t}(A_{j,i}) - A_{t,i}$.

Input / output format

Input: An image paired with a natural language instruction or question specific to the current task (e.g., image captioning, visual question answering).

Output: A natural language text response (caption or answer) generated by the model.

Scoring recipe

def compute_cl_metrics(history):
    # history[t][i] = score on task i after stage t
    t = len(history) - 1
    A_t = sum(history[t][i] for i in range(t + 1)) / (t + 1)
    F_t = sum(max(history[j][i] for j in range(t)) - history[t][i] for i in range(t)) / t
    return A_t, F_t

Common pitfalls

• The forgetting metric $F_t$ compares current performance to the maximum historical score across ALL previous stages, not just the immediately preceding stage.
• Sequential fine-tuning (Seq FT) baselines often perform worse than the initial zero-shot model on old tasks, indicating severe catastrophic forgetting rather than simple performance plateaus.
• Task-similarity-informed regularization weights are adaptive based on image-instruction-output similarity, not constant values, which drastically changes anti-forgetting behavior.

Evidence (verbatim from paper)

For each task and dataset, we report the widely adopted metrics as shown in Appendix A following [5]. Let $A_{t,i}$ be the evaluation score on task $\mathcal{T}_i$ after training on task $\mathcal{T}_t$. We compute the average performance on all seen tasks after training on each task $\mathcal{T}_i$: $$ A _ {t} = \frac {1}{t} \sum _ {i} A _ {t, i} $$ To measure the degree of forgetting, we also report the average forgetting on all old tasks after each stage $t$: $$ F _ {t} = \frac {1}{t - 1} \sum _ {i} \max _ {j < t} \left(A _ {j, i}\right) - A _ {t, i} $$

Citation

@misc{he2023continual,
  title={Continual Instruction Tuning for Large Multimodal Models},
  author={He et al. (2023)},
  year={2023},
  note={arXiv:2311.16206}
}

• arXiv: 2311.16206