qhjqhj00 oasis-cit-eval
OASIS: Online Sample Selection for Continual Visual Instruction Tuning — Minjae Lee et al. (2025) (arXiv:2506.02011, 2025)
What this evaluates
Evaluates online sample selection methods for continual visual instruction tuning by measuring how effectively models learn from sequential data subsets. It probes the model's capacity for knowledge retention across tasks and its ability to adapt to new domains without catastrophic forgetting.
Datasets
- MICVIT — total ?; splits: continual (-1)
- COAST — total ?; splits: continual (-1)
- Adapt — total ?; splits: continual (-1)
- Long Sequence — total ?; splits: continual (-1)
- TRACE — total ?; splits: continual (-1)
Metrics
$A_{last}$(primary) — range: percent- Accuracy measured at the end of the continual training process across all tasks.
$A_{avg}$— range: percent- Average accuracy computed at each task boundary during the continual learning sequence.
Kernel Density— range: other- Mean pairwise similarity of selected samples under a Gaussian kernel; lower values indicate higher sample diversity.
Input / output format
Input: Image-text pairs formatted as visual instruction tuning prompts, processed sequentially across multiple tasks in a continual learning setting.
Output: Textual responses generated by the MLLM/LLM to the given instructions.
Scoring recipe
def compute_accuracy(preds, golds):
return sum(p == g for p, g in zip(preds, golds)) / len(golds) * 100
def compute_A_last(task_accuracies):
return task_accuracies[-1]
def compute_A_avg(task_accuracies):
return sum(task_accuracies) / len(task_accuracies)
Common pitfalls
- COAST benchmark assumes balanced task sizes (20k samples per task), which contradicts real-world data imbalance and may skew selection baselines.
- Adapt benchmark contains COCO images that overlap with LLaVA's instruction-tuning data, potentially inflating reported accuracy due to data leakage.
- Accuracy metrics are reported as raw numbers in tables but represent percentages; statistical significance is determined via Welch's t-test at 0.05 level.
Evidence (verbatim from paper)
Metrics. We report $A_{last}$, the accuracy measured at the end of training, and $A_{avg}$, the average accuracy measured at each task boundary. Benchmarks. We evaluate on a range of CIT benchmarks, including text-only benchmarks (Long Sequence, TRACE) and multi-modal benchmarks (COAST, Adapt). However, COAST assumes balanced task sizes (i.e., 20k samples per task), which contradicts real-world data imbalance, while Adapt includes datasets containing COCO images that overlap with LLaVA’s instruction-tuning data. To address these limitations, we introduce Multi-image Imbalanced Continual Visual Instruction Tuning (MICVIT)...
Citation
@misc{lee2025oasis,
title={OASIS: Online Sample Selection for Continual Visual Instruction Tuning},
author={Minjae Lee et al. (2025)},
year={2025},
note={arXiv:2506.02011}
}
- arXiv: 2506.02011