Clove Clvqa Eval

clove-clvqa-eval

Symbolic Replay: Scene Graph as Prompt for Continual Learning on VQA Task — Lei et al. (2022) (arXiv:2208.12037, 2022)

What this evaluates

Evaluates continual learning capabilities on Visual Question Answering (CLVQA) by measuring how well a model retains knowledge from previous tasks while learning new ones across scene-incremental and function-incremental settings.

Datasets

• CLOVE — total ?; splits: train (-1), test (-1); repo https://github.com/showlab/CLVQA

Metrics

• average accuracy (%) (primary) — range: percent
  • Accuracy is the percentage of correctly predicted answers. The final reported metric is the average accuracy across all tasks in the sequence and the 6 randomly sampled task orders.

Input / output format

Input: An image and a natural language question (VQA prompt)

Output: A natural language answer (text generation, maximum 12 decoding steps)

Scoring recipe

def compute_cl_accuracy(predictions, golds):
    correct = sum(1 for p, g in zip(predictions, golds) if p == g)
    return (correct / len(golds)) * 100

# Evaluation is performed only at the final iteration of the last task in the sequence.
# Results are averaged across 6 randomly sampled task orders for both CLOVE-scene and CLOVE-function settings.

Common pitfalls

• Models are evaluated exclusively at the last iteration of the final task, not continuously during training.
• Task orders are randomized (6 permutations sampled), so performance varies significantly by order; always report the average across orders.
• CLOVE has two distinct incremental settings (scene and function) that must be evaluated and reported separately.

Evidence (verbatim from paper)

We randomly sample 6 task orders from all possible permutations for evaluation. ... Besides, for both settings, we set γ=1.5 for SGP and report the average accuracy. We use models at last iteration of last task for testing. (Table 2 caption: Summary of average accuracy(%) for different methods under six task orders in CLOVE-scene and CLOVE-function respectively.)

Citation

@misc{lei2022symbolicreplay,
  title={Symbolic Replay: Scene Graph as Prompt for Continual Learning on VQA Task},
  author={Lei et al. (2022)},
  year={2022},
  note={arXiv:2208.12037}
}

• arXiv: 2208.12037