Vilco Bench Eval

vilco-bench-eval

ViLCo-Bench: VIdeo Language COntinual learning Benchmark — Tang et al. (2024) (arXiv:2406.13123, 2024)

What this evaluates

Evaluates video-language continual learning by testing a model's ability to retain episodic memories across streaming, long-duration videos without catastrophic forgetting. It probes cross-modal inference and temporal localization across three non-classification tasks: moment queries, natural language queries, and visual queries.

Datasets

• ViLCo-Bench — total ?; splits: test (-1); repo https://github.com/cruiseresearchgroup/ViLCo

Metrics

• Average Recall@k (IoU=m) (primary) — range: percent
  • Cumulative average recall across a sequence of tasks: P_i = (1/i) * sum_{j=1}^{i} p_{i,j}, where p_{i,j} is the recall on task i after training on j tasks. Top-k predictions are ranked, and a prediction is correct if its Intersection over Union (IoU) with the ground truth exceeds threshold m. Reported for k in {1,5} and m in {0.3,0.5}.
• temporal AP (tAP) — range: percent
  • Average precision computed over a range of IoU thresholds for temporal localization predictions in the Visual Query task. Measures the overlap and distance between predicted and ground-truth bounding boxes.
• Backward Forgetting (BwF) — range: percent
  • BwF_i = (1/(i-1)) * sum_{j=1}^{i-1}(p_{j,j} - p_{i,j}), measuring the average performance drop on previous task j after learning new task i. Lower values indicate less catastrophic forgetting.

Input / output format

Input: A long egocentric video (average 10 minutes) paired with a text query (action category, natural language question, or object description) or a visual query frame.

Output: For MQ/NLQ: predicted temporal window (start and end timestamps). For VQ: predicted 2D bounding box coordinates and object probability.

Scoring recipe

def compute_recall_at_k(pred_windows, gt_windows, k=1, iou_thresh=0.5):
    correct = 0
    for gt in gt_windows:
        ious = [compute_iou(p, gt) for p in pred_windows[:k]]
        if max(ious) >= iou_thresh:
            correct += 1
    return correct / len(gt_windows) * 100

def compute_avg_performance(p_matrix, current_task_idx):
    return sum(p_matrix[current_task_idx][:current_task_idx+1]) / (current_task_idx + 1)

def compute_backward_forgetting(p_matrix, current_task_idx):
    drops = [p_matrix[j][j] - p_matrix[j][current_task_idx] for j in range(current_task_idx)]
    return sum(drops) / len(drops) if drops else 0

Common pitfalls

• Label overlap across tasks is common in egocentric videos; strict partitioning is required to prevent data leakage between sub-tasks.
• IoU thresholds (0.3 and 0.5) drastically change recall scores; metrics must be reported separately for each threshold.
• Long video duration (avg 10 mins) makes frame-level rehearsal buffers memory-inefficient; models must use compact prompt/key representations for long-term memory.

Evidence (verbatim from paper)

In NLQ and MQ tasks, we adopt average recall@k (IoU=m) as the performance metric, where we select top k={1,5}. This metric presents the percentage of query sentences that appear in the top-k predictions with IoU larger than the threshold m={0.3,0.5}. For the VQ task, we leverage temporal AP (tAP) as the performance metric which measures the distance between the predictions and ground-truth localizations. Again we calculate the average of tAP over the previous tasks.(2) Memory stability metrics: Following [[47]], we also consider Backward Forgetting (BwF) to evaluate the performance of CL models. BwF measures the influence caused by learning task i on the performance of the model in remembering previous tasks. BwF_i is calculated as follows: BwF_i = 1/(i-1) sum_{j=1}^{i-1}(p_{j,j} - p_{i,j}).

Citation

@misc{tang2024vilcobench,
  title={ViLCo-Bench: VIdeo Language COntinual learning Benchmark},
  author={Tang et al. (2024)},
  year={2024},
  note={arXiv:2406.13123}
}

• arXiv: 2406.13123